Omnichannel Messaging System

Home > Portfolio > Omnichannel Messaging System

Turning a 2-minute alert delay into a sub-4-millisecond notification pipeline

A distributed messaging platform built to deliver time-critical alerts and communications for a financial institution across every channel a customer might be reached on — email, SMS, and push — from a single, unified pipeline.

The system was built to solve a problem that becomes dangerous, not just inconvenient, in banking: when a customer needs to know something urgently — a fraud alert, transaction alert, an OTP, a failed transaction — every second of delay is a second of exposure.

Key metrics

  • Reduced alert dispatch time from ~2 minutes to under 4 milliseconds
  • Priority-based queue architecture with dedicated lanes for time-critical vs. bulk messaging
  • Distributed, multi-worker processing for horizontal scale
  • Dynamic source mapping from any connected database to message content
  • Automated generation of templated messages and attachments (e.g. account statements)
  • Caters 250+ messages/second
reconciliation system screenshot

Context

For a financial institution, a notification isn't just a UX nicety — it's often part of the security model itself. OTPs, fraud alerts, failed-login notices, and transaction confirmations all depend on the message reaching the customer before the moment it's relevant to has passed.

The system this replaced was slow by design, not by accident: messages of every kind — critical fraud alerts and routine promotional content alike — moved through the same processing path, competing for the same resources. A time-sensitive alert could sit behind a batch of low-priority messages with no way to jump the queue.

That created a real operational risk. A ~2-minute delay on an OTP or a fraud alert isn't a minor inconvenience — it's a window where the system's own communication becomes too slow to be useful for the thing it exists to protect.

The Challenge

The core challenge wasn't building a form to log a complaint. It was building a system that could sit on top of a live core banking system and reason about a transaction's financial state well enough to resolve disputes with minimal manual investigation.

This meant the system needed to:

  • Distinguish between message priority levels and enforce that priority end-to-end, not just at intake.
  • Scale horizontally so throughput wasn't bound to a single processing path.
  • Generate personalized content dynamically, pulling from whatever source database held the relevant data.
  • Support both simple templated messages and generated attachments (e.g. account statements as PDFs).
  • Deliver across multiple channels — email, SMS, push — from one system rather than separate per-channel tooling.
  • Stay reliable under load spikes, since financial events (fraud waves, system-wide failures) are exactly when messaging volume spikes hardest.

In effect, the system needed to behave less like a mail-merge tool and more like a real-time distributed dispatch engine, with message priority treated as a first-class architectural concern rather than a sort order.

My Role

I was involved in the architecture and development of the platform, with responsibility spanning the system rather than individual screens or endpoints.

My work included:

  • Designing the priority-based queueing architecture and the rules that enforced priority end-to-end.
  • Designing the distributed, multi-worker processing model for horizontal scale.
  • Building the dynamic source-mapping layer connecting arbitrary databases to message templates.
  • Designing the templating and attachment-generation system (e.g. account statement generation).
  • Implementing multi-channel delivery (email, SMS, push) from a single pipeline.
  • Designing failover and retry behavior for delivery failures.
  • Tuning the system to bring dispatch latency down from ~2 minutes to under 4ms for critical messages.

The key architectural responsibility was making sure that no matter how much bulk or templated traffic was moving through the system, a critical alert could still be dispatched with effectively no delay.

Engineering Challenges

1
Making priority a structural guarantee, not a queue position

The hardest part of this system wasn't sending messages fast in general — it was guaranteeing that a critical message was never meaningfully delayed, regardless of what else was happening in the system. That meant priority couldn't just be a sort order within one queue; it had to be architected as physically separate processing lanes, so bulk traffic had no path to interfere with critical dispatch. That structural separation is what got critical alert latency down from ~2 minutes to under 4ms.

2
Scaling without creating a single point of contention

A distributed, multi-worker model solves throughput, but only if workers can scale independently without becoming a new bottleneck themselves (e.g. contention on a shared queue or database). Designing the worker pool to scale horizontally while keeping priority guarantees intact — so adding workers improved throughput without weakening the latency guarantee for critical messages — was a core part of the architecture.

3
Supporting arbitrary source data without hard-coding integrations

Financial institutions store customer and transaction data across multiple systems, and message types kept expanding — OTPs, fraud alerts, statements, promotional content, each potentially sourced from a different database. Hard-coding a data pipeline per message type would have made the system unmaintainable as message types grew. The dynamic mapping layer let new sources and message types be onboarded through configuration instead of new integration work.

4
Handling two very different content models in one pipeline

Simple templated messages (an SMS OTP) and attachment-generation messages (a full account statement PDF) have very different processing costs and failure modes. Supporting both without splitting them into separate systems meant the pipeline had to treat content generation as a distinct, pluggable stage — so a heavy attachment-generation job couldn't stall the lightweight templated messages moving through the same infrastructure.

5
Reliable delivery across channels with different failure behavior

Email, SMS, and push each fail differently and have different retry semantics and provider behavior. Unifying dispatch across all three from one pipeline meant building channel-aware retry and failure handling, so a delivery failure on one channel didn't block or slow down messages on another.

Reflection

This project reframed how I think about latency in distributed systems. It's tempting to treat "make it faster" as an optimization problem you solve after the system works. Here, latency was the actual requirement — a fraud alert that arrives in two minutes isn't a slower version of the correct behavior, it's a different, wrong behavior entirely.

That meant the real engineering work wasn't tuning an existing pipeline, it was recognizing that different message priorities needed structurally different paths through the system from the very start. That's the lens I still bring to systems where "fast" isn't a nice-to-have but part of what the system is actually for.

It also changed how I think about queuing: rather than one shared queue with priority flags, we used physically separate queues for critical and bulk traffic, so a backlog on one could never bleed into the other — isolation beat prioritization. On the distributed side, multiple workers had to process each message exactly once, without duplicates or drops during retries or restarts, and without a central bottleneck — so scaling meant adding workers, not redesigning coordination.

The last piece was getting systems that were never built to talk to each other to work together. Customer, transaction, and statement data each lived in different databases with different schemas. Rather than hard-wiring an integration per source-to-channel pairing, the dynamic mapping layer treated any source as an interchangeable input and any channel as an interchangeable output — so new data sources and message types could be added without becoming a new integration project each time.

Get In touch

Feel free to email me at hi@riteshr.com.np for counsuting work, and I'll get back to you.