Notification System

The write path does four things — validate, claim the key, write the row, ack — and nothing else.

The caller is usually inside somebody's request, so accept has to return an id in single-digit milliseconds. The idempotency key is claimed first, so a retried call returns the original id rather than sending twice. The cache is the fast path and not the guarantee: the same key is a unique index on the row, so a flushed cache turns a retry into an insert conflict that hands back the original id instead of a second notification. Then the row is written — the record of intent, committed before anything downstream can see it — which is what makes it possible to answer later why a person got this.

Accepting is fast and sending is slow, so a log goes between them and the two stop caring about each other.

Delivery is remote, slow, and allowed to fail. Accepting is none of those things and must not become any of them. Publishing after the row commits is what makes the log a replayable ledger rather than a second source of truth — but committing and then publishing is a dual write, and a crash in between leaves an id handed to the caller and nothing on the log. So the row and an outbox record commit in one transaction and a relay tails the outbox onto the log: publishing becomes at-least-once, and the dedupe claim downstream absorbs the repeat. Partitioning by tenant keeps a noisy tenant inside its own partitions, and retaining a few days means a bad consumer deploy is a rewind, not an incident report.

Preferences get read here, at the last moment, not back at the API.

A message can sit in the log for seconds, or during an incident for hours, so a preference read taken at accept time is a stale read by the time it matters. The dispatcher reads the per-type channel matrix, the global unsubscribe, and the suppression list, and turns one notification into zero or more channel sends. Zero is a legitimate answer, and it gets recorded as suppressed with a reason — silently dropping it is how you lose an audit.

Each channel gets its own queue and its own workers, so they are allowed to fail separately.

Channels have nothing in common operationally: push is cheap and fast, email is slow and bursty, text is metered and costs real money per message. On a shared queue the head of the line is whatever is slowest, so one vendor having a bad afternoon stalls every other channel behind it. Separate queues let concurrency, retry policy, and drain rate be tuned per channel — a text queue that drains at twenty a second is a budget control, not a bug. Each channel is split again into an urgent lane and a bulk lane with its own drain ceiling, because the resource actually worth protecting is vendor throughput, and that is where one tenant's campaign would otherwise sit in front of everyone else's one-time codes. The senders render here rather than at the vendor: one template store, versioned and localized, read on the send path, which is what lets a channel move to a different vendor in an afternoon.

I assume the vendor is slow, down, or lying about success, and make the worker survive all three.

Each sender wraps the call in a timeout, retries timeouts and server errors with exponential backoff and full jitter, and never retries a hard rejection — a bad address does not improve on the fourth attempt. A breaker per vendor opens on a sustained error rate so we stop paying latency to something already down, and the queue holds the backlog while it is open. Work that outlives its retry budget goes to a dead-letter queue with the failure attached, because that is where the interesting bugs live and deleting it is how you find out three days later.

I'm not chasing exactly-once — I'm making a second delivery cheap to detect and cheap to drop.

Every hop here can redeliver: the log on a rebalance, the queue on a lost ack, the sender on a timeout it cannot interpret. So before handing work to a channel the dispatcher claims a key on the identity of this notification and this channel — the caller's idempotency key when there is one, a hash of tenant, event id, recipient and type when there is not. It has to be per-event, not per-type: a key of tenant, recipient, type and window would collide two genuinely different comment alerts and drop the second as a duplicate. That tuple is the rate-limit counter, which is a different mechanism living in the same store. The claim is a set-if-absent with a TTL longer than the retry budget, so the redelivered copy loses the race and exits quietly. It bounds fan-out duplicates only; the last hop is bounded separately, by the sender passing a vendor-side idempotency key on the request. Exactly-once dies there anyway — the vendor can accept a message and lose it, and no protocol on our side fixes that.

This layer is the reason people keep the app installed.

The same short-lived store holds per-user, per-type send counters and the recipient's quiet-hour window, evaluated in their time zone, both read on the dispatch path. Anything over the limit or inside quiet hours is held in the digest buffer instead of sent. A scheduled sweep on the dispatcher's own deployment rolls the held items up, writes its own notification row and idempotency claim exactly as the API would, and publishes that to the log — a digest is a notification like any other, so it inherits preferences, dedupe, and an audit trail rather than appearing downstream as a delivery with no parent. Urgent types — one-time codes, password changes, failed payments — carry a bypass and are never held, because a code that arrives an hour late is a broken login, not a polite one.

The in-app inbox is a different system with different pressure, so I draw it as one.

It is written on fan-out — one row per recipient keyed by user and time — so a page of the feed is one index scan and never a join across events. It hangs off the log rather than off preference resolution, because the inbox is the record of what happened to you: muting a type turns off the push, not the history. Unread counts are the expensive part, read on every app open, so the badge is a maintained counter that is rebuilt from the rows when it drifts. Reads outrun writes by an order of magnitude here, and a badge that is slow is worse than a badge that is briefly approximate.

A vendor accepting a message is not a delivery, so the last thing I draw is the path that tells me the truth.

Vendors report asynchronously — delivered, bounced, complained, unsubscribed — into an endpoint that verifies the signature, writes nothing synchronously, and publishes the event, so a callback storm hits a queue instead of the database. A worker folds those outcomes onto the per-attempt rows and writes hard bounces and complaints back into suppression — that stroke is the loop closing, because suppression is the same list the dispatcher reads, so an unsubscribe honors itself on the very next send. An invalid device token is the push equivalent of a hard bounce and takes the same path: the token is deleted rather than suppressed, or we spend forever pushing at a phone that was wiped. The four numbers I would page on: accept latency, consumer lag per channel, send success rate per vendor, and dead-letter arrival rate. Everything else is a dashboard someone looks at on purpose.