Case study
Scaling a time-critical event pipeline 6x
From 200K to 1.2M events per minute on a queue-based pipeline where a late event counts as a failed one.
Simplified and generalized from my own work: no confidential details, internal names or non-public metrics. Every number matches my resume.
Results
- throughput: 200K → 1.2M events per minute
- 6x
The problem
- Traffic was growing faster than the pipeline could drain it.
- Throughput metrics looked healthy while some events arrived too late to be useful.
- The queue delivers at least once, so duplicates are normal and must be harmless.
The architecture
Follow one video from the moment it is uploaded until it is searchable. Press play, or step through the stages at your own pace.
Decisions and trade-offs
Isolate the critical path
SNS fans every event out to separate queues, one per flow. Each flow scales and fails on its own, so a backlog in a secondary flow never delays a critical event.
Scale on pending work, not CPU
Consumers scale on how many messages are waiting. CPU is the wrong signal for a pipeline whose real constraint is elsewhere.
Find the real bottleneck
Throughput plateaued while CPU looked fine: consumers were starved of network connections. Raising the connection limits removed the ceiling. CPU-based scaling would never have reacted.
Make the effect exactly-once
Delivery cannot be exactly-once, so processing is: a dedup key on the event id, state-shaped writes and conditional writes.
Alarm on age, not throughput
The key alarm is the age of the oldest message, and it notifies the team through SNS. A late event is a failed event, even when events per second look great. Dead-letter queues, a throttle and a kill switch complete the runbook.