ProblemsEX0025

Kafka-Like Streaming Platform

Hard

Problem

A post-incident review just found that an event a downstream service swore it never received was, in fact, acknowledged by the cluster, and then lost when the broker that accepted it died a few seconds later with nothing else holding a copy yet. Build a distributed event-streaming platform: topics, partitions, producers, and consumer groups, with durable retention and replay, at a real-world scale of roughly 5M events/sec in production. This is the final problem in the entire curriculum; design it from a blank canvas so an acknowledged write can never quietly disappear, the cluster tolerates losing a broker, one partition getting far more traffic than any other doesn't become a permanent bottleneck, and data old enough to be rarely read can move to cheaper storage without losing the ability to replay it.

Required components

Click to add a component to the canvas.

Additional components

Browse by category. Click or drag a component onto the canvas.

Drag components from the library onto the canvas, then drag from a component's right handle to connect it to another.

100%

Design canvas

Sign up to start designing

Sign up free

Premium is free for 58 more days.

$0 / $2,500