ProblemsML0038

Cold Starts Make New Servers Slow to Help

Medium

Problem

Twice a day, a product launch email drives an 800 RPS flash spike that lasts about 90 seconds. Every time it hits, the fleet scales out reactively, and by the time a new instance finishes loading the model into memory; roughly 45 seconds; the spike is already over. Design the serving tier so the spike is actually absorbed, without paying for peak capacity sitting idle the other 23+ hours of the day.

Required components

Click to add a component to the canvas.

Additional components

Browse by category. Click or drag a component onto the canvas.

Drag components from the library onto the canvas, then drag from a component's right handle to connect it to another.

100%

Design canvas

Sign up to start designing

Sign up free

Premium is free for 58 more days.

$0 / $2,000