Problem
At 9pm, traffic tripled the second a promotion email went out, and at the exact same moment the nightly scoring run; kicked off six hours early by an on-call engineer trying to get ahead of a deploy window; started consuming the same cluster. Checkout p99 tripled and stayed there until 2am, when the batch run finally finished. Live requests and scheduled batch scoring need to be two fully separate systems: the live path has to absorb a 3x spike without batch stealing its capacity, and the batch path has to finish its full run by morning no matter what daytime traffic is doing.
Required components
Click to add a component to the canvas.
Additional components
Browse by category. Click or drag a component onto the canvas.