ProblemsML0009

Outgrowing the Nightly Window

Medium

Problem

A nightly job reprocesses the last day of raw events into training-ready features, and it used to comfortably finish before the training pipeline needed it. Data volume has grown, and now it's started missing its window; some mornings training kicks off against yesterday's stale output. Shard the job across enough instances that 2 million records finish processing well inside the nightly window, instead of quietly falling further behind as volume keeps growing.

Required components

Click to add a component to the canvas.

Additional components

Browse by category. Click or drag a component onto the canvas.

Drag components from the library onto the canvas, then drag from a component's right handle to connect it to another.

100%

Design canvas

Sign up to start designing

Sign up free

Premium is free for 58 more days.

$0 / $800