ProblemsML0012

Backfills Overwhelm the Pipeline

Medium

Problem

A one-time historical backfill; reprocessing a year of old data to fix a bug in a feature definition; ran on the exact same scheduler and batch job as the pipeline that processes today's data. Current-data processing fell behind for three straight days, and training kept reading stale features the whole time. This time, the backfill gets its own dedicated capacity, checkpoints its progress so a restart doesn't mean starting over, and the daily pipeline's own deadline doesn't move just because a 400-million-record cleanup is running in the background.

Required components

Click to add a component to the canvas.

Additional components

Browse by category. Click or drag a component onto the canvas.

Drag components from the library onto the canvas, then drag from a component's right handle to connect it to another.

100%

Design canvas

Sign up to start designing

Sign up free

Premium is free for 58 more days.