Problem
A one-time historical backfill; reprocessing a year of old data to fix a bug in a feature definition; ran on the exact same scheduler and batch job as the pipeline that processes today's data. Current-data processing fell behind for three straight days, and training kept reading stale features the whole time. This time, the backfill gets its own dedicated capacity, checkpoints its progress so a restart doesn't mean starting over, and the daily pipeline's own deadline doesn't move just because a 400-million-record cleanup is running in the background.
Required components
Click to add a component to the canvas.
Additional components
Browse by category. Click or drag a component onto the canvas.