ProblemsML0020

Checkpoint and Resume After Failure

Medium

Problem

A 6-day training run on spot GPUs just lost 3 days of progress to a single preemption; the reclaim notice gave the job 2 minutes to save itself, and it didn't. Spot pricing is the only way this run fits the budget, but spot instances can vanish at any time. Design a training setup that survives that: a worker can disappear mid-run, and the job needs to pick back up automatically, from close to where it left off, without anyone being paged.

Required components

Click to add a component to the canvas.

Additional components

Browse by category. Click or drag a component onto the canvas.

Drag components from the library onto the canvas, then drag from a component's right handle to connect it to another.

100%

Design canvas

Sign up to start designing

Sign up free

Premium is free for 58 more days.