Problem
A 6-day training run on spot GPUs just lost 3 days of progress to a single preemption; the reclaim notice gave the job 2 minutes to save itself, and it didn't. Spot pricing is the only way this run fits the budget, but spot instances can vanish at any time. Design a training setup that survives that: a worker can disappear mid-run, and the job needs to pick back up automatically, from close to where it left off, without anyone being paged.
Required components
Click to add a component to the canvas.
Additional components
Browse by category. Click or drag a component onto the canvas.