GPU Died. Training Didn't: Self-Healing Training at Scale — Crusoe

Why watch

Engineering leads from Crusoe explain how they built Managed Slurm on Kubernetes to handle GPU failures automatically at scale. The talk covers the limitations of pure Slurm for dynamic infrastructure, the automatic remediation flow for XID 79 GPU errors, and a live demo of killing a GPU mid-training and resuming from checkpoint in under 15 minutes without human intervention. For platform engineers and ML infrastructure teams managing GPU clusters.