Beyond Model Sharding: Atomic Scheduling and Disaggregated LLM Serving... Kay Yan & Chen Zicong

Why watch

This talk explores LeaderWorkerSet (LWS), a Kubernetes-native API for managing coordinated pod groups as a single workload unit for multi-host LLM inference. It covers gang scheduling integration with Volcano, scheduler-plugins, and YuniKorn to ensure leader and worker pods are scheduled together, preventing partial scheduling that wastes GPU resources. The session also introduces DisaggregatedSet for coordinating multiple LWS resources across prefill and decode stages, helping attendees understand when Deployment is insufficient and LWS becomes the right abstraction for production LLM serving.