The Noisy Neighbour Inside Your GPU:... - Sagar Utekar, CrowdStrike & Oshi Gupta, Improving Pune

Why watch

This SRE-focused talk explores the noisy neighbour problem in GPU-based LLM inference, where requests share a GPU within continuous batching systems like vLLM and TGI. It explains why traditional RED metrics mislead for LLM serving and identifies the signals that actually matter: time-to-first-token, inter-token latency, KV cache utilization, queue depth, and batch occupancy. The session demonstrates using DCGM, Prometheus, and inference-server metrics to trace latency spikes to their true causes—scheduling, memory pressure, or batch-mate interference.