What Is an Inference Engine, Anyway? — Charles Frye, Modal
AI Engineer — Oct 6, 2026 — talk
Why watch
A technical deep-dive into LLM inference engines explaining the complete pipeline from request handling through tokenization, scheduling, model execution, and detokenization. Covers performance optimization techniques like KV caching, CUDA graphs, and speculative decoding using SGLang and vLLM as reference implementations. Ideal for ML engineers and developers building production LLM applications who need to understand inference optimization and deployment patterns.