Same Model, Different Speed: Why Your Inference Provider Matters — FriendliAI

Why watch

Yunmo Koo from FriendliAI demonstrates how inference providers dramatically affect AI model performance, even when running the same model. He explains the technical reasons behind speed differences: prefix caching with cache-aware routing, a tiered KV cache hierarchy using GPU, CPU, and NVMe storage, and hybrid speculative decoding. The talk is designed for AI engineers and developers building AI-powered applications who want to understand how to optimize inference speed and reliability.