AI code benchmarks lied to us
Theo - T3 — May 31, 2026 — talk
Why watch
This video critiques existing AI code benchmarks as misleading and introduces a new benchmark (likely SWE-Bench or similar) that better reflects real-world AI coding performance. It targets developers and AI practitioners who rely on benchmark scores to evaluate AI coding tools, revealing how traditional benchmarks may not accurately predict actual coding assistant effectiveness.