Why Building an Eval Platform Is Harder Than It Looks — Braintrust

Why watch

Hossein Niazmandi from Braintrust explains why building AI agent eval platforms is more complex than adding a UI to spreadsheets. He covers the two pillars of agent quality (evals before production and observability after), the maturity stages teams traverse, and how non-deterministic LLMs make both essential. The talk details why agent traces break typical data warehouses and how Braintrust built BTQL to handle huge nested JSON with real-time and long-running queries. Finally, he explores agents running evals themselves to surface unexpected failures.