Your LLM Judge Is a Confident Liar: Building Better Verifiers — Browserbase

Why watch

Researchers from Browserbase and Microsoft Research present findings that LLM judges on WebVoyager-style benchmarks significantly overstate agent success rates (74% vs actual 38%). They introduce the Universal Verifier approach using task-specific rubrics, screenshot evidence ranking, and error isolation to reduce false positives from ~50% to near zero. The talk covers why traditional LLM-as-judge verifiers create 'confident liars' and how their method achieves human-level agreement on evaluation.