Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

Why watch

A hands-on workshop demonstrating how to build robust evaluation systems for AI agents in high-stakes domains like banking. Annabell and Doneyli from Langfuse walk through the complete AI engineering loop: implementing tracing with PII masking, constructing evaluation datasets, converting domain expert feedback into failure modes, configuring LLM-as-judge and code evaluators, and setting up CI that automatically blocks prompt changes when eval scores drop. Ideal for AI engineers building production AI systems where errors have real consequences.