An AI Research Agent That Runs Your Experiments — Tim Sweeney, Weights & Biases

Why watch

Tim Sweeney demonstrates ARIA, Weights & Biases' AI agent that autonomously runs and monitors ML experiments across GPU clusters. The talk shows real-time experiment tracking on Karpathy's autoresearch project, with ARIA achieving near-human performance (73% vs 72% baseline) in automated evaluation. Key topics include building agent observability with Weave, using LLM judges to detect user frustration, YAML-defined task unit tests, and treating evals as a CI pipeline for AI systems.