Lessons from Generating 12 Trillion Synthetic Tokens — Bogdan Gaza, DatologyAI

Why watch

Bogdan Gaza shares DatologyAI's engineering journey generating 12 trillion synthetic tokens for frontier LLM pre-training. This talk covers BeyondWeb (rephrasing-based synthetic data), migrating from Slurm/Kubernetes to unified Ray/KubeRay/vLLM on EKS HyperPod, and solving four critical bottlenecks: S3 metadata fetches (reduced from 9-11 days to 2 hours via batching), GPU failure recovery via partitioning/checkpointing, cross-cluster scheduling, and vLLM tuning achieving ~40% throughput gains. Essential viewing for ML engineers building large-scale training data pipelines.