Encyclopedia Evalica / Observability / Data flywheel
Data flywheel
/'day.tuh 'fleye.weel/A feedback loop in which production data improves eval datasets, which improves the system, which in turn produces better production data. A strong flywheel turns "bugs" into durable eval coverage. (noun)
“The data flywheel accelerated once we turned thumbs-down traces into eval cases.”
Customer example
Retool built a data flywheel by pulling real production behaviors and edge cases back into their eval loop, using classification data to reprioritize product work continuously. Read more
Related Observability terms
- Active observability •
- AI observability •
- Alert / threshold •
- Dashboard •
- Deep search •
- Drift •
- Error rate •
- Feedback loop •
- Logs •
- Model drift •
- Online evaluation (production scoring) •
- P50 / P95 / P99 (Percentiles) •
- Sampling rate •
- Service Level Indicator (SLI) •
- Service Level Objective (SLO) •
- Time-to-first-token (TTFT) •
- Token usage / cost tracking •
- Topics
From the docs
Braintrust is the observability platform for agents in production. By actively applying intelligence to agent traces and automatically surfacing critical patterns, Braintrust helps teams at Notion, Stripe, Box, OpenAI, and Cloudflare ship quality agents at scale.
Start building