How we built a real-world evaluation platform for autonomous SRE agents at scale

To combat subtle performance regressions in Datadog's Bits AI SRE agent, the team developed a custom, replayable evaluation platform that simulates diverse production incidents using historical telemetry snapshots. This system scales by automatically converting real user feedback into new test labels, enabling consistent, large-scale measurement of the agent's investigative accuracy.