Teaching a 9B model to investigate production alerts

Datadog has developed a "training-data flywheel" to make large-scale agentic incident investigations economically viable by fine-tuning a smaller, specialized model (Qwen3.5-9B) using traces from a larger teacher model. This approach achieves 87% of the teacher model's recall while reducing investigation costs by approximately 20x. By continuously iterating with new production data, this method enables high-quality, automated change attribution to be scaled across a significantly larger volume of alerts.