How we cut Spark compute costs by 44% with agentic AI and Datadog Jobs Monitoring

To optimize a massive, high-cost Spark job, Datadog engineers developed an AI agent using Claude and Jobs Monitoring to bridge the gap between execution plans and application code. By employing a multi-agent architecture of generators and validators, the team successfully identified and filtered complex performance bottlenecks. These optimizations ultimately led to a 44% reduction in daily compute costs and a 60% decrease in job duration in their largest data center.