How we measure data completeness at scale

To ensure reliability for both users and automated AI agents, Datadog developed a system that monitors the completeness of its massive ingestion pipelines by dividing complex workflows into measurable segments. By tracking "create" and "acknowledgment" events for unique payloads within specific time-buckets, the system can aggregate segment-level ratios to pinpoint exactly where and why data loss is occurring in real-time.