Airbnb Scales Prometheus to Billion Series with OpenTelemetry and vmagent
Airbnb's documented billion-scale Prometheus pipeline using OpenTelemetry and vmagent reveals production cardinality and ingestion practices rarely shared, synthesizing lessons from Uber M3 and VictoriaMetrics deployments.
Airbnb Engineering disclosed a production metrics pipeline sustaining over one billion active Prometheus time series using OpenTelemetry collectors for instrumentation and vmagent for scraping, relabeling, and remote write. Primary source details vmagent's role in deduplication and batching to control cardinality at Airbnb's service scale (https://medium.com/airbnb-engineering/building-a-high-volume-metrics-pipeline-with-opentelemetry-and-vmagent-c714d6910b45).
Original coverage emphasized component selection but omitted explicit cardinality budgets, peak ingestion rates during booking surges, and exact OTEL processor configurations required for stability. Uber's M3 platform addressed analogous Prometheus scaling limits through a dedicated TSDB and aggregation tiers before remote storage (https://eng.uber.com/m3/). VictoriaMetrics vmagent documentation reports up to 10x lower memory usage versus Prometheus servers under high cardinality remote_write loads (https://docs.victoriametrics.com/vmagent/).
The Airbnb implementation fits a pattern of shifting from monolithic Prometheus to decoupled OTEL collection and lightweight agents, reducing per-node overhead while preserving Prometheus query compatibility. This disclosure supplies production-hardened reference architectures absent from most CNCF observability surveys and vendor benchmarks.
AXIOM: Airbnb's OTEL-plus-vmagent pipeline demonstrates that commodity open-source components can reliably ingest and forward a billion Prometheus series when cardinality controls and batching are production-tuned, a pattern likely to appear in future CNCF reference architectures.
Sources (3)
- [1]Building a High-Volume Metrics Pipeline with OpenTelemetry and vmagent(https://medium.com/airbnb-engineering/building-a-high-volume-metrics-pipeline-with-opentelemetry-and-vmagent-c714d6910b45)
- [2]M3: Uber’s Open Source, Large-scale Metrics Platform(https://eng.uber.com/m3/)
- [3]vmagent - VictoriaMetrics Documentation(https://docs.victoriametrics.com/vmagent/)