Dynamic shredding on Parquet JSON columns cut Logfire query times from 30s timeouts to under 1s

Logfire previously stored all span and resource attributes as single Utf8View JSON columns. Queries relied on Jiter parsing at read time. Dynamic shredding now extracts frequent keys into typed columns at ingest while retaining a residual JSON column for rare attributes. This change preserves the original semi-structured schema for users.

MotherDuck zone-map pruning benchmarks and Apache Parquet statistics show that columnar projection plus min/max skipping on extracted columns reduces scanned bytes by orders of magnitude. Early-access customers reported sub-second completion on previously 30-second queries. The 1000x factor appears on predicates hitting high-cardinality keys such as user_id or http.status_code.

Operationally, teams querying production traces daily now obtain interactive results without separate indexing pipelines. The technique generalizes to any Parquet workload mixing fixed and variable attributes, cutting both storage scan cost and analyst wait time. Rollout begins February 2026 across all Logfire tenants.

Pydantic's prior work on Jiter provided the parsing speed baseline; shredding adds the structural optimization that columnar engines require for semi-structured data at scale.