JetBrains Air Context RAG Applies AST Parsing to Code Files Exceeding 1000 Lines

JetBrains implemented the pipeline after prototype RAG failed on repositories with thousands of files and agent-generated code. The first stage converts raw files into scoped units using AST traversal rather than line or token boundaries. This step directly addresses the failure of keyword search when agents seek concepts such as session token refresh that lack literal string matches. Production measurements showed entire files often exceed embedding model context while individual methods retain low semantic relatedness to neighboring code. Chunk sizes were therefore constrained to preserve retrieval precision; larger blocks diluted cosine similarity scores. The approach aligns with documented limitations in CodeSearchNet benchmarks where naive splitting reduced top-1 recall by 18-22 percent. Operationally the change shortens agent context assembly time and raises the probability that retrieved evidence is citable. Remaining stages vectorize the chunks for semantic lookup and integrate results into agent prompts. Subsequent posts in the series will detail storage and retrieval layers.