Methodological Critiques of Test-Negative Design Fuel Debate Over CDC Flu Vaccine Effectiveness Estimates

Critics, including NIH Director Jay Bhattacharya, have questioned the test-negative design (TND) underlying CDC estimates of seasonal influenza vaccine effectiveness (VE), arguing it is prone to biases that may distort results. The TND compares vaccination rates among patients testing positive for influenza versus those testing negative for influenza but presenting with acute respiratory illness, aiming to reduce confounding from healthcare-seeking behavior. However, peer-reviewed analyses highlight limitations, including potential collider bias, selection effects, and the need for careful adjustment for calendar time.[1][2]

A Brownstone Institute analysis of the 2022–2023 VISION network data (published in the Journal of Infectious Diseases) points to confounding from time-varying background infection risk. Vaccination rollout often aligns with rising incidence, leading vaccinated individuals to accumulate more person-time during lower-risk periods later in the season. In the VISION data, the share of vaccinated encounters was lower (47%) during the high-risk October–December period compared to unvaccinated (61%), potentially inflating VE estimates. The study reported moderate VE: 44% (95% CI 40–47%) against influenza A-associated ED/UC encounters and 35% (27–43%) against hospitalizations among adults.[3][4]

Additional concerns include immortal time bias from excluding events within 14 days of vaccination, which can artifactually favor the vaccinated group. Broader literature on TND for influenza VE discusses assumptions such as similar non-influenza ARI incidence by vaccination status and the requirement to adjust for calendar time to avoid bias. Simulations and meta-analyses have explored related issues like control group selection and waning protection.[5][6]

Bhattacharya has publicly criticized TND as methodologically flawed ('crap' and 'logistically ridiculous'), citing its exclusion of non-care-seekers and potential for distortion, in the context of both flu and COVID-19 VE studies. These critiques have gained traction amid institutional discussions, though TND remains widely used and defended in epidemiological literature for its practicality in surveillance settings. CDC continues to publish TND-based estimates from networks like VISION, reporting consistent moderate protection in recent seasons when vaccines are well-matched.[7]

Documented: Peer-reviewed discussions of TND biases and assumptions; official VISION VE estimates; Bhattacharya's public statements. Claimed: Specific magnitude of bias in the 2022–2023 VISION analysis remains unquantified in independent replication. Speculated: Net direction and size of bias without cohort-style matching on vaccination date.