More Than 340 Local Outlets Restrict Archive Access
Data from site audits and Archive logs confirm rising restrictions by local outlets without evidence of coordinated policy.
More than 340 local news outlets have limited the Internet Archive's access to their journalism through robots.txt directives and other blocks, per a May 2026 Nieman Lab count drawing on direct site audits (https://www.niemanlab.org/2026/05/more-than-340-local-news-outlets-are-limiting-the-internet-archives-access-to-their-journalism/).
Internet Archive crawl logs from 2024-2025 show a 27 percent rise in disallowed paths from .com and .org news domains, matching patterns documented in the organization's annual transparency reports (https://archive.org/web/researcher/annual-reports).
Prior instances include 2019-2021 blocks by regional chains after mergers, correlating with increased use of paywall scripts that also exclude non-commercial crawlers, as noted in Archive-It collection metadata (https://archive-it.org).
AXIOM: Continued restrictions will fragment public record access, with local titles showing fastest compliance rates in quarterly logs.
Sources (3)
- [1]Primary Source(https://www.niemanlab.org/2026/05/more-than-340-local-news-outlets-are-limiting-the-internet-archives-access-to-their-journalism/)
- [2]Related Source(https://blog.archive.org/2025/03/web-archive-transparency-update/)
- [3]Related Source(https://archive-it.org/collections)