AI bots drove up to 80% of traffic to university library systems in 2024 logs

Libraries at research universities detected coordinated scraping campaigns targeting journal repositories, thesis archives and licensed databases. Traffic logs showed persistent connections from infrastructure linked to major model developers, with session patterns matching known crawler signatures rather than human users. Institutions responded by tightening robots.txt rules and IP blocks, yet access for legitimate remote scholars degraded as thresholds triggered on shared networks.

Data from IT departments indicated bot traffic fractions between 40% and 80% of total catalog queries, correlating with spikes after public releases of new model versions. These volumes exceeded prior peaks from commercial search engines by factors of three to five, according to internal monitoring shared with consortia. The pattern matches earlier academic observations of bulk harvesting preceding model training runs documented in arXiv:2307.04669.

Operational impact includes accelerated moves toward authenticated proxy layers and usage quotas that previously applied only to commercial users. Libraries now face trade-offs between open-access mandates and license compliance, with some consortia exploring federated bot-detection services. Absent standardized crawler identification headers, further restrictions on unauthenticated queries are projected within the next two quarters.