• Wikimedia Foundation foots bill for others’ AI training.
• Open-source projects and creators hit by similar bot traffic.
• Foundation plans to tighten access, while devs more proactive.

The Wikimedia project is the latest victim of the indiscriminate scraping of online information by bots from AI organisations.

The Wikimedia Foundation states that in the last 14 months, downloads of its content grew by 50%, a rise not emanating from human readers, but from “automated programs that scrape the Wikimedia Commons image catalog of openly licensed images to feed images to AI models.”

It also claims that 65% of the most resource-consuming traffic comprises of automated bots that bulk-access multiple pages, often targeting material that’s less commonly-requested by humans. The organisation pointed out that bots want to absorb as many sources of information as possible, unlike ‘normal’ users who tend to request many of the same commonly-accessed pages or resources. The latter are typically cached by the Foundation’s servers for immediate and low-cost re-transmission, whereas the bots’ preferences for all they can get means a retrieval from slower archives and a higher running cost for Wikimedia.

In addition to the publicly-accessible archives, bots are also reported as scraping key systems in the Foundation’s developer infrastructure, such as its code review platform and bug tracking systems.

Open-source projects get the same treatment

Several large open-source projects’ git repos have been similarly hit over the last 12 months, with some project maintainers identifying that the source of up to 85% of traffic to their resources emanates from avaricious bots designed to feed AI learning corpora. The founder and CEO of SourceHut, Drew DeVault, complained that he’s “spent anywhere from 20-100% of my time in any given week mitigating hyper-aggressive LLM [large language model] crawlers at scale.”

While the Wikimedia Foundation’s language is less colourful than DeVault’s (“Silicon Valley ripping off all the data they can get their hands on at everyone else’s expense… I am sick and tired of having all of these costs externalized directly into my [expletive deleted] face”), its Annual plan for 2025-2026 does contain a pledge to reduce the amount of traffic it will allocate to automated requests, such as those from AI bots.

“We aim to increase the percentage of automated traffic that is attributed to a known account, by requiring authentication and authorization based on tiered levels of access for high volume scraping and API use. This will help us to identify who is reusing content at scale, enabling us to protect our infrastructure and improve governance around fair use.”

The Foundation admits that to date it’s had limited ways to determine who is behind each bot instance, but intends to simplify its API frameworks to bring about what it calls “consistent pathways and governance” with the goal of reducing infrastructure costs.

Some open-source developers and project maintainers have been less delicate in their responses than the Wikimedia Foundation. Initially the work of a single developer, Anubis presents visitors to Git repositories (where open-source code is stored) with in-page JavaScript that creates a proof-of-work challenge. While individuals landing on a URL may have to wait less than a second while their browser solves the sha256-based digital puzzle in the background, automated bots will most likely be unable to pass, as they rarely have the ability to parse JavaScript.

“In case a scraper is dedicated enough to solve the challenge, Anubis lets them through because at that point they are functionally a browser,” writes Xe Iaso, who wrote Anubis in response to the massively-increased traffic to his Git infrastructure. Initially, Xe thought his Git servers were experiencing a distributed-denial-of-service attack, given the amount of traffic they were being asked to serve. He soon discovered, however, that it was rapacious AI bots in their thousands that were causing his server stats to go through the roof.

Since its release, the project has attracted dozens of additional contributors and has gone viral among the open-source community, as developers seek to protect their infrastructure running costs and resources, and content creators erect barriers to try to prevent rights infringements of their creative work.

Technical challenges to bot prevention

The difficult issue facing the Wikimedia Foundation and thousands of other owners of freely-available information repositories is that the code running AI bots can change the digital persona and method used to access information, in order to get round measures placed to ensure reasonable use (of resources and copyright material). In some cases, project maintainers and publishers have taken to draconian methods to block bots, including in one instance, temporarily blocking any inbound request from the entirety of Brazil.

The irony is that the automated systems trawling for freely-available information are often run by organisations that don’t have much time for the concept of freely-available information: Elon Musk, whose dislike of Wikipedia is well-publicised, once offered a $1 billion donation to the Foundation if it changed the site’s name to “Dickipedia”. As Seth Stern, director of advocacy at the Freedom of the Press Foundation told The Independent, “[Musk] is highly tolerant of inaccuracy, misinformation, criticism, even hate. But when the subject of the speech is himself, or his agenda, his objectives, and his companies, it’s quite the opposite.”

The Wikimedia foundation has to fight its corner to maintain its policy of peer review and fact-checking. It also has to defend itself from scraping systems that are often controlled by its strongest critics. To add insult to injury, a good percentage of third-party organisations’ AI training costs are being passed on to the Foundation and projects run on volunteers’ personally-funded resources, in the form of increased infrastructure and management bills.