50,000 scrapes per reader: AI bots drain publisher content

Thesis: The ratio of scrapes to pageviews has shifted from exchange to expropriation. Without licensing, high-quality information becomes a cost no one wants to bear.

Cloudflare published its "Attribution Business Insights" report on 1 July 2026. The data reveals a scale that is hard to ignore. Comparing bot fetches to real reader visits, the ratios ranged from 118:1 to nearly 50,000:1. In extreme cases, for every human who visited the site, bots performed almost 50,000 fetches. Cloudflare categorizes traffic into training, search, and agent types. Only search traffic benefits the publisher.

The market is reacting because voluntary cooperation has ended. Cloudflare introduced rules for "mixed" bots that perform multiple functions. From 15 September, these bots pass search traffic by default but block training and agent activity on ad-supported sites. The company called this the end of the free ride. Licensing deals are also emerging. Netflix reached an agreement with Penske Media, publisher of "Variety" and "Rolling Stone." Major players prefer paying for content over waiting for regulation.

User expectations are clear, quantified by the ARD/ZDF Medienstudie 2026. 59% of Germans aged 14 and over use AI tools occasionally, and 37% do so weekly, up from 22% a year earlier. Among 14- to 29-year-olds, 92% use AI tools, compared with 15% of those over 70. Additionally, 54% approach model outputs with scepticism and check for errors. 81% consider transparent labelling of training data and sources important. Audiences do not want free magic; they want to know the source of an answer.

This creates a conflict of interest. The same users who demand labels use tools that scraped content without a licence or payment. Publishers, who funded journalism through advertising, are losing revenue while supplying the material for models that act as intermediaries in information access. There is no symmetry: the information broker does not bear the cost of production yet profits from it.

The solution is not to close access but to price it. Three elements are necessary: a public register of bots and their functions, a standard licensing mechanism available to small publishers, and transparent source indication in model answers. Without the first, we do not know who is fetching. Without the second, only large newsrooms will secure deals. Without the third, readers cannot distinguish synthesis from fact. Cloudflare showed the data exists. Now it must serve as the basis for an agreement, not a complaint.