A shocking new report reveals that AI search engines are devastating publisher traffic while aggressively scraping their content. According to data from content licensing platform TollBit, AI-powered search tools send 96% less referral traffic to news and blog sites compared to traditional Google search.
The report analyzed 160 websites, including national and local news outlets, over the last quarter of 2023. It found that AI companies like OpenAI, Perplexity, and Meta scraped these sites an average of 2 million times, with each page being scraped approximately seven times.
"We are seeing an influx of bots hammering these sites every time a user asks a question," said TollBit CEO Toshit Panigrahi. "The demand for publisher content is nontrivial."
The impact on publishers has been severe. Education technology company Chegg reported a devastating 49% traffic decline in January year-over-year after Google began including AI-generated summaries of their content. The damage has been so substantial that Chegg is now considering going private or seeking acquisition.
Publishers face multiple challenges beyond traffic loss. The intensive bot activity is driving up server costs, while AI search engines continue to scrape content even when explicitly blocked. TollBit found that Perplexity, for instance, continued accessing one publisher's site 500 times despite being blocked, suggesting the use of unidentified web crawlers.
Legal battles are mounting as publishers fight back. The New York Post and Dow Jones have sued Perplexity for alleged copyright infringement. A group of major publishers including Condé Nast, Vox, and The Atlantic recently filed a lawsuit against AI company Cohere over unauthorized content scraping.
Some publishers have opted to strike licensing deals with AI companies. The Associated Press, Axel Springer, and the Financial Times have established content agreements with OpenAI. Meanwhile, companies like TollBit are emerging with new models to help publishers monetize AI access to their content.
The situation appears likely to worsen as AI companies launch increasingly sophisticated research agents that autonomously crawl hundreds of sites. "AI does not read like humans do," noted Panigrahi. "Humans will click one or two links and move on. AI will read 10 to 20 links to get their answer."