Google AI Web Scraping Harm
Google's unified web crawler serving dual purposes for both search indexing and AI training creates an untenable choice for publishers between losing traffic or losing search visibility.
Too little corroboration in the last 3 days to call a trend (8 articles). Watching for it to gain traction.
"As AI continues to kill traffic to websites, Google on Thursday threw a bone to those publishers negatively impacted by the change."
"Users are spending one to nine more minutes in AI Mode than on traditional Google searches, according to three studies from researchers tallied by The New York Times. One October study from Growth Memo, a newsletter focused on search and marketing, found that in about 75% of sessions, users never left AI Mode for the web."
"For twenty years, the deal between Google and the web was simple. Publishers let Google crawl their pages. Google sent readers back in return. That bargain is breaking."
"Google only uses one singular web crawler to 'both index sites and train its AI models.' That means if publishers opt out of appearing in Google's AI Overview summaries, they also opt out of being indexed in Google's search results."
"At the start of June 2026, the Competition and Markets Authority regulator announced that it would force Google to provide a way for publishers to remove their results from Google's AI Overview without being deindexed. In response, Google announced a trial that would enable publishers in the UK to opt out."
"Gemini's integration into Google's search function has led to a "human traffic decline" of almost "40% in less than one year." Google's AI is scraping information from websites, regurgitating it within Gemini's "AI Overview," and not providing the websites it steals said content from with a page view."
"Google has had a "unique role" in contributing to this decline, as the search engine accounts "for approximately 88% of referral traffic." Websites have a choice of either allowing Google to use the information scraped from their website in AI Overview, which doesn't provide said websites with a page view, or they opt out of appearing in Google's search results altogether."
""52% of crawler requests are now for AI training as of June 2026, up from 22% in Spring 2025." Google only uses one singular web crawler to "both index sites and train its AI models.""