Web crawlers ignore blocking efforts

The news: News sites are getting increased referral traffic from AI searches, whether they’ve blocked web scrapers or not.

  • Forbes’ user traffic from Perplexity AI increased about 2,000% YoY in August, to 236,000 visitors from 10,800, per SimilarWeb. Perplexity’s referral traffic for The New York Times went up 733%.
  • Perplexity has a licensing deal with Forbes, while The New York Times continues to block its web crawler.

First Published on Oct 7, 2024