Cloudflare Review: Googlebot Leads AI Crawler Traffic in 2025

Cloudflare’s 2025 Year in Review report reveals significant insights about AI crawler activity, global internet traffic, and security trends. According to the Cloudflare review findings, Googlebot, the web crawler responsible for Google’s search indexing, dominated the traffic from AI crawlers by a large margin, far surpassing other crawlers like PerplexityBot. The global internet traffic also grew by 19%, marking a noteworthy shift in the digital landscape.

Cloudflare Review: Googlebot's Dominance in Crawler Traffic

In the period of October and November 2025, Googlebot crawled 200 times more web pages than PerplexityBot. Here are the key statistics:

• Googlebot: 11.6% of unique pages crawled
• OpenAI’s GPTBot: 3.6% of pages
• PerplexityBot: Only 0.06% of pages
• Bingbot: 2.6% of pages
• Meta-ExternalAgent & ClaudeBot: Both at 2.4%

This data highlights Googlebot's dual-purpose function, crawling both for search indexing and AI training, creating a dilemma for web publishers who may face the risk of losing search visibility if they block Googlebot's AI training.

If you're a content or Web site owner, are you blocking AI or search crawlers in your robots.txt file?

See how bot blocking across the top 10,000 domains changed throughout 2025 in our new Year in Review: https://t.co/RGhRDbsfJ0 pic.twitter.com/SeKRRXVlOO

— Cloudflare Radar (@CloudflareRadar) December 15, 2025

Key Findings from Cloudflare’s Report

• AI Bots and Human Traffic: In 2025, AI bots (excluding Googlebot) accounted for 4.2% of HTML requests, while Googlebot itself made up 4.5%.
• User-Action Crawling Growth: AI crawlers visiting sites based on user queries grew 15 times from January to December 2025, reflecting the increased use of AI tools like ChatGPT in educational and workplace settings.
• AI Crawlers Blocked: AI crawlers such as GPTBot and ClaudeBot were the most frequently blocked through robots.txt files, with publishers often choosing to block AI bots while keeping Googlebot active for search purposes.
• Sector Targeted by Attacks: For the first time, organizations in civil society, including nonprofits and libraries, faced the highest number of cyberattacks.

Also Read: InstantProxies Review: Best for Private and High-Speed Proxies?

This year’s data underscores the growing influence of AI in web traffic and highlights the challenges faced by web administrators in balancing crawling and security. As AI platforms evolve, Cloudflare anticipates further changes in how bots interact with websites in 2026.

More News To Read: Gemini 2.5 Translate And Search Incorporated by Google

Disney Partners with OpenAI: Sora AI Disney Characters in 2026