Bot vs. Human
Copy linkBot (automated) vs. human HTTP requests distribution
Bot traffic sources
Copy linkBot traffic describes any non-human Internet traffic, and monitoring bot levels can help spot potential malicious activities
Verified Bots
Copy linkCloudflare maintains a list of "Verified" bots.
Verified bots, such as the ones from search engines, are usually transparent about who they are. Cloudflare manually approves well-behaved services that follow our guidelines.
Latest Bots blog posts
Copy linkPosts tagged with "Bots" from the Cloudflare blog
Cloudflare Radar now shows how often a given AI model sends traffic to a site relative to how often it crawls that site. This helps site owners make decisions about which AI bots to allow or block.
David Belson, Sam Rhea
Cloudflare is making it easier for publishers and content creators of all sizes to prevent their content from being scraped for AI training by managing robots.txt on their behalf.
Jin-Hee Lee, Dipunj Gupta, Brian Mitchell, Reid Tatoris, Henry Clausen
Pay per crawl is a new feature to allow content creators to charge AI crawlers for access to their content.
Will Allen, Simon Newton