Cloudflare is developing a way to assess Verified bots by what they do after identification. In an October 16 announcement, the company said it is observing whether crawlers honor website preferences over time, including instructions in robots.txt and signals to reduce their request rate. The system is already being used internally; customer-facing trust controls and public scores are planned for a later stage.
Why a verified identity is not enough
A verified crawler can identify its operator and still create problems for a website. It might visit paths the publisher has excluded, or keep requesting pages at the same pace after the origin has asked it to slow down. Cloudflare's new approach separates the question of who operates a bot from the question of whether that bot consistently respects a site's instructions. Cloudflare says its Verified bots must identify themselves accurately, behave as described and avoid abuse.
The company plans to translate observed behavior into an explainable trust band, showing how consistently a bot follows site preferences and which signals contributed to that assessment. It says the band should reflect sustained patterns rather than a single request or an isolated mistake. This is a proposed measure of behavior on Cloudflare's network, not a universal judgment about every request the crawler makes elsewhere.
What Cloudflare will measure
The initial checks focus on two signals. First, Cloudflare compares a bot's observed visits with the site's published robots.txt preferences. Second, it looks at what the bot does after receiving an HTTP 429 Too Many Requests response. A crawler that pauses or slows down in response is behaving differently from one that continues at the same rate. Cloudflare says repeated disregard can lower the bot's trust band.
These signals address different problems. robots.txt expresses where a site prefers a crawler to go; a 429 tells a visitor that its current pace is too high. Neither signal by itself proves malicious intent. Cloudflare explicitly notes that a misconfiguration or software bug can also cause unwanted crawler behavior. That is why the company says it will look for patterns over time.
What site operators can use today
For now, Cloudflare says it is building the history needed to refine the system. It intends to make trust thresholds available to Cloudflare customers so they can choose how Verified bot traffic is handled, and later publish scores on Cloudflare Radar. Those future features should not be confused with controls already shipped. Site operators can currently use Cloudflare's Bot Management and Bot Preference Sync to handle automated traffic and align published preferences with configured policies.
The announcement points toward a practical change in bot governance: identity can establish accountability, while continued access can depend on observed conduct. Whether this improves publisher control will depend on how clearly the eventual trust bands explain their evidence, how quickly they reflect corrected bot behavior and how much choice sites have when applying them.