Bad Bots are my #1 Frog Blog readers

Not true, but not that far off!

Well, isn't this another feel-good ego boost?! Only 59% of my Frog Blog visitors are human. The rest of my traffic is bots... and all of them bad ones. Bots and cowardly trolls are one of my biggest reasons for not permitting comments on Frog Blog. Readers are free to respond to the FA page that day or email me privately, if they wish. 
 

I asked AI about the difference:

Good bots are automated software programs designed to perform helpful, beneficial tasks for users or website owners, such as indexing content for search engines or providing customer support.  They typically operate transparently, identify themselves clearly, and respect website rules like the robots.txt file. 

Bad bots, in contrast, are automated programs created with malicious intent to exploit systems, steal data, or disrupt online services. 

They often disguise their identity, ignore website rules, and engage in harmful activities like spamming, credential stuffing, or launching DDoS attacks.

The key distinctions between the two include:

Intent: Good bots aim to enhance user experience or efficiency, 

while bad bots seek to manipulate, deceive, or cause harm.

Behavior: Good bots follow ethical guidelines and rate limits, 

whereas bad bots often scrape data, flood servers, or impersonate users.

Impact: Good bots improve discoverability and automate routine processes, while bad bots distort analytics, drain resources, and compromise security.

How do good bots interact with robots.txt?

Good bots follow the Robots Exclusion Protocol 

(standardized as RFC 9309) and interact with 

robots.txt in a few consistent ways:

Fetch first, crawl second. A well-behaved bot requests 

the robots.txt file at the site's root before crawling 

any other page. If the file doesn't exist or contains no 

matching directives, the bot proceeds to crawl freely. 

Match their user-agent. The bot looks for a User-agent 

block that matches its own identifier (e.g., Googlebot, 

GPTBot, ClaudeBot). If multiple blocks match, it 

follows the most specific one.

Obey Allow and Disallow directives. If a matching 

block contains Disallow: /private/, the bot will not 

request any URL under that path. If it contains 

Allow: /, the bot is free to crawl.

Voluntary compliance. The protocol is purely 

advisory — there's no technical enforcement.  Good 

bots comply as a matter of policy and reputation. 

Major examples include Googlebot, Bingbot, GPTBot, 

ClaudeBot, and CCBot. 

Honest identification. Good bots use their real, 

published user-agent strings rather than spoofing 

browser identifiers, making them easy to verify 

and target in robots.txt rules.

A practical caveat: even among "good" bots, behavior 

isn't 100% uniform. A 2026 test of 12 AI assistants 

found only 3 (ChatGPT, Claude, Perplexity) reliably 

honored Disallow rules for real-time fetching, 

while the rest ignored them. So robots.txt is best 

understood as a polite request to well-behaved 

crawlers, not an access-control mechanism.