I recently came across an article via Hacker News on Sean Conner’s The Boston Diaries blog titled “Bro, ban me at the IP level if you don’t like me! (August 21, 2025). From the title alone, I had a small hunch as a writing website administrator about what had prompted the article. Would my hunch be correct? I read the first paragraph to find out.
For reasons, I’m looking into web activity and so far, the top webbot this month is one identifying itself as “Thinkbot,” which may be related to this AI company but I can’t be sure. Here’s how it identifies itself: ‘Mozilla/5.0 (compatible; Thinkbot/0.5.8; +In_the_test_phase,_if_the_Thinkbot_brings_you_trouble,_please_block_its_IP_address._Thank_you.)’.
My hunch was correct!
I have had some cause lately to monitor my server logs. In recent months, we have been having occasional issues with web scraper traffic, which I suspect coincides with everyone trying to build a large language model. Every time I open the logs, I see “Thinkbot” with the same user-agent string reproduced in the article.
Like the author of the post, I do not know for certain what Thinkbot is or who is sending it. I share the author’s suspicions that it is an AI bot based on the fact that. insofar as I can tell, it crawls posts and archives on my site instead of looking for vulnerabilities like actively malicious scanners. Before endeavoring to block it, I conducted some cursory research to make sure it was not a crawler I might want, such as one working toward a new search engine. I recall having been persuaded by the limited information I stumbled upon that it was most likely an AI crawler, so I moved to block it.
However, I did not block it by blocking its IP address, which it suggests in its own user agent string. The reason I avoided this approach is noted in The Boston Diaries post:
Yeah, block its IP address. You mean the 74 unique addresses it used this month alone? Checking each IP address for the ASN it’s from shows the 74 address coming from 41 (41!) network blocks!
I did not count how many unique IP addresses Thinkbot has, but I did notice in my logs that it was rotating IP addresses. This does serve to make its recommendation that I block its IP address disingenuous. Blocking one IP address obviously would not block the others. Instead, I used Jeff Starr’s excellent BBQ Pro plugin (see the free version) for WordPress to block the “Thinkbot” user-agent, which does not deter Thinkbot from trying to crawl my site but does block it from finding what it looks for.
I did that weeks before coming across The Boston Diaries post and did not think much of Thinkbot again other than always seeing it whenever I open my server logs. The Boston Diaries was a bit more interested than I was and discovered the source of all of the IP addresses:
A further check showed that all the network blocks are owned by one organization—Tencent. I’m seriously thinking that the CCP encourage this with maybe the hope of externalizing the cost of the Great Firewall to the rest of the world. If China scrapes content, that’s fine as far as the CCP goes; If it’s blocked, that’s fine by the CCP too (I say, as I adjust my tin foil hat).
While the tin foil hat theory is possible, I suspect Thinkbot’s focus is more scraping websites to build a large language model or some similar end. However, that is not to say its scraping is not constrained by the laws and policies of the Chinese Communist Party. A follow-up post on Boston Diaries noted that at least one Hacker News poster had tried adding notes about Falun Gong and Tienanmen Square to his or her server banner. This stopped scanning attempts from Tencent AS but the attempts still came from Russia, Lithuania, and Iran. That is amusing, but it is good to be careful since we have documented how U.S. big tech companies have on occasion exported China’s “Great Firewall” to international audiences.
For whatever it is worth, Thinkbot’s crawling rate, while high enough to never escape my attention when I check my server logs, has not been so high that I would consider it a problem for site load reasons. For example, what I have tended to have occasional problems with is when a crawler or scraper makes a huge number of requests in rapid succession (this led me to block “Scrapy,” the user-agent for an open source web scraper, wholesale). Thinkbot’s requests, from when I have looked at them, seem spaced out enough to not cause performance problems for my site. I blocked it more because (1) it is all but telling me to block it but doing so with a disingenuous recommendation; and (2) because it is not being forthright about who or what is sending it and for what purpose, leading me to suspect in conjunction with what it is crawling that it is connected to some sort of large language model.
