alexm Active member Apr 2, 2024 #1 Just in case you didn't see this, in recent weeks suddenly... Added to robots.txt: Code: User-agent: anthropic-ai Disallow: / It seems to obey the rules with robots.txt at this moment in time, so that is one solution. Regards, Alex Last edited: Apr 3, 2024
Just in case you didn't see this, in recent weeks suddenly... Added to robots.txt: Code: User-agent: anthropic-ai Disallow: / It seems to obey the rules with robots.txt at this moment in time, so that is one solution. Regards, Alex
philmckrackon Active member Apr 3, 2024 #2 Also, if it stops obeying robots.txt. Add this to the .htaccess file. Code: BrowserMatchNoCase "anthropic-ai" bad_bot Order Deny,Allow Deny from env=bad_bot
Also, if it stops obeying robots.txt. Add this to the .htaccess file. Code: BrowserMatchNoCase "anthropic-ai" bad_bot Order Deny,Allow Deny from env=bad_bot
RandallC Well-known member May 4, 2024 #3 Not going to lie- they have been tearing my site up and I didnt know who it was until I installed the KnownBots addon. I even started to block Amazon AWS IP addresses.
Not going to lie- they have been tearing my site up and I didnt know who it was until I installed the KnownBots addon. I even started to block Amazon AWS IP addresses.
duderuud Well-known member May 4, 2024 #4 Wow, just blocked that useragent in Cloudflare and it already blocked 300 requests in a few minutes. Looks like it is a bot that gathers data for AI. I (as a website owner) have an opinion about that...
Wow, just blocked that useragent in Cloudflare and it already blocked 300 requests in a few minutes. Looks like it is a bot that gathers data for AI. I (as a website owner) have an opinion about that...