XF 2.2 AppleBot - causing over 100,000 hits in 5 days - causing cpu limits to be hit

ak38

Active member
Hi guys,

I've got an issue with applebot relentlessly crawling my site causing some drain on the server cpu limits.

Is there a way to slow these down or is the only solution to block the applebot ?

Thanks for your help!
 
Use robots.txt:


Applebot robots.txt info:


~~~

Try a robots.txt with:

User-agent: Applebot
Crawl-delay: 1

That will ask Applebot to wait 1 second between each page they crawl. If you want them to wait longer, you can adjust the number 1 to another number.

Applebot may or may not abide the crawl-delay directive, but it doesn't cost you anything to try it.
 
Basically yes.
And i think since robots.txt is a rulebook for either all bots, or specific ones, if they are not following the rules you set, they should be banned,

I hope you did not set crawl delay for all to 20. If you did that, it might interfere with the bots that actually follow the rules such as google and bing.
 
Back
Top Bottom