XF 2.2 AppleBot - causing over 100,000 hits in 5 days - causing cpu limits to be hit

[ak38]

Hi guys,

I've got an issue with applebot relentlessly crawling my site causing some drain on the server cpu limits.

Is there a way to slow these down or is the only solution to block the applebot ?

Thanks for your help!

 

[cwe]

Use robots.txt:

Robots.txt Introduction and Guide | Google Search Central | Documentation | Google for Developers

Robots.txt is used to manage crawler traffic. Explore this robots.txt introduction guide to learn what robot.txt files are and how to use them.

developers.google.com

Applebot robots.txt info:

About Applebot

Learn about Applebot, the web crawler for Apple.

support.apple.com

~~~

Try a robots.txt with:

User-agent: Applebot
Crawl-delay: 1

That will ask Applebot to wait 1 second between each page they crawl. If you want them to wait longer, you can adjust the number 1 to another number.

Applebot may or may not abide the crawl-delay directive, but it doesn't cost you anything to try it.

 

[ak38]

Thanks @cwe

I've set the delay to 20 and will check the stats in a few days, if this doesn't work is my only other option to block the bot?

 

[ebikes.com]

Basically yes.
And i think since robots.txt is a rulebook for either all bots, or specific ones, if they are not following the rules you set, they should be banned,

I hope you did not set crawl delay for all to 20. If you did that, it might interfere with the bots that actually follow the rules such as google and bing.