XF 2.1 Google Bots are Slowing Down the Server

Recep Baltaş · Mar 18, 2020

I have no idea why but lately, Google Bots are peaking the server load. I came to a point where I challenge all the traffic form US to avoid the problem. We serve 200K+ daily unique users from Turkey but the moment I allow Google Bots, server load goes crazy.

What could be the issue and how can ve avoid it?

Our forum: https://www.technopat.net/sosyal/

Thanks.

Kacee · Mar 18, 2020

I believe you can tell Google to calm the heck down in Google Search Console.

Recep Baltaş · Mar 18, 2020

Kacee said:
I believe you can tell Google to calm the heck down in Google Search Console.

Yes, there is an option for that but it's not a long term solution. Although we'll upgrade our server soon, I'd like to have an actual solution.

Masetrix · Mar 18, 2020

Recep Baltaş said:
I have no idea why but lately, Google Bots are peaking the server load. I came to a point where I challenge all the traffic form US to avoid the problem. We serve 200K+ daily unique users from Turkey but the moment I allow Google Bots, server load goes crazy.

What could be the issue and how can ve avoid it?

Our forum: https://www.technopat.net/sosyal/

Thanks.

robots.txt...

Code:

User-agent: Googlebot
Disallow: /login/
Disallow: /admin.php
Disallow: /install/
Disallow: /includes/
Allow: /
Crawl-delay: 60
Sitemap: https://www.[...].tld/sitemap.xml

Robots.txt - What is the proper format for a Crawl Delay for multiple user agents?

Below is a sample robots.txt file to Allow multiple user agents with multiple crawl delays for each user agent. The Crawl-delay values are for illustration purposes and will be different in a real ...

stackoverflow.com

HtH

djbaxter · Mar 18, 2020

This was actually deprecated in 2019:

Google To Drop Support For crawl-delay, nofollow, and noindex in robots.txt

Google posted this morning that they are going to stop unofficially supporting the noindex, nofollow and crawl-delay directives within robots.txt files. Google has been saying this for years actually and hinted this was coming really soon and now it is here.

www.seroundtable.com