[error] File does not exist: /public_html/robots.txt

[Saburov]

My hosting cpanel's error log, there is two error log like this:

[Wed Oct 05 13:29:15 2016] [error] [client 173.245.52.231] File does not exist: /home/xxx/public_html/www.muhasebecafe.com/robots.txt

What does it mean? Should I create a blank robots.txt file in this folder or something else?

 

[Paul B]

Manually create a file and upload it to the server.

You can use this as a template: https://xenforo.com/robots.txt

 

[fly]

BTW, if you're wondering what it's for, the file describes certain aspects of your site to search engines. You can use it to tell them not to spider parts of your site, link them to a sitemap, and probably other things that I can't think of right now.

 

[Saburov]

Thanks.

 

[Diana]

My site was recently under massive attack by spam bots. The host gave me a list of the bots to block in a robot.txt file. I didn't have a robot.txt file prior to the attacks on this new site, but I have had one on an old site. My question is should I use the file I have as is or add the list the host gave me to it?

Here is what I have from the old site:

User-agent: *
Disallow: /forum/find-new/
Disallow: /forum/account/
Disallow: /forum/attachments/
Disallow: /forum/goto/
Disallow: /forum/posts/
Disallow: /forum/login/
Disallow: /forum/events/
Disallow: /forum/profile/
Disallow: /forum/calendar/
Disallow: /forum/misc/
Disallow: /forum/search/
Disallow: /forum/members/
Disallow: /forum/register/
Disallow: /forum/online/
Disallow: /forum/recent-activity/
Disallow: /forum/lost-password/
Disallow: /forum/admin.php
Disallow: /forum/help/
Allow: /