1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

[error] File does not exist: /public_html/robots.txt

Discussion in 'Server Configuration and Hosting' started by Saburov, Oct 5, 2016.

  1. Saburov

    Saburov Active Member

    My hosting cpanel's error log, there is two error log like this:

    [Wed Oct 05 13:29:15 2016] [error] [client 173.245.52.231] File does not exist: /home/xxx/public_html/www.muhasebecafe.com/robots.txt

    What does it mean? Should I create a blank robots.txt file in this folder or something else?
     
  2. Brogan

    Brogan XenForo Moderator Staff Member

    Saburov likes this.
  3. fly

    fly Active Member

    BTW, if you're wondering what it's for, the file describes certain aspects of your site to search engines. You can use it to tell them not to spider parts of your site, link them to a sitemap, and probably other things that I can't think of right now.
     
    Saburov likes this.
  4. Saburov

    Saburov Active Member

    Thanks.
     
  5. Diana

    Diana Active Member

    My site was recently under massive attack by spam bots. The host gave me a list of the bots to block in a robot.txt file. I didn't have a robot.txt file prior to the attacks on this new site, but I have had one on an old site. My question is should I use the file I have as is or add the list the host gave me to it?

    Here is what I have from the old site:

    User-agent: *
    Disallow: /forum/find-new/
    Disallow: /forum/account/
    Disallow: /forum/attachments/
    Disallow: /forum/goto/
    Disallow: /forum/posts/
    Disallow: /forum/login/
    Disallow: /forum/events/
    Disallow: /forum/profile/
    Disallow: /forum/calendar/
    Disallow: /forum/misc/
    Disallow: /forum/search/
    Disallow: /forum/members/
    Disallow: /forum/register/
    Disallow: /forum/online/
    Disallow: /forum/recent-activity/
    Disallow: /forum/lost-password/
    Disallow: /forum/admin.php
    Disallow: /forum/help/
    Allow: /
     

Share This Page