1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

XF 1.3 GBot Unable To Fetch Robots.txt

Discussion in 'Troubleshooting and Problems' started by usAdultAds, May 2, 2015.

  1. usAdultAds

    usAdultAds Active Member

    Site has been up a few days, maybe a week, gbot was coming here and
    there, 4-5 pages was indexed, when I logged into wtools, it tells
    me that is unable to fetch robots.txt so gbot will not be coming....

    Those 4-5 pages are no longer indexed.
    gbot has not been around for 3 days.

    wtools robot.txt test says the robots.txt is Allowed.
    so what I get from this is that the robots.txt is good.

    Any ideas on why gbot can not fetch robots.txt?

    I am not using my home page, so I am redirecting home to

    however, I am not sure why that would prevent
    gbot from fetching the robots.txt

  2. usAdultAds

    usAdultAds Active Member

    I got it to work, for some reason google could not find the robots.txt, even
    though it was clearly there, and I am wondering if these redirects have
    something to do with it. I am also wondering if these redirects are a good idea...

    domain.com and www.domain.com redirect too

    Then I have another redirect that redirects:
    http://domain.com/community too

    none the less, I have a dozen spiders on my site now where
    I pretty much had zero before, talk about opening the flood gates...

Share This Page