• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

XF 1.3 GBot Unable To Fetch Robots.txt

Site has been up a few days, maybe a week, gbot was coming here and
there, 4-5 pages was indexed, when I logged into wtools, it tells
me that is unable to fetch robots.txt so gbot will not be coming....

Those 4-5 pages are no longer indexed.
gbot has not been around for 3 days.

wtools robot.txt test says the robots.txt is Allowed.
so what I get from this is that the robots.txt is good.

Any ideas on why gbot can not fetch robots.txt?

I am not using my home page, so I am redirecting home to

however, I am not sure why that would prevent
gbot from fetching the robots.txt

I got it to work, for some reason google could not find the robots.txt, even
though it was clearly there, and I am wondering if these redirects have
something to do with it. I am also wondering if these redirects are a good idea...

domain.com and www.domain.com redirect too

Then I have another redirect that redirects:
http://domain.com/community too

none the less, I have a dozen spiders on my site now where
I pretty much had zero before, talk about opening the flood gates...