XF 1.3 GBot Unable To Fetch Robots.txt

[usAdultAds]

Site has been up a few days, maybe a week, gbot was coming here and
there, 4-5 pages was indexed, when I logged into wtools, it tells
me that is unable to fetch robots.txt so gbot will not be coming....

Those 4-5 pages are no longer indexed.
gbot has not been around for 3 days.

wtools robot.txt test says the robots.txt is Allowed.
so what I get from this is that the robots.txt is good.

Any ideas on why gbot can not fetch robots.txt?

I am not using my home page, so I am redirecting home to
http://www.domain.com/community

however, I am not sure why that would prevent
gbot from fetching the robots.txt

Thanks

 

[usAdultAds]

I got it to work, for some reason google could not find the robots.txt, even
though it was clearly there, and I am wondering if these redirects have
something to do with it. I am also wondering if these redirects are a good idea...

domain.com and www.domain.com redirect too
http://www.domain.com/community/

Then I have another redirect that redirects:
http://domain.com/community too
http://www.domain.com/community

none the less, I have a dozen spiders on my site now where
I pretty much had zero before, talk about opening the flood gates...