understanding the behavior of google bots

[Diana]

I don't know much about crawlers or SEO, and I don't understand how google bots operate.

Watching the behavior of google bots on my forum it appears that they're stuck in some kind of netherland. All it shows is that they're "viewing unknown page." Once in awhile they view the "events calendar." I've never seen them view a thread, and there are quite a lot of them now.

Can someone help me understand this process better, and is there anything I can do to optimize how google crawls my forum?

btw, I do have a robots file.

User-agent: *
Disallow: /forum/find-new/
Disallow: /forum/forums/-/
Disallow: /forum/account/
Disallow: /forum/attachments/
Disallow: /forum/goto/
Disallow: /forum/posts/
Disallow: /forum/login/
Disallow: /forum/admin.php
Allow: /

 

[Diana]

Like so?

Disallow: /forum/style-chooser/

 

[dieketzer]

Like so?

Disallow: /forum/style-chooser/

no, it would be
Disallow: /forum/misc/
you can figure out the popup urls by opening them in a new tab.

my robots for what its worth:

User-agent: *
Disallow: /community/attachments/
Disallow: /community/misc/
Disallow: /community/help/
Disallow: /community/search/
Disallow: /community/members/
Disallow: /community/register/
Disallow: /community/login/
Disallow: /community/online/
Disallow: /community/lost-password/
Disallow: /community/recent-activity/
Disallow: /community/forums/mature-nsfw.12/
Disallow: /community/account/
Allow: /

 

[Diana]

Thanks. That's very helpful.

 

[Diana]

Just to make sure. Is my robots file in the right place and with the correct permissions?

forum/robots.txt

0644

Correct?

 

[Diana]

argh, google is still going to that "unknown page" which is the same style-chooser page as before.

How do I disallow that page? disallowing "misc" doesn't seem to do it.

this is the style chooser popup URL in a new tab:

forum/index.php?misc/style&redirect=%2Fforum%2Findex.php

That's where it goes, and stays (except with that strangely long URL).

 

[Brett Peters]

Just to make sure. Is my robots file in the right place

put robots.txt in http://www.example.com/robots.txt instead of http://www.example.com/forum/robots.txt

 

[Diana]

Thanks, I'll try that.

 

[Diana]

hmm still no change. googlebot is still going to the same URL.

I'll check again tomorrow.

Thanks everyone for your help

 

[dieketzer]

i believe google fetches robots.txt once every 24 hours. if you set up webmaster tools and check the robots section it will tell you when it was last fetched and show you a copy of the robots it is using.
the changes you make to robots are not going to be instantly reflected by google.

 

[Brett Peters]

I setup a bad bot trap on one of my sites a while back and did notice that googles bot took a few days to respect the new robots.txt file.

 

[Deepmartini]

Is it possible to allow googlebots inside of protected members area of the forum so they can index the content? Otherwise the content behind the register wall won't be indexed right? I have a private forum for members only but would like the SEO benefits of indexing the content.

 

[Jake Bunce]

Is it possible to allow googlebots inside of protected members area of the forum so they can index the content? Otherwise the content behind the register wall won't be indexed right? I have a private forum for members only but would like the SEO benefits of indexing the content.

See this thread:

http://xenforo.com/community/threads/letting-google-adsense-index-your-private-boards.22902/

But this is for the purpose of advertising, not search indexing. Restricted areas (not accessible to guests) will not show in search results.