Spider recognition

[Dream]

I worked a while back on the main spiders database for vBulletin, which is now being run by Mosh.

Does xenForo have a spider recognition system that uses a xml? If it has, I could talk to Mosh so the list he is maintaining also supports the xenForo xml format. It has 500 or so spiders.

 

[Deepmartini]

Does Xenforo allow google spiders into the thread content if it is blocked for members only?

I am curious if something like Google first click criteria can be used. That way more content can be indexed into the search engine and help with SEO? I searched and found this thread but can't find anything specific to allowing access inside for spiders and registered members only.

How does Xenforo deal with spidering issues and being blocked?

 

[Panupat]

When I look at my own site, I see at least 2-3 guests with google ip all the time. I'm not sure if I can grant them any specific permission tho.

 

[James]

How does Xenforo deal with spidering issues and being blocked?

It doesn't. Whatever guests have access to, that's what spiders have access to.

 

[Deepmartini]

Is there a way to let google spiders in so that they can index that content, but keep it members only?

 

[Paul B]

No.

It's also classed as cloaking and against Google's T&C's.

 

[Deepmartini]

What about leaving everything open but use Google's first click free preview. That would solve the issue wouldn't it? Is this functionality offered in Xenforo anywhere?

http://www.google.com/support/webmasters/bin/answer.py?answer=74536