Spider recognition

[Dream]

I worked a while back on the main spiders database for vBulletin, which is now being run by Mosh.

Does xenForo have a spider recognition system that uses a xml? If it has, I could talk to Mosh so the list he is maintaining also supports the xenForo xml format. It has 500 or so spiders.

 

[chousho]

Their software works pretty much like this:

Spider:

Not a Spider:

 

[Dream]

Sorry is this a trolling post? lol

 

[Floris]

I think Cho is just trying to have a bit of fun, but it's not the offtopic forum

When I go to 'online users' here, I do not see the IP, but I also dont' see search engine bots specified.

 

[Dream]

I see, must be something that will be looked into at a later date. Thanks Floris.

 

[James]

Feel free to make a suggestion, I'm sure a lot of forum owners would like to know how many bots have crawled their forum over a period of time!

 

[Brandon_R]

Feel free to make a suggestion, I'm sure a lot of forum owners would like to know how many bots have crawled their forum over a period of time!

I think this is more of an admin thing than a user thing as a user don't really care about spiders etc. With that being said, Google Analytics shows you how much spiders crawls the website and xenForo has Google Anal integration so i don't think searching each user's IP to determine if it's a spider or not is such a good idea as the feature is already there.

 

[James]

I think this is more of an admin thing than a user thing as a user don't really care about spiders etc. With that being said, Google Analytics shows you how much spiders crawls the website and xenForo has Google Anal integration so i don't think searching each user's IP to determine if it's a spider or not is such a good idea as the feature is already there.

Bad abbreviation

Yeah, this should be an admin feature rather than a user one. Users don't care about spiders, they just want more users.

Google Analytics has this yeah, would be awesome if we could access it via the XF ACP though!

 

[Brandon_R]

Bad abbreviation

Yeah, this should be an admin feature rather than a user one. Users don't care about spiders, they just want more users.

Google Analytics has this yeah, would be awesome if we could access it via the XF ACP though!

Im pretty sure The Analytics API has some kind of method to pull in the info as i've seen it done on another forum. Ill have to verify that though.

 

[Peggy]

Feel free to make a suggestion, I'm sure a lot of forum owners would like to know how many bots have crawled their forum over a period of time!

Yep, I always kept track of spiders on my forum.

 

[Sador]

Yeh, you've got to keep track of the spiders on your forum, or the forumchicks will start complaining about them.

It's one of the more basic functions of all forumsoftware, so I'd be surprised if XF wouldn't get it.

 

[DaKat]

Yep, I always kept track of spiders on my forum.

I like to keep an eye on which ones crawl my site, too.

 

[DSF]

I like to keep an eye on which ones crawl my site, too.

What for I need this information?
Help me

 

[Kuma]

Never understood why people were so worried about what spiders were browsing their site.

 

[Dream]

For me, it's knowing who is a human guest (lurker) and who's not.

 

[Kuma]

For me, it's knowing who is a human guest (lurker) and who's not.

Do you really spend all of your time watching who's online? If I know who the major SE's are, I really do not care about random tiny search engines.

It is easy to spoof a user-agent anyway.

 

[Floris]

I used to idle on who's online just to see where traffic would come from. I've learned over the years it doesn't benefit or improve my day

 

[KURTZ]

back in the vB 3.8.x i really loved this product made by Nexia: http://vbenhancer.com/showthread.php?t=4826

so, personally, i think which this is the better way to handle the bots onto our board ...

 

[Dream]

Just a question of knowing who are humans, nothing so complicated as to live inside the WOL page. But I won't die of suffocation if this feature isn't there.

 

[ijeffers]

Im pretty sure The Analytics API has some kind of method to pull in the info as i've seen it done on another forum. Ill have to verify that though.

This is possible, I'm using a wordpress plugin which does it at the moment. It displays the visitors graph, site usage, top pages, top referrers and top searches on the dashboard which is pretty hand to have.