Lack of interest Spiders XML?

[FredC]

I looked but didnt see one, But then im famous for overlooking things.. So is there a Spiders XML in the XF package? and if so how would one go about adding spiders to the list?
I know Dream (i think it was dream) had a modified spider xml for vB with hundreds of spiders listed.. Id like something like that again.

 

[Peggy]

I don't believe there is one, at least I didn't see it.

 

[James]

I don't think this is included ATM, but I requested a move to Suggestions.

 

[Dean]

I'll move it.

 

[xf_phantom]

what's the status of this?
Are the robots hardcoded in xf 1.2?
is there a global database which sync feature or any other import feature?

 

[Jeremy]

Without seeing code, I know it was hardcoded into v1.1 somewhere. I know there was a thread about it not to long ago.

@xf_phantom: 1.1 has it in XenForo_Session::$knownRobots and XenForo_Session::isRobot().

 

[Paul B]

I believe it's still a file/code change.

I've asked @Mike to confirm.

 

[Jake Bunce]

library/XenForo/Session.php

	/**
	 * Search engine domains (excluding TLD)
	 *
	 * @var array
	 */
	public static $searchDomains = array
	(
		'alltheweb',
		'altavista',
		'ask',
		'bing',
		'dogpile',
		'excite',
		'google',
		'lycos',
		'mamma',
		'metacrawler',
		'search',
		'webcrawler',
		'yahoo',
	);

	/**
	 * Known robot user agent substrings
	 *
	 * @var array
	 */
	public static $knownRobots = array
	(
		'avsearch',
		'baiduspider',
		'bingbot',
		'crawler',
		'facebookexternalhit',
		'feedfetcher-google',
		'feedzirra',
		'googlebot',
		'kscrawler',
		'magpie-crawler',
		'nutch',
		'php/',
		'scooter',
		'scoutjet',
		'sogou web spider',
		'twitterbot',
		'xenforo signature generator',
		'yahoo! slurp',
		'yandexbot',
		'zend_http_client',
	);

This can be changed with a file edit or an addon that changes these public static vars of XenForo_Session.

 

[Mike]

The code for the robots has changed, but it is roughly in the same location as what Jake has pointed out. The manner in which an add-on would modify it won't be the same as in 1.1.

 

[Jeremy]

Look at that, Mike is making it easy for us! Glad its changed for the better.

 

[Alpha1]

Is the XF 1.2 spiderlist much expanded? The above is a very short list compared to the spiderlist that mosh made for vb or that bad behavior has.

 

[xf_phantom]

IMO this is already available
We can add to $_knownRobots and $_robotMap via event listeners

correction: we can't, because they're not static

 

[Paul B]

Not everyone is a developer.

I suspect the suggestion is to do with an easy way to create a list in the ACP.

 

[xf_phantom]

Not everyone is a developer.

I suspect the suggestion is to do with an easy way to create a list in the ACP.

ok

and i was also wrong, we can't just add them..:/
We would have to overwrite the session class instead of just adding it to the array, because he's not static

 

[Mike]

We would have to overwrite the session class instead of just adding it to the array, because he's not static

Which you can do now...

 

[xf_phantom]

Which you can do now...

yes, and it's still my favorite^^