Lack of interest Spiders XML?

FredC

Well-known member
I looked but didnt see one, But then im famous for overlooking things.. So is there a Spiders XML in the XF package? and if so how would one go about adding spiders to the list?
I know Dream (i think it was dream) had a modified spider xml for vB with hundreds of spiders listed.. Id like something like that again.
 

xf_phantom

Well-known member
what's the status of this?
Are the robots hardcoded in xf 1.2?
is there a global database which sync feature or any other import feature?
 

Jeremy

Well-known member
Without seeing code, I know it was hardcoded into v1.1 somewhere. I know there was a thread about it not to long ago.

@xf_phantom: 1.1 has it in XenForo_Session::$knownRobots and XenForo_Session::isRobot().
 

Jake Bunce

XenForo moderator
Staff member
library/XenForo/Session.php

Code:
	/**
	 * Search engine domains (excluding TLD)
	 *
	 * @var array
	 */
	public static $searchDomains = array
	(
		'alltheweb',
		'altavista',
		'ask',
		'bing',
		'dogpile',
		'excite',
		'google',
		'lycos',
		'mamma',
		'metacrawler',
		'search',
		'webcrawler',
		'yahoo',
	);

	/**
	 * Known robot user agent substrings
	 *
	 * @var array
	 */
	public static $knownRobots = array
	(
		'avsearch',
		'baiduspider',
		'bingbot',
		'crawler',
		'facebookexternalhit',
		'feedfetcher-google',
		'feedzirra',
		'googlebot',
		'kscrawler',
		'magpie-crawler',
		'nutch',
		'php/',
		'scooter',
		'scoutjet',
		'sogou web spider',
		'twitterbot',
		'xenforo signature generator',
		'yahoo! slurp',
		'yandexbot',
		'zend_http_client',
	);
This can be changed with a file edit or an addon that changes these public static vars of XenForo_Session.
 

Mike

XenForo developer
Staff member
The code for the robots has changed, but it is roughly in the same location as what Jake has pointed out. The manner in which an add-on would modify it won't be the same as in 1.1.
 

Alfa1

Well-known member
Is the XF 1.2 spiderlist much expanded? The above is a very short list compared to the spiderlist that mosh made for vb or that bad behavior has.
 

xf_phantom

Well-known member
IMO this is already available
We can add to $_knownRobots and $_robotMap via event listeners

correction: we can't, because they're not static
 
Last edited:

xf_phantom

Well-known member
Not everyone is a developer.

I suspect the suggestion is to do with an easy way to create a list in the ACP.
ok:)

and i was also wrong, we can't just add them..:/
We would have to overwrite the session class instead of just adding it to the array, because he's not static
 
Top