• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

Lack of interest Spiders XML?

FredC

Well-known member
#1
I looked but didnt see one, But then im famous for overlooking things.. So is there a Spiders XML in the XF package? and if so how would one go about adding spiders to the list?
I know Dream (i think it was dream) had a modified spider xml for vB with hundreds of spiders listed.. Id like something like that again.
 

xf_phantom

Well-known member
#5
what's the status of this?
Are the robots hardcoded in xf 1.2?
is there a global database which sync feature or any other import feature?
 

Jeremy

Well-known member
#6
Without seeing code, I know it was hardcoded into v1.1 somewhere. I know there was a thread about it not to long ago.

@xf_phantom: 1.1 has it in XenForo_Session::$knownRobots and XenForo_Session::isRobot().
 

Jake Bunce

XenForo moderator
Staff member
#8
library/XenForo/Session.php

Code:
	/**
	 * Search engine domains (excluding TLD)
	 *
	 * @var array
	 */
	public static $searchDomains = array
	(
		'alltheweb',
		'altavista',
		'ask',
		'bing',
		'dogpile',
		'excite',
		'google',
		'lycos',
		'mamma',
		'metacrawler',
		'search',
		'webcrawler',
		'yahoo',
	);

	/**
	 * Known robot user agent substrings
	 *
	 * @var array
	 */
	public static $knownRobots = array
	(
		'avsearch',
		'baiduspider',
		'bingbot',
		'crawler',
		'facebookexternalhit',
		'feedfetcher-google',
		'feedzirra',
		'googlebot',
		'kscrawler',
		'magpie-crawler',
		'nutch',
		'php/',
		'scooter',
		'scoutjet',
		'sogou web spider',
		'twitterbot',
		'xenforo signature generator',
		'yahoo! slurp',
		'yandexbot',
		'zend_http_client',
	);
This can be changed with a file edit or an addon that changes these public static vars of XenForo_Session.
 

Mike

XenForo developer
Staff member
#9
The code for the robots has changed, but it is roughly in the same location as what Jake has pointed out. The manner in which an add-on would modify it won't be the same as in 1.1.
 

Alfa1

Well-known member
#11
Is the XF 1.2 spiderlist much expanded? The above is a very short list compared to the spiderlist that mosh made for vb or that bad behavior has.
 

xf_phantom

Well-known member
#12
IMO this is already available
We can add to $_knownRobots and $_robotMap via event listeners

correction: we can't, because they're not static
 
Last edited:

xf_phantom

Well-known member
#14
Not everyone is a developer.

I suspect the suggestion is to do with an easy way to create a list in the ACP.
ok:)

and i was also wrong, we can't just add them..:/
We would have to overwrite the session class instead of just adding it to the array, because he's not static