How To Add User Agents to Spider's Listing?

Jake Bunce

XenForo moderator
Staff member
library/XenForo/Session.php

Code:
	/**
	 * Known robot user agent substrings
	 *
	 * @var array
	 */
	public static $knownRobots = array
	(
		'avsearch',
		'baiduspider',
		'bingbot',
		'crawler',
		'facebookexternalhit',
		'feedfetcher-google',
		'feedzirra',
		'googlebot',
		'kscrawler',
		'magpie-crawler',
		'nutch',
		'php/',
		'scooter',
		'scoutjet',
		'sogou web spider',
		'twitterbot',
		'xenforo signature generator',
		'yahoo! slurp',
		'yandexbot',
		'zend_http_client',
	);
It checks the user agent for those strings.
 

DRE

Well-known member
library/XenForo/Session.php

Code:
/**
* Known robot user agent substrings
*
* @var array
*/
public static $knownRobots = array
(
'avsearch',
'baiduspider',
'bingbot',
'crawler',
'facebookexternalhit',
'feedfetcher-google',
'feedzirra',
'googlebot',
'kscrawler',
'magpie-crawler',
'nutch',
'php/',
'scooter',
'scoutjet',
'sogou web spider',
'twitterbot',
'xenforo signature generator',
'yahoo! slurp',
'yandexbot',
'zend_http_client',
);
It checks the user agent for those strings.
Thanks but how do I add the spiders from this list? http://www.wolfshead-solutions.com/spiders-list

Do I just type a new line and put the name of the user agent between quotation?
 

Jake Bunce

XenForo moderator
Staff member
Rich (BB code):
	/**
	 * Known robot user agent substrings
	 *
	 * @var array
	 */
	public static $knownRobots = array
	(
		'avsearch',
		'baiduspider',
		'bingbot',
		'crawler',
		'facebookexternalhit',
		'feedfetcher-google',
		'feedzirra',
		'googlebot',
		'kscrawler',
		'magpie-crawler',
		'nutch',
		'php/',
		'scooter',
		'scoutjet',
		'sogou web spider',
		'twitterbot',
		'xenforo signature generator',
		'yahoo! slurp',
		'yandexbot',
		'zend_http_client',
		'new',
		'user',
		'agents',
	);
 

DRE

Well-known member
I'm sorry I'm a little slow Jake but I need clarification again after looking at this file of 620 spiders from Wolfhead solutions. In the excerpt below I see two names for one spider. Do I enter the name of the 'spider ident' line as the user agent or do I enter the 'name' as the user agent?

Code:
    <spider ident="BigmirSpider">
        <name>Bigmir</name>
        <info>http://www.bigmir.net/</info>
    </spider>
    <spider ident="bingbot">
        <name>Bing</name>
        <info>http://www.bing.com/bingbot.htm</info>
I'm using this example because you have bingbot as the name of the user agent in your example but in his file the name says Bing.
 

DRE

Well-known member
You know what nevermind. I think the spider-ident line is the name of the user agent. I'mma go from there. Thanks!
 

mrGTB

Well-known member
Does adding them in large numbers greatly effect your server performance, or does it make little difference?
 

mrGTB

Well-known member
I've just added some spiders and then ran the "Health Check" in Admin CP and it fails because the file has changed. Might be a good idea to not include that file in health checking, or whatever so everything still passes OK if adding extra spiders many people will do?
 
Top