• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

How To Add User Agents to Spider's Listing?

Jake Bunce

XenForo moderator
Staff member
#2
library/XenForo/Session.php

Code:
	/**
	 * Known robot user agent substrings
	 *
	 * @var array
	 */
	public static $knownRobots = array
	(
		'avsearch',
		'baiduspider',
		'bingbot',
		'crawler',
		'facebookexternalhit',
		'feedfetcher-google',
		'feedzirra',
		'googlebot',
		'kscrawler',
		'magpie-crawler',
		'nutch',
		'php/',
		'scooter',
		'scoutjet',
		'sogou web spider',
		'twitterbot',
		'xenforo signature generator',
		'yahoo! slurp',
		'yandexbot',
		'zend_http_client',
	);
It checks the user agent for those strings.
 

DRE

Well-known member
#3
library/XenForo/Session.php

Code:
/**
* Known robot user agent substrings
*
* @var array
*/
public static $knownRobots = array
(
'avsearch',
'baiduspider',
'bingbot',
'crawler',
'facebookexternalhit',
'feedfetcher-google',
'feedzirra',
'googlebot',
'kscrawler',
'magpie-crawler',
'nutch',
'php/',
'scooter',
'scoutjet',
'sogou web spider',
'twitterbot',
'xenforo signature generator',
'yahoo! slurp',
'yandexbot',
'zend_http_client',
);
It checks the user agent for those strings.
Thanks but how do I add the spiders from this list? http://www.wolfshead-solutions.com/spiders-list

Do I just type a new line and put the name of the user agent between quotation?
 

Jake Bunce

XenForo moderator
Staff member
#4
Code:
	/**
	 * Known robot user agent substrings
	 *
	 * @var array
	 */
	public static $knownRobots = array
	(
		'avsearch',
		'baiduspider',
		'bingbot',
		'crawler',
		'facebookexternalhit',
		'feedfetcher-google',
		'feedzirra',
		'googlebot',
		'kscrawler',
		'magpie-crawler',
		'nutch',
		'php/',
		'scooter',
		'scoutjet',
		'sogou web spider',
		'twitterbot',
		'xenforo signature generator',
		'yahoo! slurp',
		'yandexbot',
		'zend_http_client',
		'new',
		'user',
		'agents',
	);
 

DRE

Well-known member
#6
I'm sorry I'm a little slow Jake but I need clarification again after looking at this file of 620 spiders from Wolfhead solutions. In the excerpt below I see two names for one spider. Do I enter the name of the 'spider ident' line as the user agent or do I enter the 'name' as the user agent?

Code:
    <spider ident="BigmirSpider">
        <name>Bigmir</name>
        <info>http://www.bigmir.net/</info>
    </spider>
    <spider ident="bingbot">
        <name>Bing</name>
        <info>http://www.bing.com/bingbot.htm</info>
I'm using this example because you have bingbot as the name of the user agent in your example but in his file the name says Bing.
 

DRE

Well-known member
#8
You know what nevermind. I think the spider-ident line is the name of the user agent. I'mma go from there. Thanks!
 

mrGTB

Well-known member
#12
I've just added some spiders and then ran the "Health Check" in Admin CP and it fails because the file has changed. Might be a good idea to not include that file in health checking, or whatever so everything still passes OK if adding extra spiders many people will do?