1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How To Add User Agents to Spider's Listing?

Discussion in 'Styling and Customization Questions' started by DRE, Mar 10, 2012.

  1. DRE

    DRE Well-Known Member

  2. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    library/XenForo/Session.php

    Code:
    	/**
    	 * Known robot user agent substrings
    	 *
    	 * @var array
    	 */
    	public static $knownRobots = array
    	(
    		'avsearch',
    		'baiduspider',
    		'bingbot',
    		'crawler',
    		'facebookexternalhit',
    		'feedfetcher-google',
    		'feedzirra',
    		'googlebot',
    		'kscrawler',
    		'magpie-crawler',
    		'nutch',
    		'php/',
    		'scooter',
    		'scoutjet',
    		'sogou web spider',
    		'twitterbot',
    		'xenforo signature generator',
    		'yahoo! slurp',
    		'yandexbot',
    		'zend_http_client',
    	);
    
    It checks the user agent for those strings.
     
    mrGTB and Blue like this.
  3. DRE

    DRE Well-Known Member

    Thanks but how do I add the spiders from this list? http://www.wolfshead-solutions.com/spiders-list

    Do I just type a new line and put the name of the user agent between quotation?
     
  4. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    Code:
    	/**
    	 * Known robot user agent substrings
    	 *
    	 * @var array
    	 */
    	public static $knownRobots = array
    	(
    		'avsearch',
    		'baiduspider',
    		'bingbot',
    		'crawler',
    		'facebookexternalhit',
    		'feedfetcher-google',
    		'feedzirra',
    		'googlebot',
    		'kscrawler',
    		'magpie-crawler',
    		'nutch',
    		'php/',
    		'scooter',
    		'scoutjet',
    		'sogou web spider',
    		'twitterbot',
    		'xenforo signature generator',
    		'yahoo! slurp',
    		'yandexbot',
    		'zend_http_client',
    		'new',
    		'user',
    		'agents',
    	);
    
     
    The8thLegion and Jarod like this.
  5. DRE

    DRE Well-Known Member

    lol thanks
     
  6. DRE

    DRE Well-Known Member

    I'm sorry I'm a little slow Jake but I need clarification again after looking at this file of 620 spiders from Wolfhead solutions. In the excerpt below I see two names for one spider. Do I enter the name of the 'spider ident' line as the user agent or do I enter the 'name' as the user agent?

    Code:
        <spider ident="BigmirSpider">
            <name>Bigmir</name>
            <info>http://www.bigmir.net/</info>
        </spider>
        <spider ident="bingbot">
            <name>Bing</name>
            <info>http://www.bing.com/bingbot.htm</info>
    
    I'm using this example because you have bingbot as the name of the user agent in your example but in his file the name says Bing.
     
  7. Brogan

    Brogan XenForo Moderator Staff Member

    Use the ident.
     
  8. DRE

    DRE Well-Known Member

    You know what nevermind. I think the spider-ident line is the name of the user agent. I'mma go from there. Thanks!
     
  9. DRE

    DRE Well-Known Member

    lol just figured it out, I'm slow. Thanks!
     
  10. mrGTB

    mrGTB Well-Known Member

    Does adding them in large numbers greatly effect your server performance, or does it make little difference?
     
  11. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    Little.
     
    The8thLegion likes this.
  12. mrGTB

    mrGTB Well-Known Member

    I've just added some spiders and then ran the "Health Check" in Admin CP and it fails because the file has changed. Might be a good idea to not include that file in health checking, or whatever so everything still passes OK if adding extra spiders many people will do?
     

Share This Page