1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Lack of Interest Spiders XML?

Discussion in 'Closed Suggestions' started by FredC, Jan 30, 2011.

  1. FredC

    FredC Well-Known Member

    I looked but didnt see one, But then im famous for overlooking things.. So is there a Spiders XML in the XF package? and if so how would one go about adding spiders to the list?
    I know Dream (i think it was dream) had a modified spider xml for vB with hundreds of spiders listed.. Id like something like that again.
    James likes this.
  2. Peggy

    Peggy Well-Known Member

    I don't believe there is one, at least I didn't see it.
    FredC likes this.
  3. James

    James Well-Known Member

    I don't think this is included ATM, but I requested a move to Suggestions.
  4. Dean

    Dean Well-Known Member

    I'll move it.
  5. xf_phantom

    xf_phantom Well-Known Member

    what's the status of this?
    Are the robots hardcoded in xf 1.2?
    is there a global database which sync feature or any other import feature?
  6. Jeremy

    Jeremy XenForo Moderator Staff Member

    Without seeing code, I know it was hardcoded into v1.1 somewhere. I know there was a thread about it not to long ago.

    @xf_phantom: 1.1 has it in XenForo_Session::$knownRobots and XenForo_Session::isRobot().
  7. Brogan

    Brogan XenForo Moderator Staff Member

    I believe it's still a file/code change.

    I've asked @Mike to confirm.
  8. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member


    	 * Search engine domains (excluding TLD)
    	 * @var array
    	public static $searchDomains = array
    	 * Known robot user agent substrings
    	 * @var array
    	public static $knownRobots = array
    		'sogou web spider',
    		'xenforo signature generator',
    		'yahoo! slurp',
    This can be changed with a file edit or an addon that changes these public static vars of XenForo_Session.
  9. Mike

    Mike XenForo Developer Staff Member

    The code for the robots has changed, but it is roughly in the same location as what Jake has pointed out. The manner in which an add-on would modify it won't be the same as in 1.1.
    Brogan and Jeremy like this.
  10. Jeremy

    Jeremy XenForo Moderator Staff Member

    Look at that, Mike is making it easy for us! Glad its changed for the better. (y)
  11. Alfa1

    Alfa1 Well-Known Member

    Is the XF 1.2 spiderlist much expanded? The above is a very short list compared to the spiderlist that mosh made for vb or that bad behavior has.
  12. xf_phantom

    xf_phantom Well-Known Member

    IMO this is already available
    We can add to $_knownRobots and $_robotMap via event listeners

    correction: we can't, because they're not static
    Last edited: Jun 20, 2013
  13. Brogan

    Brogan XenForo Moderator Staff Member

    Not everyone is a developer.

    I suspect the suggestion is to do with an easy way to create a list in the ACP.
    xf_phantom likes this.
  14. xf_phantom

    xf_phantom Well-Known Member


    and i was also wrong, we can't just add them..:/
    We would have to overwrite the session class instead of just adding it to the array, because he's not static
  15. Mike

    Mike XenForo Developer Staff Member

    Which you can do now...
  16. xf_phantom

    xf_phantom Well-Known Member

Share This Page