1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Robots 1.5

Disallows robots from indexing your site unless explicitly allowed.

  1. AndyB

    AndyB Well-Known Member

    AndyB submitted a new resource:

    Robots - Disallows robots from indexing your site unless explicitly allowed.

    Read more about this resource...
  2. AndyB

    AndyB Well-Known Member

  3. AndyB

    AndyB Well-Known Member

  4. surfinsaxman

    surfinsaxman Active Member

    I installed the add-on, but where do I see the robots page?
  5. surfinsaxman

    surfinsaxman Active Member

    ok. found it
  6. AndyB

    AndyB Well-Known Member

  7. AzzidReign

    AzzidReign Well-Known Member

    Weird question, do you have to define the bots in order for them to be blocked if they aren't on the allowed list?

    Also, I've been noticing people talking about using search engines that value privacy and one of them I keep hearing about is DuckDuckGo. I've added it to my list but anyone reading this, I figured I would put that out there for them:
  8. valdet

    valdet Active Member

    Andy, this may be outside the scope of your product, but can you add an option, where admins/mod can disable robots on individual forums or threads.

    Also an option to append noindex and nofollow tags in <head> would be very useful.
  9. AndyB

    AndyB Well-Known Member

  10. BentDreams

    BentDreams Active Member

    @AndyB I have the same question as @valdet ... Is there a way to prevent the robots from crawling a specific category or sub-forum?

    We have recently made a private area for founding members and they are asking about the content coming up in search engines. So far I haven't found anything that addressed just a partial block of robots for certain areas of a site.

  11. BentDreams

    BentDreams Active Member

    @AndyB I installed this addon and am getting an error in my server log which will be below:

    Currently I have the following listed in "Robot Names":
    360spider,ahrefsbot,alexa,ask jeeves,archive.org_bot,baiduspider,bingbot,blexbot,CrystalSemanticsBot,facebookexternalhit,google-adsense,googlebot,magpie-crawler,linkdexbot,mj12bot,msnbot,NewsGatorOnline,orangebot,proximic,scoutjet,sogou web spider,yahoo! slurp,yandex

    And the following in "Allowed Robots":
    bingbot,googlebot,yahoo! slurp,DuckDuckBot,yahoo!-slurp,slurp,msnbot,google-adsense,teoma,robozilla,scoutjet

    This is the error from my server logs, note that "Roguexx" is a registered member of my site, not a bot, but all the error logs list him as the "Generated By:" user.

    Error Info
    Zend_Db_Statement_Mysqli_Exception: Mysqli statement execute error : Data too long for column 'ip' at row 1 - library/Zend/Db/Statement/Mysqli.php:214
    Generated By: Roguexx, 7 minutes ago
    Stack Trace
    #0 /home/headblip/public_html/library/Zend/Db/Statement.php(297): Zend_Db_Statement_Mysqli->_execute(Array)
    #1 /home/headblip/public_html/library/Zend/Db/Adapter/Abstract.php(479): Zend_Db_Statement->execute(Array)
    #2 /home/headblip/public_html/library/Andy/Robots/Index.php(100): Zend_Db_Adapter_Abstract->query('\n\t\t\tINSERT INTO...', Array)
    #3 [internal function]: Andy_Robots_Index::getHtml('', Array, Object(XenForo_Template_Public))
    #4 /home/headblip/public_html/library/XenForo/Template/Abstract.php(330): call_user_func(Array, '', Array, Object(XenForo_Template_Public))
    #5 /home/headblip/public_html/library/XenForo/Template/Abstract.php(265) : eval()'d code(98): XenForo_Template_Abstract->callTemplateCallback('Andy_Robots_Ind...', 'getHtml', '', Array)
    #6 /home/headblip/public_html/library/XenForo/Template/Abstract.php(265): eval()
    #7 /home/headblip/public_html/library/XenForo/Template/Abstract.php(191): XenForo_Template_Abstract->_renderInternal('$__output = '';...', Array)
    #8 /home/headblip/public_html/library/XenForo/Template/Public.php(110): XenForo_Template_Abstract->render()
    #9 /home/headblip/public_html/library/XenForo/ViewRenderer/HtmlPublic.php(139): XenForo_Template_Public->render()
    #10 /home/headblip/public_html/library/XenForo/FrontController.php(618): XenForo_ViewRenderer_HtmlPublic->renderContainer(Object(XenForo_Template_Public), Array)
    #11 /home/headblip/public_html/library/XenForo/FrontController.php(158): XenForo_FrontController->renderView(Object(XenForo_ControllerResponse_View), Object(XenForo_ViewRenderer_HtmlPublic), Array)
    #12 /home/headblip/public_html/index.php(13): XenForo_FrontController->run()
    #13 {main}
    Request State
    array(3) {
    ["url"] => string(23) "http://vapingblips.com/"
    ["_GET"] => array(0) {
    ["_POST"] => array(0) {

    Delete... Close
  12. AndyB

    AndyB Well-Known Member

  13. BentDreams

    BentDreams Active Member

    Thanks for the quick response @AndyB the user happened to be trying to get on-line as I applied the fix and everything seems to be working fine now.
    AndyB likes this.
  14. adwade

    adwade Active Member

    I too have the same question about how the add-on functions.

    The answer is no. Listing various user-agent strings simply redefines the visitor from a type=guest to a (for example) type=googlebot.
    Last edited: Feb 22, 2016
  15. Ranger375

    Ranger375 Well-Known Member

    I installed the addon and removed baidu from the list but baidu is still crawling my forum...

  16. Bombardier

    Bombardier Active Member

    Yep I just installed this and still get blocked bots spidering?, is there some kind of time before we see them blocked?
    Also could you please tell me if I need to set any other usergroups to 'Allow' or is it just unregistered and registered?

    thanks for taking the time to build this, much appreciated
  17. Chillers

    Chillers Member

    As above, this does not appear to be block baidu, i have resorted to blocking the common ip used by baidu 180.76.15.*
  18. adwade

    adwade Active Member

    Just a note to all, don't overlook this in the instructions:
    If you fail to do this, then the add-on cannot render a blank-page to any un-approved robot(s).
  19. adwade

    adwade Active Member

    Quick question: What is the format of the Date code?(i.e. 1456128510)

    (y) Thanxxx to AndyB, I now know it's the number of seconds since Jan 01 1970. (UTC)
    See Epoch Unix Time Stamp Converter for more info.
    Last edited: Feb 22, 2016
  20. Håkan Olsson

    Håkan Olsson Member


    I suppose it solely uses robot names -- but by any chance, can I set it up to *blank-page* all robots independent of robot name? (Kind of -- don't give any [identified as a robot] anything)


Share This Page