• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.
Robots

Robots 2.1

No permission to download

AndyB

Well-known member
#1
AndyB submitted a new resource:

Robots - Disallows robots from indexing your site unless explicitly allowed.

Description:

Disallows robots from indexing your site unless explicitly allowed.

(Example of Robots page)

View attachment 92404

(Example of Options page)

View attachment 92405

(Example of User Group Permissions)

View attachment 92406

Questions and Answers:

Q: What is the purpose of this add-on?
A: With many robots crawling your site your site, pages load slower. This add-on will make sure your site is always running at its peak performance.

Q: I...
Read more about this resource...
 

AzzidReign

Well-known member
#7
Weird question, do you have to define the bots in order for them to be blocked if they aren't on the allowed list?

Also, I've been noticing people talking about using search engines that value privacy and one of them I keep hearing about is DuckDuckGo. I've added it to my list but anyone reading this, I figured I would put that out there for them:
https://duckduckgo.com/duckduckbot
 

valdet

Active member
#8
Andy, this may be outside the scope of your product, but can you add an option, where admins/mod can disable robots on individual forums or threads.

Also an option to append noindex and nofollow tags in <head> would be very useful.
 
#10
@AndyB I have the same question as @valdet ... Is there a way to prevent the robots from crawling a specific category or sub-forum?

We have recently made a private area for founding members and they are asking about the content coming up in search engines. So far I haven't found anything that addressed just a partial block of robots for certain areas of a site.

Thanks
 
#11
@AndyB I installed this addon and am getting an error in my server log which will be below:

Currently I have the following listed in "Robot Names":
360spider,ahrefsbot,alexa,ask jeeves,archive.org_bot,baiduspider,bingbot,blexbot,CrystalSemanticsBot,facebookexternalhit,google-adsense,googlebot,magpie-crawler,linkdexbot,mj12bot,msnbot,NewsGatorOnline,orangebot,proximic,scoutjet,sogou web spider,yahoo! slurp,yandex

And the following in "Allowed Robots":
bingbot,googlebot,yahoo! slurp,DuckDuckBot,yahoo!-slurp,slurp,msnbot,google-adsense,teoma,robozilla,scoutjet

This is the error from my server logs, note that "Roguexx" is a registered member of my site, not a bot, but all the error logs list him as the "Generated By:" user.

Error Info
Zend_Db_Statement_Mysqli_Exception: Mysqli statement execute error : Data too long for column 'ip' at row 1 - library/Zend/Db/Statement/Mysqli.php:214
Generated By: Roguexx, 7 minutes ago
Stack Trace
#0 /home/headblip/public_html/library/Zend/Db/Statement.php(297): Zend_Db_Statement_Mysqli->_execute(Array)
#1 /home/headblip/public_html/library/Zend/Db/Adapter/Abstract.php(479): Zend_Db_Statement->execute(Array)
#2 /home/headblip/public_html/library/Andy/Robots/Index.php(100): Zend_Db_Adapter_Abstract->query('\n\t\t\tINSERT INTO...', Array)
#3 [internal function]: Andy_Robots_Index::getHtml('', Array, Object(XenForo_Template_Public))
#4 /home/headblip/public_html/library/XenForo/Template/Abstract.php(330): call_user_func(Array, '', Array, Object(XenForo_Template_Public))
#5 /home/headblip/public_html/library/XenForo/Template/Abstract.php(265) : eval()'d code(98): XenForo_Template_Abstract->callTemplateCallback('Andy_Robots_Ind...', 'getHtml', '', Array)
#6 /home/headblip/public_html/library/XenForo/Template/Abstract.php(265): eval()
#7 /home/headblip/public_html/library/XenForo/Template/Abstract.php(191): XenForo_Template_Abstract->_renderInternal('$__output = '';...', Array)
#8 /home/headblip/public_html/library/XenForo/Template/Public.php(110): XenForo_Template_Abstract->render()
#9 /home/headblip/public_html/library/XenForo/ViewRenderer/HtmlPublic.php(139): XenForo_Template_Public->render()
#10 /home/headblip/public_html/library/XenForo/FrontController.php(618): XenForo_ViewRenderer_HtmlPublic->renderContainer(Object(XenForo_Template_Public), Array)
#11 /home/headblip/public_html/library/XenForo/FrontController.php(158): XenForo_FrontController->renderView(Object(XenForo_ControllerResponse_View), Object(XenForo_ViewRenderer_HtmlPublic), Array)
#12 /home/headblip/public_html/index.php(13): XenForo_FrontController->run()
#13 {main}
Request State
array(3) {
["url"] => string(23) "http://vapingblips.com/"
["_GET"] => array(0) {
}
["_POST"] => array(0) {
}
}

Delete... Close
 

adwade

Active member
#14
Weird question, do you have to define the bots in order for them to be blocked if they aren't on the allowed list?
I too have the same question about how the add-on functions.

/EDIT
The answer is no. Listing various user-agent strings simply redefines the visitor from a type=guest to a (for example) type=googlebot.
 
Last edited:
#16
Yep I just installed this and still get blocked bots spidering?, is there some kind of time before we see them blocked?
Also could you please tell me if I need to set any other usergroups to 'Allow' or is it just unregistered and registered?

thanks for taking the time to build this, much appreciated
 

adwade

Active member
#18
Just a note to all, don't overlook this in the instructions:
Setup:
Set the Unregistered / Unconfirmed -and- Registered group permissions to Allow. The Allow setting means the Robots add-on will function for that user group.
If you fail to do this, then the add-on cannot render a blank-page to any un-approved robot(s).
 
#20
Hi,

I suppose it solely uses robot names -- but by any chance, can I set it up to *blank-page* all robots independent of robot name? (Kind of -- don't give any [identified as a robot] anything)

/Thanks