Robots

Robots 2.1

No permission to download
Overview Updates (11) Reviews (5) History Discussion
AndyB

AndyB

Well-known member
AndyB submitted a new resource:

Robots - Disallows robots from indexing your site unless explicitly allowed.

Description:

Disallows robots from indexing your site unless explicitly allowed.

(Example of Robots page)

View attachment 92404

(Example of Options page)

View attachment 92405

(Example of User Group Permissions)

View attachment 92406

Questions and Answers:

Q: What is the purpose of this add-on?
A: With many robots crawling your site your site, pages load slower. This add-on will make sure your site is always running at its peak performance.

Q: I...
Click to expand...

Read more about this resource...
 
Weird question, do you have to define the bots in order for them to be blocked if they aren't on the allowed list?

Also, I've been noticing people talking about using search engines that value privacy and one of them I keep hearing about is DuckDuckGo. I've added it to my list but anyone reading this, I figured I would put that out there for them:
https://duckduckgo.com/duckduckbot
 
Andy, this may be outside the scope of your product, but can you add an option, where admins/mod can disable robots on individual forums or threads.

Also an option to append noindex and nofollow tags in <head> would be very useful.
 
@AndyB I have the same question as @valdet ... Is there a way to prevent the robots from crawling a specific category or sub-forum?

We have recently made a private area for founding members and they are asking about the content coming up in search engines. So far I haven't found anything that addressed just a partial block of robots for certain areas of a site.

Thanks
 
@AndyB I installed this addon and am getting an error in my server log which will be below:

Currently I have the following listed in "Robot Names":
360spider,ahrefsbot,alexa,ask jeeves,archive.org_bot,baiduspider,bingbot,blexbot,CrystalSemanticsBot,facebookexternalhit,google-adsense,googlebot,magpie-crawler,linkdexbot,mj12bot,msnbot,NewsGatorOnline,orangebot,proximic,scoutjet,sogou web spider,yahoo! slurp,yandex

And the following in "Allowed Robots":
bingbot,googlebot,yahoo! slurp,DuckDuckBot,yahoo!-slurp,slurp,msnbot,google-adsense,teoma,robozilla,scoutjet

This is the error from my server logs, note that "Roguexx" is a registered member of my site, not a bot, but all the error logs list him as the "Generated By:" user.

Error Info
Zend_Db_Statement_Mysqli_Exception: Mysqli statement execute error : Data too long for column 'ip' at row 1 - library/Zend/Db/Statement/Mysqli.php:214
Generated By: Roguexx, 7 minutes ago
Stack Trace
#0 /home/headblip/public_html/library/Zend/Db/Statement.php(297): Zend_Db_Statement_Mysqli->_execute(Array)
#1 /home/headblip/public_html/library/Zend/Db/Adapter/Abstract.php(479): Zend_Db_Statement->execute(Array)
#2 /home/headblip/public_html/library/Andy/Robots/Index.php(100): Zend_Db_Adapter_Abstract->query('\n\t\t\tINSERT INTO...', Array)
#3 [internal function]: Andy_Robots_Index::getHtml('', Array, Object(XenForo_Template_Public))
#4 /home/headblip/public_html/library/XenForo/Template/Abstract.php(330): call_user_func(Array, '', Array, Object(XenForo_Template_Public))
#5 /home/headblip/public_html/library/XenForo/Template/Abstract.php(265) : eval()'d code(98): XenForo_Template_Abstract->callTemplateCallback('Andy_Robots_Ind...', 'getHtml', '', Array)
#6 /home/headblip/public_html/library/XenForo/Template/Abstract.php(265): eval()
#7 /home/headblip/public_html/library/XenForo/Template/Abstract.php(191): XenForo_Template_Abstract->_renderInternal('$__output = '';...', Array)
#8 /home/headblip/public_html/library/XenForo/Template/Public.php(110): XenForo_Template_Abstract->render()
#9 /home/headblip/public_html/library/XenForo/ViewRenderer/HtmlPublic.php(139): XenForo_Template_Public->render()
#10 /home/headblip/public_html/library/XenForo/FrontController.php(618): XenForo_ViewRenderer_HtmlPublic->renderContainer(Object(XenForo_Template_Public), Array)
#11 /home/headblip/public_html/library/XenForo/FrontController.php(158): XenForo_FrontController->renderView(Object(XenForo_ControllerResponse_View), Object(XenForo_ViewRenderer_HtmlPublic), Array)
#12 /home/headblip/public_html/index.php(13): XenForo_FrontController->run()
#13 {main}
Request State
array(3) {
["url"] => string(23) "http://vapingblips.com/"
["_GET"] => array(0) {
}
["_POST"] => array(0) {
}
}

Delete... Close
 
AzzidReign said:
Weird question, do you have to define the bots in order for them to be blocked if they aren't on the allowed list?
Click to expand...

I too have the same question about how the add-on functions.

/EDIT
The answer is no. Listing various user-agent strings simply redefines the visitor from a type=guest to a (for example) type=googlebot.
 
Last edited:
Yep I just installed this and still get blocked bots spidering?, is there some kind of time before we see them blocked?
Also could you please tell me if I need to set any other usergroups to 'Allow' or is it just unregistered and registered?

thanks for taking the time to build this, much appreciated
 
As above, this does not appear to be block baidu, i have resorted to blocking the common ip used by baidu 180.76.15.*
 
Just a note to all, don't overlook this in the instructions:
Setup:
Set the Unregistered / Unconfirmed -and- Registered group permissions to Allow. The Allow setting means the Robots add-on will function for that user group.
Click to expand...
If you fail to do this, then the add-on cannot render a blank-page to any un-approved robot(s).
 
Hi,

I suppose it solely uses robot names -- but by any chance, can I set it up to *blank-page* all robots independent of robot name? (Kind of -- don't give any [identified as a robot] anything)

/Thanks
 
You must log in or register to reply here.

Similar threads

AndyB
Disallowed extensions [Paid]
Replies
0
Views
259
AndyB
AndyB
AndyB
Direct message reply limit [Paid]
Replies
0
Views
73
AndyB
AndyB
AndyB
Auto delete direct messages [Paid]
Replies
0
Views
75
AndyB
AndyB
AndyB
Delete direct messages [Paid]
Replies
0
Views
86
AndyB
AndyB
AndyB
Auto delete conversations [Paid]
Replies
0
Views
307
AndyB
AndyB
Top Bottom