Robots 2.1

AndyB · Dec 21, 2014

AndyB submitted a new resource:

Robots - Disallows robots from indexing your site unless explicitly allowed.

Description:

Disallows robots from indexing your site unless explicitly allowed.

(Example of Robots page)

View attachment 92404

(Example of Options page)

View attachment 92405

(Example of User Group Permissions)

View attachment 92406

Questions and Answers:

Q: What is the purpose of this add-on?
A: With many robots crawling your site your site, pages load slower. This add-on will make sure your site is always running at its peak performance.

Q: I...

Read more about this resource...

AndyB · Feb 3, 2015

AndyB updated Robots with a new update entry:

See description

Robots v1.1 changes:

Fixed minor PHP code problem which would on occasion send a server error log.

Read the rest of this update entry...

AndyB · Feb 18, 2015

AndyB updated Robots with a new update entry:

See description

Robots v1.2 changes:

Fixed Install.php code as it had a problem creating the index properly.

Read the rest of this update entry...

surfinsaxman · Feb 21, 2015

I installed the add-on, but where do I see the robots page?

surfinsaxman · Feb 21, 2015

ok. found it

AndyB · Mar 4, 2015

AndyB updated Robots with a new update entry:

See description

Robots v1.3 changes:

Fixed xf_robots index. Please uninstall previous version of this add-on and install this new version.

Read the rest of this update entry...

AzzidReign · Mar 13, 2015

Weird question, do you have to define the bots in order for them to be blocked if they aren't on the allowed list?

Also, I've been noticing people talking about using search engines that value privacy and one of them I keep hearing about is DuckDuckGo. I've added it to my list but anyone reading this, I figured I would put that out there for them:
https://duckduckgo.com/duckduckbot

valdet · Mar 25, 2015

Andy, this may be outside the scope of your product, but can you add an option, where admins/mod can disable robots on individual forums or threads.

Also an option to append noindex and nofollow tags in <head> would be very useful.

AndyB · May 29, 2015

AndyB updated Robots with a new update entry:

See description

Robots v1.4 changes:

Made the Allowed Robots textbox larger in the Options page.

Read the rest of this update entry...

BentDreams · Jun 18, 2015

@AndyB I have the same question as @valdet ... Is there a way to prevent the robots from crawling a specific category or sub-forum?

We have recently made a private area for founding members and they are asking about the content coming up in search engines. So far I haven't found anything that addressed just a partial block of robots for certain areas of a site.

Thanks

BentDreams · Jun 21, 2015

@AndyB I installed this addon and am getting an error in my server log which will be below:

Currently I have the following listed in "Robot Names":
360spider,ahrefsbot,alexa,ask jeeves,archive.org_bot,baiduspider,bingbot,blexbot,CrystalSemanticsBot,facebookexternalhit,google-adsense,googlebot,magpie-crawler,linkdexbot,mj12bot,msnbot,NewsGatorOnline,orangebot,proximic,scoutjet,sogou web spider,yahoo! slurp,yandex

And the following in "Allowed Robots":
bingbot,googlebot,yahoo! slurp,DuckDuckBot,yahoo!-slurp,slurp,msnbot,google-adsense,teoma,robozilla,scoutjet

This is the error from my server logs, note that "Roguexx" is a registered member of my site, not a bot, but all the error logs list him as the "Generated By:" user.

Error Info
Zend_Db_Statement_Mysqli_Exception: Mysqli statement execute error : Data too long for column 'ip' at row 1 - library/Zend/Db/Statement/Mysqli.php:214
Generated By: Roguexx, 7 minutes ago
Stack Trace
#0 /home/headblip/public_html/library/Zend/Db/Statement.php(297): Zend_Db_Statement_Mysqli->_execute(Array)
#1 /home/headblip/public_html/library/Zend/Db/Adapter/Abstract.php(479): Zend_Db_Statement->execute(Array)
#2 /home/headblip/public_html/library/Andy/Robots/Index.php(100): Zend_Db_Adapter_Abstract->query('\n\t\t\tINSERT INTO...', Array)
#3 [internal function]: Andy_Robots_Index::getHtml('', Array, Object(XenForo_Template_Public))
#4 /home/headblip/public_html/library/XenForo/Template/Abstract.php(330): call_user_func(Array, '', Array, Object(XenForo_Template_Public))
#5 /home/headblip/public_html/library/XenForo/Template/Abstract.php(265) : eval()'d code(98): XenForo_Template_Abstract->callTemplateCallback('Andy_Robots_Ind...', 'getHtml', '', Array)
#6 /home/headblip/public_html/library/XenForo/Template/Abstract.php(265): eval()
#7 /home/headblip/public_html/library/XenForo/Template/Abstract.php(191): XenForo_Template_Abstract->_renderInternal('$__output = '';...', Array)
#8 /home/headblip/public_html/library/XenForo/Template/Public.php(110): XenForo_Template_Abstract->render()
#9 /home/headblip/public_html/library/XenForo/ViewRenderer/HtmlPublic.php(139): XenForo_Template_Public->render()
#10 /home/headblip/public_html/library/XenForo/FrontController.php(618): XenForo_ViewRenderer_HtmlPublic->renderContainer(Object(XenForo_Template_Public), Array)
#11 /home/headblip/public_html/library/XenForo/FrontController.php(158): XenForo_FrontController->renderView(Object(XenForo_ControllerResponse_View), Object(XenForo_ViewRenderer_HtmlPublic), Array)
#12 /home/headblip/public_html/index.php(13): XenForo_FrontController->run()
#13 {main}
Request State
array(3) {
["url"] => string(23) "http://vapingblips.com/"
["_GET"] => array(0) {
}
["_POST"] => array(0) {
}
}

Delete... Close

AndyB · Jun 21, 2015

AndyB updated Robots with a new update entry:

See description

Robots v1.5 changes:

Added code to ensure IP address is always 39 characters or less.

Read the rest of this update entry...

BentDreams · Jun 21, 2015

Thanks for the quick response @AndyB the user happened to be trying to get on-line as I applied the fix and everything seems to be working fine now.

adwade · Aug 27, 2015

AzzidReign said:
Weird question, do you have to define the bots in order for them to be blocked if they aren't on the allowed list?

I too have the same question about how the add-on functions.

/EDIT
The answer is no. Listing various user-agent strings simply redefines the visitor from a type=guest to a (for example) type=googlebot.

Ranger375 · Dec 26, 2015

I installed the addon and removed baidu from the list but baidu is still crawling my forum...

Bombardier · Jan 25, 2016

Yep I just installed this and still get blocked bots spidering?, is there some kind of time before we see them blocked?
Also could you please tell me if I need to set any other usergroups to 'Allow' or is it just unregistered and registered?

thanks for taking the time to build this, much appreciated

Chillers · Feb 10, 2016

As above, this does not appear to be block baidu, i have resorted to blocking the common ip used by baidu 180.76.15.*

adwade · Feb 22, 2016

Just a note to all, don't overlook this in the instructions:

Setup:
Set the Unregistered / Unconfirmed -and- Registered group permissions to Allow. The Allow setting means the Robots add-on will function for that user group.

If you fail to do this, then the add-on cannot render a blank-page to any un-approved robot(s).

adwade · Feb 22, 2016

Quick question: What is the format of the Date code?(i.e. 1456128510)

/EDIT

Thanxxx to AndyB, I now know it's the number of seconds since Jan 01 1970. (UTC)
See Epoch Unix Time Stamp Converter for more info.

Håkan Olsson · Jun 25, 2016

Hi,

I suppose it solely uses robot names -- but by any chance, can I set it up to *blank-page* all robots independent of robot name? (Kind of -- don't give any [identified as a robot] anything)

/Thanks

Robots 2.1

Well-known member

Well-known member

Well-known member

Active member

Active member

Well-known member

Well-known member

Active member

Well-known member

Active member

Active member

Well-known member

Active member

Well-known member

Well-known member

Active member

Member

Well-known member

Well-known member

Member

Similar threads

We value your privacy