Custom Robots 1.1.4

Liam W · Jan 23, 2015

kaieivindm said:
@Liam W

So I am having some issues with i.e Twitterbot.

From Xenforo
View attachment 95824

From Twitter
View attachment 95825

So we know the user-agent for the bot is Twitterbot.
And as you can see, this is what I use as well.

However the server logs says this:

ErrorException: Undefined index: twitterbot - library/XenForo/Session.php:877
Generert av: Ukjent konto, I går kl 23:19
Stack Trace
#0 /home/4/s/spillforumet/www/library/XenForo/Session.php(877): XenForo_Application::handlePhpError(8, 'Undefined index...', '/home/4/s/spill...', 877, Array)
#1 /home/4/s/spillforumet/www/library/XenForo/Session.php(350): XenForo_Session->getRobotId('Twitterbot/1.0')
#2 /home/4/s/spillforumet/www/library/XenForo/Session.php(257): XenForo_Session::getPublicSession(Object(Zend_Controller_Request_Http))
#3 /home/4/s/spillforumet/www/library/XenForo/Controller.php(291): XenForo_Session::startPublicSession(Object(Zend_Controller_Request_Http))
#4 /home/4/s/spillforumet/www/library/XenForo/Controller.php(304): XenForo_Controller->_setupSession('Index')
#5 /home/4/s/spillforumet/www/library/XenForo/FrontController.php(346): XenForo_Controller->preDispatch('Index', 'XenForo_Control...')
#6 /home/4/s/spillforumet/www/library/XenForo/FrontController.php(134): XenForo_FrontController->dispatch(Object(XenForo_RouteMatch))
#7 /home/4/s/spillforumet/www/index.php(13): XenForo_FrontController->run()
#8 {main}
Forespurt status
array(3) {
["url"] => string(84) "http://www.spillforumet.no/traad/halo-5-guardians-multiplayer-beta-desember-2014.71/"
["_GET"] => array(0) {
}
["_POST"] => array(0) {
}
}

Is this something on my side? Something I can do to fix?
Or is it something else?

What if you use a lowercase T in the user agent?

Liam

kaieivindm · Jan 23, 2015

Liam W said:
What if you use a lowercase T in the user agent?

Liam

OK - done! Will keep you updated next time I see the bot on or error logs comes.

Liam W · Jan 23, 2015

Liam W updated Custom Robots with a new update entry:

Bug Fixes & Improvements

This version adds basic caching, as well as fixing the case sensitivity bug.

It also has improvements for the export robots page.

Read the rest of this update entry...

kaieivindm · Jan 23, 2015

Liam W said:
Liam W updated Custom Robots with a new update entry:

Bug Fixes & Improvements

Read the rest of this update entry...

@Liam W it was ok for twitterbot with small T in the beginning

Thanks!

Liam W · Jan 24, 2015

Liam W updated Custom Robots with a new update entry:

Bug Fixes

This update fixes a few bugs that I didn't catch and/or introduced in the last version.

This also adds an option to completely replace the built in robots with your custom robots.

Read the rest of this update entry...

Sunka · Jan 25, 2015

On my former vBulletin forum, I have xml file with robots. I can not import that list couse format is different.
Export list from this mod is:

Code:

<robots version="10104">
  <robot name="AhrefsBot" link="http://ahrefs.com/robot/" user_agent="ahrefsBot"/>
  <robot name="AboutUsBot" link="http://www.aboutus.org/AboutUs:Bot" user_agent="aboutUsBot"/>
</robots>

but spiders list from vBulletin is in this format:

Code:

<spider ident="AhrefsBot">
        <name>AhrefsBot</name>
        <info>http://ahrefs.com/robot/</info>
        <email>support@ahrefs.com</email>
    </spider>
    <spider ident="AideRSS">
        <name>AideRSS (PostRank.com)</name>
        <info>http://www.postrank.com</info>
    </spider>
    <spider ident="ia_archiver">
        <name>Alexa</name>
        <info>http://www.alexa.com/site/help/webmasters</info>
    </spider>

Any chance to import that kind of list?
Also, User Agent (substring) field allways need to begin with sentence letter (no Capital letter)?

Liam W · Jan 25, 2015

Sunka said:
On my former vBulletin forum, I have xml file with robots. I can not import that list couse format is different.
Export list from this mod is:

Code:

<robots version="10104"> <robot name="AhrefsBot" link="http://ahrefs.com/robot/" user_agent="ahrefsBot"/> <robot name="AboutUsBot" link="http://www.aboutus.org/AboutUs:Bot" user_agent="aboutUsBot"/> </robots>

but spiders list from vBulletin is in this format:

Code:

<spider ident="AhrefsBot"> <name>AhrefsBot</name> <info>http://ahrefs.com/robot/</info> <email>support@ahrefs.com</email> </spider> <spider ident="AideRSS"> <name>AideRSS (PostRank.com)</name> <info>http://www.postrank.com</info> </spider> <spider ident="ia_archiver"> <name>Alexa</name> <info>http://www.alexa.com/site/help/webmasters</info> </spider>

Any chance to import that kind of list?
Also, User Agent (substring) field allways need to begin with sentence letter (no Capital letter)?

The user agent substring can start with any case letter now - although the code does turn everything into lowercase. The only reason for this is that array keys appear to be case sensitive, so if it isn't the correct case, it'll error out.

I'll look into accepting vB import XML.

Sunka · Jan 25, 2015

Liam W said:
I'll look into accepting vB import XML

Thanks...

In vBulletin XML file there are a different formats.
Here is example of spiders defenition (with email, without email, with/without link...)

Code:

<spider ident="DotBot">
     <name>DotBot/1.1</name>
  <info>http://www.opensiteexplorer.org/dotbot</info>
   </spider>
spider ident="Voltron">
     <name>Voltron</name>
  <info>voltron</info>  
   </spider>
<spider ident="Crawlzilla">
     <name>Crawlzilla</name>
     <info>http://www.crawlzilla.com</info>
     <email>crawler@crawlzilla.com</email>
   </spider>   
<spider ident="CCBot">
     <name>CommonCrawl</name>
     <info>http://www.commoncrawl.org/</info>
     <email>ccbot@commoncrawl.org</email>
   </spider>
   <spider ident="Coccoc">
     <name>Coccoc</name>
     <info>+http://help.coccoc.com/</info>
   </spider>  
   <spider ident="COMODOspider">
     <name>COMODO</name>
   </spider>
   <spider ident="Comodo Spider 1.2">
     <name>Comodo</name>
   </spider>      
   <spider ident="ComputingSite Robi">
     <name>ComputingSite Robi</name>
   </spider>
   <spider ident="conceptbot">
     <name>ConceptBot</name>
   </spider>
   <spider ident="ContextAd">
     <name>ContextAd Bot</name>
   </spider>
   <spider ident="Cooby.de Crawler">
     <name>Cooby</name>
     <info>Cooby.de</info>
   </spider>
   <spider ident="CoolBot">
     <name>CoolBot</name>
   </spider>

Liam W · Jan 25, 2015

Sunka said:

Thanks...

In vBulletin XML file there are a different formats.
Here is example of spiders defenition (with email, without email, with/without link...)

Code:

<spider ident="DotBot">
     <name>DotBot/1.1</name>
  <info>http://www.opensiteexplorer.org/dotbot</info>
   </spider>
spider ident="Voltron">
     <name>Voltron</name>
  <info>voltron</info> 
   </spider>
<spider ident="Crawlzilla">
     <name>Crawlzilla</name>
     <info>http://www.crawlzilla.com</info>
     <email>crawler@crawlzilla.com</email>
   </spider>  
<spider ident="CCBot">
     <name>CommonCrawl</name>
     <info>http://www.commoncrawl.org/</info>
     <email>ccbot@commoncrawl.org</email>
   </spider>
   <spider ident="Coccoc">
     <name>Coccoc</name>
     <info>+http://help.coccoc.com/</info>
   </spider> 
   <spider ident="COMODOspider">
     <name>COMODO</name>
   </spider>
   <spider ident="Comodo Spider 1.2">
     <name>Comodo</name>
   </spider>     
   <spider ident="ComputingSite Robi">
     <name>ComputingSite Robi</name>
   </spider>
   <spider ident="conceptbot">
     <name>ConceptBot</name>
   </spider>
   <spider ident="ContextAd">
     <name>ContextAd Bot</name>
   </spider>
   <spider ident="Cooby.de Crawler">
     <name>Cooby</name>
     <info>Cooby.de</info>
   </spider>
   <spider ident="CoolBot">
     <name>CoolBot</name>
   </spider>

There needs to be a main container for XF to be able to correctly read the XML (the security scanner method requires it) - that file contains loads of individual elements.

kaieivindm · Feb 18, 2016

I suddenly can't add robots like "FeedlyBot/1.0" due to the / in the field of user strings. Is this something that has always been there? Because I have other robots added like this with the / in the field.

Liam W · Feb 18, 2016

kaieivindm said:
I suddenly can't add robots like "FeedlyBot/1.0" due to the / in the field of user strings. Is this something that has always been there? Because I have other robots added like this with the / in the field.

It was introduced in a version a while ago I believe.

You should really just use the 'FeedlyBot' substring (ignoring the version).

Liam

kaieivindm · Feb 18, 2016

Liam W said:
It was introduced in a version a while ago I believe.

You should really just use the 'FeedlyBot' substring (ignoring the version).

Liam

Yeah it was what I did... just waiting for the bot to kick off again

thanks!

Custom Robots 1.1.4

Liam W

in memoriam 1998-2020

kaieivindm

Well-known member

Liam W

in memoriam 1998-2020

kaieivindm

Well-known member

Liam W

in memoriam 1998-2020

Sunka

Well-known member

Liam W

in memoriam 1998-2020

Sunka

Well-known member

Liam W

in memoriam 1998-2020

kaieivindm

Well-known member

Liam W

in memoriam 1998-2020

kaieivindm

Well-known member

Similar threads

We value your privacy