Robots

Robots [Paid] 2.2

No permission to buy ($25.00)

AndyB

Well-known member
#1
AndyB submitted a new resource:

Robots - Disallows robots from indexing your site unless explicitly allowed.

Premium upgrade:

For $25.00 USD this add-on along with the entire collection may be downloaded for up to one year. Please see the colletion in the Resources area at this URL:

https://www.xf2addons.com/

Description:

Disallows robots from indexing your site unless explicitly allowed.

(Robots link in Visitors tab)

View attachment 157056

(Example of Robots page)

View attachment 157057

(Example of Options page)

View attachment 157058

(Example of User group...
Read more about this resource...
 
#11
Andy when I enable Robots on my site I get the following error

An exception occurred: [XF\Db\Exception] MySQL statement prepare error [1146]: Table 'daedalus_TGR.xf_andy_robots' doesn't exist in src/XF/Db/AbstractStatement.php on line 212
  1. XF\Db\AbstractStatement->getException() in src/XF/Db/Mysqli/Statement.php at line 196
  2. XF\Db\Mysqli\Statement->getException() in src/XF/Db/Mysqli/Statement.php at line 40
  3. XF\Db\Mysqli\Statement->prepare() in src/XF/Db/Mysqli/Statement.php at line 54
  4. XF\Db\Mysqli\Statement->execute() in src/XF/Db/AbstractAdapter.php at line 79
  5. XF\Db\AbstractAdapter->query() in src/addons/Andy/Robots/Listener.php at line 174
  6. Andy\Robots\Listener::appPubRenderPage()
  7. call_user_func_array() in src/XF/Extension.php at line 67
  8. XF\Extension->fire() in src/XF/App.php at line 2350
  9. XF\App->fire() in src/XF/Pub/App.php at line 447
  10. XF\Pub\App->renderPageHtml() in src/XF/App.php at line 1724
  11. XF\App->renderPage() in src/XF/Mvc/Dispatcher.php at line 310
  12. XF\Mvc\Dispatcher->render() in src/XF/Mvc/Dispatcher.php at line 44
  13. XF\Mvc\Dispatcher->run() in src/XF/App.php at line 1889
  14. XF\App->run() in src/XF.php at line 328
  15. XF::runApp() in index.php at line 13
 

CNK

Active member
#15
Nice addon, I suggest to add more bots from Google to our Whitelist:
https://support.google.com/webmasters/answer/1061943?hl=en ;-)

My whitelist:
Code:
bingbot,
Googlebot,
Mediapartners,
Yahoo! Slurp,
Mediapartners-Google,
AdsBot-Google,
AdsBot-Google-Mobile,
APIs-Google,
Googlebot-Image,
Googlebot-News,
Googlebot-Video,
AdsBot-Google-Mobile-Apps,
and Yandex bots (because I have some entries from Russia / Ukraine):
https://yandex.com/support/webmaster/controlling-robot/robots-txt.html
Code:
YandexBot,
YandexDirect,
YandexMedia,
YandexImages,
YaDirectFetcher,
YandexBlogs,
YandexNews,
YandexPagechecker,
YandexMetrika,

My Blacklist:
Code:
Clickagy,
facebookexternalhit,
Sogou,
MJ12bot,
SearchmetricsBot,
AhrefsBot,
ZyBorg,
 
Last edited:

RSI

Active member
#16
This is just a note for people using this add-on, if you are using cloudfare then make sure you add CloudFlare, to the whitelist, otherwise it wont let thier bots in like the one that scans your board for the Always Online feature.
.
 
Top