Robots [Deleted]

AndyB

AndyB

Well-known member
AndyB submitted a new resource:

Robots - Disallows robots from indexing your site unless explicitly allowed.

Premium upgrade:

For $25.00 USD this add-on along with the entire collection may be downloaded for up to one year. Please see the colletion in the Resources area at this URL:

https://www.xf2addons.com/

Description:

Disallows robots from indexing your site unless explicitly allowed.

(Robots link in Visitors tab)

View attachment 157056

(Example of Robots page)

View attachment 157057

(Example of Options page)

View attachment 157058

(Example of User group...
Click to expand...

Read more about this resource...
 
Andy when I enable Robots on my site I get the following error

An exception occurred: [XF\Db\Exception] MySQL statement prepare error [1146]: Table 'daedalus_TGR.xf_andy_robots' doesn't exist in src/XF/Db/AbstractStatement.php on line 212
  1. XF\Db\AbstractStatement->getException() in src/XF/Db/Mysqli/Statement.php at line 196
  2. XF\Db\Mysqli\Statement->getException() in src/XF/Db/Mysqli/Statement.php at line 40
  3. XF\Db\Mysqli\Statement->prepare() in src/XF/Db/Mysqli/Statement.php at line 54
  4. XF\Db\Mysqli\Statement->execute() in src/XF/Db/AbstractAdapter.php at line 79
  5. XF\Db\AbstractAdapter->query() in src/addons/Andy/Robots/Listener.php at line 174
  6. Andy\Robots\Listener::appPubRenderPage()
  7. call_user_func_array() in src/XF/Extension.php at line 67
  8. XF\Extension->fire() in src/XF/App.php at line 2350
  9. XF\App->fire() in src/XF/Pub/App.php at line 447
  10. XF\Pub\App->renderPageHtml() in src/XF/App.php at line 1724
  11. XF\App->renderPage() in src/XF/Mvc/Dispatcher.php at line 310
  12. XF\Mvc\Dispatcher->render() in src/XF/Mvc/Dispatcher.php at line 44
  13. XF\Mvc\Dispatcher->run() in src/XF/App.php at line 1889
  14. XF\App->run() in src/XF.php at line 328
  15. XF::runApp() in index.php at line 13
 
Nice addon, I suggest to add more bots from Google to our Whitelist:
https://support.google.com/webmasters/answer/1061943?hl=en ;-)

My whitelist:
Code: 
bingbot,
Googlebot,
Mediapartners,
Yahoo! Slurp,
Mediapartners-Google,
AdsBot-Google,
AdsBot-Google-Mobile,
APIs-Google,
Googlebot-Image,
Googlebot-News,
Googlebot-Video,
AdsBot-Google-Mobile-Apps,

and Yandex bots (because I have some entries from Russia / Ukraine):
https://yandex.com/support/webmaster/controlling-robot/robots-txt.html
Code: 
YandexBot,
YandexDirect,
YandexMedia,
YandexImages,
YaDirectFetcher,
YandexBlogs,
YandexNews,
YandexPagechecker,
YandexMetrika,


My Blacklist:
Code: 
Clickagy,
facebookexternalhit,
Sogou,
MJ12bot,
SearchmetricsBot,
AhrefsBot,
ZyBorg,
 
Last edited:
This is just a note for people using this add-on, if you are using cloudfare then make sure you add CloudFlare, to the whitelist, otherwise it wont let thier bots in like the one that scans your board for the Always Online feature.
.
 
  • Like
Reactions: CNK
You must log in or register to reply here.

Similar threads

AndyB
Direct message reply limit [Paid]
Replies
0
Views
66
AndyB
AndyB
AndyB
Disallowed extensions [Paid]
Replies
0
Views
258
AndyB
AndyB
AndyB
Delete direct messages [Paid]
Replies
0
Views
83
AndyB
AndyB
AndyB
Remove image edit options [Deleted]
Replies
1
Views
107
AndyB
AndyB
AndyB
Last activity [Paid]
Replies
0
Views
135
AndyB
AndyB
Top Bottom