Codeless
Active member
Hy All xenforo experts,
I need modules which can handle Robots & Crawlers in xenforo forum.
I know robots have default guest permission, but they can't crawl protected forums they don't have permission.
I want a module which can give full permissions to robots to crawl forums for
security reason we can add crawler user agent + IP to whitelist it.
Right now I am facing the same issue with Semrush / Moz / Alexa / GoogleBot / Baidu Crawler / Yandex Bot / and some other important for me to get fast indexing and avoid 404 error in webmaster panel
so I will add + robot agent to put on the safe list like
Crawler name: Googlebot
User agent : Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) or Googlebot/2.1 (+http://www.google.com/bot.html)
IP Address: 64.233.160.0/32 & 66.102.0.0/32 & 66.249.64.0/32 & 72.14.192.0/32 & 74.125.0.0/32
so basically we will be allowing Googlebot from these it with full permissions
I will also need lifetime updates we can decide yearly based
if anyone willing to do it let me know
I need modules which can handle Robots & Crawlers in xenforo forum.
I know robots have default guest permission, but they can't crawl protected forums they don't have permission.
I want a module which can give full permissions to robots to crawl forums for
security reason we can add crawler user agent + IP to whitelist it.
Right now I am facing the same issue with Semrush / Moz / Alexa / GoogleBot / Baidu Crawler / Yandex Bot / and some other important for me to get fast indexing and avoid 404 error in webmaster panel
so I will add + robot agent to put on the safe list like
Crawler name: Googlebot
User agent : Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) or Googlebot/2.1 (+http://www.google.com/bot.html)
IP Address: 64.233.160.0/32 & 66.102.0.0/32 & 66.249.64.0/32 & 72.14.192.0/32 & 74.125.0.0/32
so basically we will be allowing Googlebot from these it with full permissions
I will also need lifetime updates we can decide yearly based
if anyone willing to do it let me know