XF 1.5 Google crawling error

abdfahim

Well-known member
I am getting two kind of Google crawling errors in my XF installation

1) 403 error for password protected pages, like /account/personal-details

2) 404 error where Google just add the phrase "mobile" or "m" in the link, like /mobile/threads/abcd.10/ instead of /threads/abcd.10/. As per some posts in StackOverflow, it can happen if google thinks the website is not responsive, which is ofcourse note true for XF.

Any idea/advise how I can avoid these errors.
 

abdfahim

Well-known member
I guess the second error has something to do with the fact that recently I have created a rule in htaccess to forward all traffic to https:// (because the error started from the same day). I'll investigate more.

But, I'd love to know how can I handle the first error (throwing error for password protected page).
 

Tracy Perry

Well-known member
Do you have a robots.txt? If not, you need to create one and do a disallow statement for that URL

https://xenforo.com/robots.txt

Code:
User-agent: *
Disallow: /community/find-new/
Disallow: /community/account/
Disallow: /community/attachments/
Disallow: /community/goto/
Disallow: /community/posts/
Disallow: /community/login/
Disallow: /community/admin.php
Allow: /
The above is the one in use on XenForo. You would probably need to strip the /community out.
 
Top