XF 1.5 Google crawling error

[abdfahim]

I am getting two kind of Google crawling errors in my XF installation

1) 403 error for password protected pages, like /account/personal-details

2) 404 error where Google just add the phrase "mobile" or "m" in the link, like /mobile/threads/abcd.10/ instead of /threads/abcd.10/. As per some posts in StackOverflow, it can happen if google thinks the website is not responsive, which is ofcourse note true for XF.

Any idea/advise how I can avoid these errors.

 

[abdfahim]

I guess the second error has something to do with the fact that recently I have created a rule in htaccess to forward all traffic to https:// (because the error started from the same day). I'll investigate more.

But, I'd love to know how can I handle the first error (throwing error for password protected page).

 

[TPerry]

Do you have a robots.txt? If not, you need to create one and do a disallow statement for that URL

https://xenforo.com/robots.txt

User-agent: *
Disallow: /community/find-new/
Disallow: /community/account/
Disallow: /community/attachments/
Disallow: /community/goto/
Disallow: /community/posts/
Disallow: /community/login/
Disallow: /community/admin.php
Allow: /

The above is the one in use on XenForo. You would probably need to strip the /community out.