Zimply
Member
Lately I found hundreds of link in the Google Webmaster Tools report showing pages which are blocked for unregistered users but included in the sitemap.
I am not technical enough to know if its a sitemap generation issue OR I need some other steps to block these URLs from robots.txt.
The links are all pointing to the front-end page in this URL structure:
forum.com/threads/thread-title.123/edit-prefix
The edit-prefix page is only accessible to logged in users who have permission to edit thread prefixes. But, google is trying to view these pages and reports a permission error warning.
To check this on your forum, try to view a thread as a GUEST and add this at the end of the URL /edit-prefix and view the page, you should most likely get a login page to continue.
If this is not the sitemap issue, sorry for posting here, please move my thread to appropriate location.
And if anyone has an idea how I can block robots.txt from accessing just the /edit-prefix type of URLs please advice.
Thanks everyone
I am not technical enough to know if its a sitemap generation issue OR I need some other steps to block these URLs from robots.txt.
The links are all pointing to the front-end page in this URL structure:
forum.com/threads/thread-title.123/edit-prefix
The edit-prefix page is only accessible to logged in users who have permission to edit thread prefixes. But, google is trying to view these pages and reports a permission error warning.
To check this on your forum, try to view a thread as a GUEST and add this at the end of the URL /edit-prefix and view the page, you should most likely get a login page to continue.
If this is not the sitemap issue, sorry for posting here, please move my thread to appropriate location.
And if anyone has an idea how I can block robots.txt from accessing just the /edit-prefix type of URLs please advice.
Thanks everyone