XF 2.0 Stop Google indexing member profiles

[Mr Lucky]

Does anyone know how to stop Google from indexing member profiles?

Google cannot see the profile (no unregistered permissions) but can access the page so it returns a result with the following:

No page information in search results
If a Google search result says no information is available for a page, like this:

It means that the website prevented Google from creating a page description, but didn't actually hide the page from Google.
If you own this page you can improve this result either by letting Google read the page to create a good description, or hiding the page entirely from Google Search results. Read below to learn how.

 

[djbaxter]

Have you explicitly instructed googlebot not to crawl /members/ in a robots.txt file?

 

[Nirjonadda]

Does anyone know how to stop Google from indexing member profiles?

You can do with @au lait SEO add-on.

 

[Mr Lucky]

Have you explicitly instructed googlebot not to crawl /members/ in a robots.txt file?

No I hadn't because in the past I did that and same thing, they still indexed them. In my experience the only way to stop an index was to pot the no index in the head. May be worth trying again.

EDIT: yes actually it seems I do have /members/disallowed, just checked

 

[SyTry]

Hello,

You need to uncheck this too :

• /admin.php?options/groups/sitemap/

And follow this :

XF 2.0 - Stop Google indexing member profiles

Does anyone know how to stop Google from indexing member profiles? Google cannot see the profile (no unregistered permissions) but can access the page so it returns a result with the following: No page information in search results If a Google search result says no information is available for...

xenforo.com

Regards, SyTry

 

[Mr Lucky]

Hmmm, just checked. User is unticked for sitemap and /members/ is actually already disabled in robots.txt

 

[nrep]

As strange as it sounds, you'll likely want to allow /members/ in robots.txt, but to add a "noindex" metatag in the header of the member_view page. Adding files to robots.txt doesn't stop them from appearing in the index, it just stops them from being crawled. If a page has enough links to it, Google will still index that page, but can't fetch the description as it can't crawl the page. If you've got "noindex" in the header, but Google can't crawl the page, it won't know that it shouldn't be indexed.

 

[jgaulard]

If you have the View Member Profiles and View Member Lists options set to No in the permissions area for the Unregistered/Unconfirmed group, Google shouldn't be returning any results for these pages at all. The pages will return a 403 Forbidden status code and those pages will have no trace in the search engines. Now, if the /members/ directory is blocked in the robots.txt file, these pages may be returned in search engine results as you indicated above.

 

[diego222]

If you have the View Member Profiles and View Member Lists options set to No in the permissions area for the Unregistered/Unconfirmed group, Google shouldn't be returning any results for these pages at all. The pages will return a 403 Forbidden status code and those pages will have no trace in the search engines. Now, if the /members/ directory is blocked in the robots.txt file, these pages may be returned in search engine results as you indicated above.

so.... if we don't want members profiles appearing on Google... we shouldn't block "/members/" on robots.txt ? its not a little bit strange that?

 

[jgaulard]

so.... if we don't want members profiles appearing on Google... we shouldn't block "/members/" on robots.txt ? its not a little bit strange that?

Hi - I'm not sure what you mean. If you leave the pages unblocked, Google will crawl them to see that they should not be indexed. If you block them, Google won't be able to crawl them to see the 403 response code.

 

[diego222]

Lets see... I Have this:

1 - /members/ blocked on robots.txt
2 - members option unchecked on options (to not be include in sitemaps)
3 - View Members Profiles and VIew Members List options set to "No" in permissions area for Unregistered/Unconfirmed group

So, my question "translated" is:

-If I have the number 2 and 3... despite being "strange".... should I remove the block on robots.txt from number 1? Do I have to add something more (for example "noindex" on some template) or just removing "option 1" is enough?

 

[jgaulard]

Lets see... I Have this:

1 - /members/ blocked on robots.txt
2 - members option unchecked on options (to not be include in sitemaps)
3 - View Members Profiles and VIew Members List options set to "No" in permissions area for Unregistered/Unconfirmed group

So, my question "translated" is:

-If I have the number 2 and 3... despite being "strange".... should I remove the block on robots.txt from number 1? Do I have to add something more (for example "noindex" on some template) or just removing "option 1" is enough?

Hi - If your goal is to have all member profile pages removed from Google's index and to never see them returned in Google's search results in any way, then yes, you are correct. Remove the block from robots.txt. There is nothing more you need to do. You don't need to add noindex to any templates because search engines and users that aren't logged in will never see the member profile pages. All they'll see is the login page, which already has noindex in the template, by the way.

To test this out, be sure that you are not logged into your site. Then, click a member page and see where you end up. It should be on the login page that says something like, "You need to be logged in to do this..." If you put this URL into a header checker (https://www.webconfs.com/http-header-check.php), you should see it return a 403 code. Google does not index 403 pages and if they are already indexed, they'll be removed over time. They'll need to be recrawled first though.

 

[diego222]

To test this out, be sure that you are not logged into your site. Then, click a member page and see where you end up. It should be on the login page that says something like, "You need to be logged in to do this..." If you put this URL into a header checker (https://www.webconfs.com/http-header-check.php), you should see it return a 403 code. Google does not index 403 pages and if they are already indexed, they'll be removed over time. They'll need to be recrawled first though.

Ok, but the last part... I checked it out (another browser not logged in) and already is restricted with "You need to be logged in to do this" (the robots.txt block is still there)

 

[jgaulard]

Ok, but the last part... I checked it out (another browser not logged in) and already is restricted with "You need to be logged in to do this" (the robots.txt block is still there)

Perfect. Just remove the robots.txt block.