Partial fix Google search console: sitemap pending due to query string param

[sross]

For some reason google is testing my sitemap, throwing an error for /forum/tags/dev

It then claims it will index everything else, but it has stopped indexing the forum. For weeks I would visit the search console and see how many urls were submitted and indexed. Now it states 0 urls submitted. I have tried updating robots.txt and re-submitting the index but nothing is working. Can anyone help? Thanks

 

[Mike]

What's the error it's displaying for that URL?

There isn't really any reason for it to "block" indexing though. It's more likely just not come back to your site for any reason, though you can likely check that via your server logs. This would be down to something internal in Google's algorithms.

 

[sross]

The urls submitted used to always have a graph, for years.. it's been empty for weeks:



Here is the odd error. Even deleting the sitemap and re-adding after i told robots.txt to ignore tags does no good, it still seems to say Sept 9th and throw the error:



Thanks for any help.

 

[TPerry]

One thing I notice... you shouldn't have /forum in that URL. It should be www.yourdomain.com/sitemap.php - unless you actually have your site in a subdirectory off the root called /forum.

 

[sross]

yeah it's in a subfolder, and nothing has really changed.. the maps always worked fine, then out of the blue this weirdness. thanks

 

[Dnyan]

Add disallow in robot text
Which urls shows 403, 404 errors.

Sample robots.txt

https://xenforo.com/community/robots.txt

Forum setting do not allow some urls to show publically and robots get errors.

Either change settings in user permissions or add robots.txt in root of public html derectory

 

[Dnyan]

Also use SEO friendly Url option.

Ur url error shows its not seo friendly

I too had lot of errors in seo

I added robots.txt

Checked the robots.txt in search console for correctness of blocking location

If your site is at /forum then you should disallow /forum/attachments/ - its just example

If its direct at domain
Then

Disallow /attachments/

This will ensure blocking of 403 404 error displayed to robots

 

[sross]

I have enabled friendly urls,, what about my robots txt is this ok or should I change something? (forum name removed) Thanks!

User-agent: *
Disallow: /forum/find-new/
Disallow: /forum/conversations/
Disallow: /forum/members/
Disallow: /forum/media/users/
Disallow: /forum/account/
Disallow: /forum/attachments/
Disallow: /forum/goto/
Disallow: /forum/register/
Disallow: /forum/posts/
Disallow: /forum/js/
Disallow: /forum/gallery/
Disallow: /forum/media/
Disallow: /forum/login/
Disallow: /forum/admin.php
Disallow: /forum/credits/
Disallow: /forum/tags
Allow: /

Sitemap: https://www___________com/forum/sitemap.php

 

[Dnyan]

seems to be ok

had you poor robots.txt in root directory

go to search console
in crawl click on robots.txt tester

it should open your robots.txt and will also show if there is any error

if its showing it what you uploaded then you have successfully blocked the robots from accessing this content and resulting into crawler errors.

If not showing then it means you have not placed it in root directory (it should be placed at ******.com/robots.txt and not at ******.com/forum/robots.txt

do check spelling of your file to its should be robots.txt

if console is showing your robots file then ok

go to fetch as google bot type your site URL then if it crawls successfully, submit it for indexing along with clicking option submit associated urls.

after this google bots will start crawling your site and soon it will accept sitemap and indexing will be done.

Hope this will help you.

 

[Dnyan]

Keep checking your console for crawl errors and see the urls which gives error.

to fix it either update permissions to user group or block them in robots.txt

Also consider rebuilding sitemap and resubmitting it.

Test the sitemap for correctness before submitting

 

[sross]

Thank you! Will work on this in the morning

 

[sross]

Well I am still getting the same error, the 403 for tags/dev eventhough I have disallowed this in the robots.txt and updated that. I am unsure why google keeps using this url. My robots.txt validates with no errors. How do I remove specific urls from the sitemap generation? Thanks

 

[sross]

here is my current robots.txt

---------------------
User-agent: *
Disallow: /forum/find-new/
Disallow: /forum/conversations/
Disallow: /forum/members/
Disallow: /forum/media/users/
Disallow: /forum/account/
Disallow: /forum/attachments/
Disallow: /forum/goto/
Disallow: /forum/register/
Disallow: /forum/posts/
Disallow: /forum/js/
Disallow: /forum/gallery/
Disallow: /forum/media/
Disallow: /forum/login/
Disallow: /forum/admin.php
Disallow: /forum/credits/
Disallow: /forum/tags/
Disallow: /forum/groupdiscussions/
Allow: /

Sitemap: https://www.__________.com/forum/sitemap.php

---------------------

Notice I have /forum/tags/ to try to stop the sitemap error but it seems to have zero effect. Google persists in saying I have submitted zero links to be indexed.

I have also validated the sitemap, even more odd when testing the sitemap it says successful and zero errors, then I submit and it throws the error.

 

[dutchbb]

I have the same problem in webmaster but no error. I think it is a google problem, when i check indexing status in webmaster or site: on search everything is ok. I always use those because they are a correct view of what is indexed.

 

[sross]

I have the same problem in webmaster but no error. I think it is a google problem, when i check indexing status in webmaster or site: on search everything is ok. I always use those because they are a correct view of what is indexed.

Yeah, I suspect there is no way to get actual support from google either..

 

[dutchbb]

Yeah, I suspect there is no way to get actual support from google either..

Propably not. If it is a google problem they will find it eventually and fix it, the less people that have the problem the longer it takes.

 

[Joe Blow]

I have the same problem as well. My sitemap has always worked fine then a couple of weeks ago problems started.



Google Search Console is not recognising the 2 child sitemaps and the sitemap is constantly listed as "pending".

I have made no changes in recent times. This happened entirely without warning and for no apparent reason.

 

[CTXMedia]

I'm experiencing the same as @Joe Blow - if I submit the child sitemaps from within the parent they list okay, but Google is always showing the main sitemap.php file as Pending.

Any idea what's changed or what could be wrong?

 

[DavidXLD]

Hi,
Exactly the same problem than JoeBlow and CTX.
Any news about this issue?
Thanks
David

 

[CTXMedia]

@Mike @Chris D any thoughts as to why the sitemap.php file is continually showing as pending and never shows graphs/bars/numbers- whereas when I submit the individual sitemap files (4 of them for CycleChat) they show up just fine and show numbers and data as expected?

Anything changed in the formatting of sitemap.php in XF 1.5.15? Or maybe Google has moved the goalposts?

It worked fine up until recently.