• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

Partial fix Google search console: sitemap pending due to query string param

sross

Active member
#1
For some reason google is testing my sitemap, throwing an error for /forum/tags/dev

It then claims it will index everything else, but it has stopped indexing the forum. For weeks I would visit the search console and see how many urls were submitted and indexed. Now it states 0 urls submitted. I have tried updating robots.txt and re-submitting the index but nothing is working. Can anyone help? Thanks
 

Mike

XenForo developer
Staff member
#2
What's the error it's displaying for that URL?

There isn't really any reason for it to "block" indexing though. It's more likely just not come back to your site for any reason, though you can likely check that via your server logs. This would be down to something internal in Google's algorithms.
 

sross

Active member
#3
The urls submitted used to always have a graph, for years.. it's been empty for weeks:

Screen Shot 2017-09-29 at 11.55.14 AM.png

Here is the odd error. Even deleting the sitemap and re-adding after i told robots.txt to ignore tags does no good, it still seems to say Sept 9th and throw the error:

Screen Shot 2017-09-29 at 11.55.42 AM.png

Thanks for any help.
 
#7
Also use SEO friendly Url option.

Ur url error shows its not seo friendly

I too had lot of errors in seo

I added robots.txt

Checked the robots.txt in search console for correctness of blocking location

If your site is at /forum then you should disallow /forum/attachments/ - its just example

If its direct at domain
Then

Disallow /attachments/

This will ensure blocking of 403 404 error displayed to robots
 

sross

Active member
#8
I have enabled friendly urls,, what about my robots txt is this ok or should I change something? (forum name removed) Thanks!

User-agent: *
Disallow: /forum/find-new/
Disallow: /forum/conversations/
Disallow: /forum/members/
Disallow: /forum/media/users/
Disallow: /forum/account/
Disallow: /forum/attachments/
Disallow: /forum/goto/
Disallow: /forum/register/
Disallow: /forum/posts/
Disallow: /forum/js/
Disallow: /forum/gallery/
Disallow: /forum/media/
Disallow: /forum/login/
Disallow: /forum/admin.php
Disallow: /forum/credits/
Disallow: /forum/tags
Allow: /

Sitemap: https://www___________com/forum/sitemap.php
 
#9
seems to be ok

had you poor robots.txt in root directory

go to search console
in crawl click on robots.txt tester

it should open your robots.txt and will also show if there is any error

if its showing it what you uploaded then you have successfully blocked the robots from accessing this content and resulting into crawler errors.

If not showing then it means you have not placed it in root directory (it should be placed at ******.com/robots.txt and not at ******.com/forum/robots.txt

do check spelling of your file to its should be robots.txt

if console is showing your robots file then ok

go to fetch as google bot type your site URL then if it crawls successfully, submit it for indexing along with clicking option submit associated urls.

after this google bots will start crawling your site and soon it will accept sitemap and indexing will be done.

Hope this will help you.
 
#10
Keep checking your console for crawl errors and see the urls which gives error.

to fix it either update permissions to user group or block them in robots.txt

Also consider rebuilding sitemap and resubmitting it.

Test the sitemap for correctness before submitting
 

sross

Active member
#12
Well I am still getting the same error, the 403 for tags/dev eventhough I have disallowed this in the robots.txt and updated that. I am unsure why google keeps using this url. My robots.txt validates with no errors. How do I remove specific urls from the sitemap generation? Thanks
 

sross

Active member
#13
here is my current robots.txt

---------------------
User-agent: *
Disallow: /forum/find-new/
Disallow: /forum/conversations/
Disallow: /forum/members/
Disallow: /forum/media/users/
Disallow: /forum/account/
Disallow: /forum/attachments/
Disallow: /forum/goto/
Disallow: /forum/register/
Disallow: /forum/posts/
Disallow: /forum/js/
Disallow: /forum/gallery/
Disallow: /forum/media/
Disallow: /forum/login/
Disallow: /forum/admin.php
Disallow: /forum/credits/
Disallow: /forum/tags/
Disallow: /forum/groupdiscussions/
Allow: /

Sitemap: https://www.__________.com/forum/sitemap.php

---------------------

Notice I have /forum/tags/ to try to stop the sitemap error but it seems to have zero effect. Google persists in saying I have submitted zero links to be indexed.

I have also validated the sitemap, even more odd when testing the sitemap it says successful and zero errors, then I submit and it throws the error.
 

dutchbb

Well-known member
#14
I have the same problem in webmaster but no error. I think it is a google problem, when i check indexing status in webmaster or site: on search everything is ok. I always use those because they are a correct view of what is indexed.
 

sross

Active member
#15
I have the same problem in webmaster but no error. I think it is a google problem, when i check indexing status in webmaster or site: on search everything is ok. I always use those because they are a correct view of what is indexed.
Yeah, I suspect there is no way to get actual support from google either..
 
#17
I have the same problem as well. My sitemap has always worked fine then a couple of weeks ago problems started.

screenshot-www.google.com-2017-09-30-12-27-50.png

Google Search Console is not recognising the 2 child sitemaps and the sitemap is constantly listed as "pending".

I have made no changes in recent times. This happened entirely without warning and for no apparent reason.
 

CTXMedia

Formerly CyclingTribe
#18
I'm experiencing the same as @Joe Blow - if I submit the child sitemaps from within the parent they list okay, but Google is always showing the main sitemap.php file as Pending.

Any idea what's changed or what could be wrong?
 

CTXMedia

Formerly CyclingTribe
#20
@Mike @Chris D any thoughts as to why the sitemap.php file is continually showing as pending and never shows graphs/bars/numbers- whereas when I submit the individual sitemap files (4 of them for CycleChat) they show up just fine and show numbers and data as expected?

Anything changed in the formatting of sitemap.php in XF 1.5.15? Or maybe Google has moved the goalposts?

It worked fine up until recently.