XF 1.5 Google not indexing all pages/ sitemap.php issue

[snoopy5]

Hi,

I have a couple of different forums. All are XF1.5.21. All are older than 4 years. All have the same setup on the same server. I have created on all the same way a sitemap and the cronjob for it and put the sitemap.php link into Google search years ago.

But two of them differ dramatically in Google Search for getting indexed. They do get indexed, but only around 15-20% of all submitted pages. All other sites have somethimg like 98%.

According to Google, the test with the sitemap.php is o.k.

Now I looked into this and whe I type into my browser the path to the sitemap.php file, I get for those 2 "problem.kids" different results in the way how it is displayed than with the well indexed sites. See screenshots.

What can be the reason that they look so different? Is there any setting I did wrong?

The good ones:




the bad ones:

 

[snoopy5]

anybody?

 

[AndyB]

Don't use a sitemap as it's not needed.

 

[snoopy5]

Don't use a sitemap as it's not needed.

???

How do you mean that?

Google recommends to give them a sitemap link and XF is offering a cron job for this. So it should have a positive effect?

 

[AndyB]

I find that new content is indexed in a very short time without the need for a sitemap.

Do I need a sitemap?

If your site’s pages are properly linked, our web crawlers can usually discover most of your site.

Source: https://support.google.com/webmasters/answer/156184?hl=en

 

[snoopy5]

o.k., thanks. But I want to use a sitemap for various reasons.

And I need to know why the sitemap gives with two of my sites such a strange output as shown in the screenshot.

 

[snoopy5]

o.k., thanks. But I want to use a sitemap for various reasons.

And I need to know why the sitemap gives with two of my sites such a strange output as shown in the screenshot.

Anybody?

 

[Chris D]

What’s wrong with the screenshot exactly?

The first screenshot looks like that because it isn’t fully loaded yet.

The second screenshot has loaded successfully.

 

[snoopy5]

What’s wrong with the screenshot exactly?

The first screenshot looks like that because it isn’t fully loaded yet.

The second screenshot has loaded successfully.

ah, o.k., so the second one is the one which is actually "o.k."?

Then my thesis that the reason for Google not indexing all sites is in a wrong setting of my site.php is not true.

What else could be a reason that Google is indexing only a fraction of a forum (10-20%)? Any common mistakes an admin can do in XF to cause that?

 

[Chris D]

Yep the second version is correct. Most browsers apply some default styling to XML documents when they are fully loaded. The first example is just indicative of your browser not having downloaded the full file yet. But that’s really nothing to worry about. Most sitemap files can be as large as 50MB IIRC, though XF does limit them to a max of 10MB. It takes quite a significant amount of time to download and render such a document, but I’m sure Google’s crawler is more than able to cope.

Google will only index pages it can access plus the figure in relation to the sitemap entries may not include pages that it already indexed naturally.

The only mistake that could be made really is if you’re blocking certain things with robots.txt but that seems unlikely.

I really wouldn’t worry about any discrepancy here.