Better Sitemap Magement

BT10

Active member
While it's great that XenForo includes the ability to build and submit forum's sitemap, I think improvements can be made. I recently came across this article by Oliver H.G. Mason, where he talks about optimizing his XML Sitemap.

A great read from Oliver H.G. Mason on his findings on splitting XML sitemaps into smaller chunks. Instead of a few XML Sitemaps capped at 50,000 URLs the magic number Google recommends, he decided to go with many XML Sitemaps containing 10,000 URLs.

Some SEOs recommend 1,000 URL Sitemaps so they can get around Search Console limits so it can be used to monitor which URLs aren’t getting indexed.

I have found that limiting sitemaps to only 10,000 URLs leads to more thorough levels of indexing. I’m not sure why – I suspect that smaller lists of URLs are easier for Google to process and crawl – but it’s been proven time and again that smaller sitemaps lead to higher degrees of indexing.
To my knowledge, in XenForo, all of our URLs are being included in a single sitemap.php file.

If you are an owner of a big forum, how do you manage your sitemap?

Suggestion: each forum and nodes will have its own sitemap link?
 
Upvote 3

Brogan

XenForo moderator
Staff member
It's done automatically once there are 50,000 entries in the sitemap.

There's no configurable option for it,.
 

BT10

Active member
@Brogan Is it possible to have a configurable amount (the ability to change from 50,000 to another value?) I would like to reduce that amount and have multiple sitemap files generated.
 

arn

Well-known member
Interesting. Seems others think it also makes a difference, even though I'm not sure why it would.

I've separated our sitemap out into 1000 url sitemaps. So we can see what happens.

arn
 

Anatoliy

Well-known member
Interesting. Seems others think it also makes a difference, even though I'm not sure why it would.

I've separated our sitemap out into 1000 url sitemaps. So we can see what happens.

arn
You really believe that Google's crawlers need help from webmasters to discover their pages these days? )
They start watching you as soon as you register a new domain. ))
I have a site with 20k pages. Without a sitemap at all. All the pages are listed.
 

arn

Well-known member
You really believe that Google's crawlers need help from webmasters to discover their pages these days? )
They start watching you as soon as you register a new domain. ))
I have a site with 20k pages. Without a sitemap at all. All the pages are listed.

I do think a sitemap helps for discoverability. Sites, especially forums, are a bit of a mess for crawlers with lots of dead-end links (member pages) and loops.

Though I wouldn't think sitemap being split into more files or not would make a difference. But people seemed to think so based on the link above and some googling.

It seemed easy enough to test. Though when looking closer at it, I do think we should probably remove some really old pages from our sitemap... but I didn't want to make too many changes at once. So I'll let this settle for a month or so and see if there's any noticeable changes.

arn
 

Anatoliy

Well-known member
Though when looking closer at it, I do think we should probably remove some really old pages from our sitemap
Accordingly to Search Console, top 10 pages that bring me traffic are 10 year old threads.
I made AMP copies for 4 of them (thank you one more time), it took google one day to discover them and start to show them for those who search from mobiles.
That's how a non amp page looks in search results.

0.jpg

And that's how it's amp version.

1.jpg

2.JPEG

So I think, that webmasters should believe big G that content is a king, and they just need to follow big G guidelines, instead of waisting time trying to perform some voodoo by splitting sitemap in equal piles. )
But I could be wrong, of course.
 

arn

Well-known member
yeah... like I said, I don't know if I believe it. It's just easy to test.

Regrading AMP. You seen better positioning or CTR on those AMP pages from before and after?
 

Anatoliy

Well-known member
Regrading AMP. You seen better positioning or CTR on those AMP pages from before and after?
those 4 pages had 1st places for their key phrases, so amp can't do better. ) waiting at least for a week to compare CTR. but I don't expect much.
those pages that are second and lower, that's where I expect amps really bump it up.
But after I created 4 pages by hands I clear understood that I should better start building a php script that will process all the pages.
even better if that script will be build by xenForo team. )
So @BT10 vote for amp and you will not care about a sitemap.
 

arn

Well-known member
I mean it's too early to say it's more than natural variation, but... it moved a little. Arrow is when change was made.

973,000 -> 1,020,590

but highest recorded back in April was 1,083,322

Screen Shot 2020-07-14 at 9.46.14 PM.png
 

arn

Well-known member
@arn What did you change for this to happen?
 

arn

Well-known member
so I think it did bump up our index numbers, but didn't help our traffic. I'm gonna end this experiment because I want to make more changes to our sitemap, and prune it more. But I do think it increased our indexing.

Screen Shot 2020-08-01 at 8.19.57 AM.png
 
Top