Lack of interest Better Sitemap Magement

[BT10]

While it's great that XenForo includes the ability to build and submit forum's sitemap, I think improvements can be made. I recently came across this article by Oliver H.G. Mason, where he talks about optimizing his XML Sitemap.

A great read from Oliver H.G. Mason on his findings on splitting XML sitemaps into smaller chunks. Instead of a few XML Sitemaps capped at 50,000 URLs the magic number Google recommends, he decided to go with many XML Sitemaps containing 10,000 URLs.

Some SEOs recommend 1,000 URL Sitemaps so they can get around Search Console limits so it can be used to monitor which URLs aren’t getting indexed.

I have found that limiting sitemaps to only 10,000 URLs leads to more thorough levels of indexing. I’m not sure why – I suspect that smaller lists of URLs are easier for Google to process and crawl – but it’s been proven time and again that smaller sitemaps lead to higher degrees of indexing.

To my knowledge, in XenForo, all of our URLs are being included in a single sitemap.php file.

If you are an owner of a big forum, how do you manage your sitemap?

Suggestion: each forum and nodes will have its own sitemap link?

 

[Paul B]

The sitemap is split into multiple files for larger sites.

https://xenforo.com/community/sitemap.php

 

[BT10]

@Brogan How do you guys define "larger sites"? How do we activate this feature even if our forum isn't large enough?

 

[Paul B]

It's done automatically once there are 50,000 entries in the sitemap.

There's no configurable option for it,.

 

[BT10]

@Brogan Is it possible to have a configurable amount (the ability to change from 50,000 to another value?) I would like to reduce that amount and have multiple sitemap files generated.

 

[Paul B]

You can post a suggestion here: https://xenforo.com/community/forums/xenforo-suggestions.18/

 

[arn]

Interesting. Seems others think it also makes a difference, even though I'm not sure why it would.

I've separated our sitemap out into 1000 url sitemaps. So we can see what happens.

arn

 

[Anatoliy]

Interesting. Seems others think it also makes a difference, even though I'm not sure why it would.

I've separated our sitemap out into 1000 url sitemaps. So we can see what happens.

arn

You really believe that Google's crawlers need help from webmasters to discover their pages these days? )
They start watching you as soon as you register a new domain. ))
I have a site with 20k pages. Without a sitemap at all. All the pages are listed.

 

[arn]

You really believe that Google's crawlers need help from webmasters to discover their pages these days? )
They start watching you as soon as you register a new domain. ))
I have a site with 20k pages. Without a sitemap at all. All the pages are listed.

I do think a sitemap helps for discoverability. Sites, especially forums, are a bit of a mess for crawlers with lots of dead-end links (member pages) and loops.

Though I wouldn't think sitemap being split into more files or not would make a difference. But people seemed to think so based on the link above and some googling.

It seemed easy enough to test. Though when looking closer at it, I do think we should probably remove some really old pages from our sitemap... but I didn't want to make too many changes at once. So I'll let this settle for a month or so and see if there's any noticeable changes.

arn

 

[Anatoliy]

Though when looking closer at it, I do think we should probably remove some really old pages from our sitemap

Accordingly to Search Console, top 10 pages that bring me traffic are 10 year old threads.
I made AMP copies for 4 of them (thank you one more time), it took google one day to discover them and start to show them for those who search from mobiles.
That's how a non amp page looks in search results.



And that's how it's amp version.





So I think, that webmasters should believe big G that content is a king, and they just need to follow big G guidelines, instead of waisting time trying to perform some voodoo by splitting sitemap in equal piles. )
But I could be wrong, of course.

 

[arn]

yeah... like I said, I don't know if I believe it. It's just easy to test.

Regrading AMP. You seen better positioning or CTR on those AMP pages from before and after?

 

[Anatoliy]

Regrading AMP. You seen better positioning or CTR on those AMP pages from before and after?

those 4 pages had 1st places for their key phrases, so amp can't do better. ) waiting at least for a week to compare CTR. but I don't expect much.
those pages that are second and lower, that's where I expect amps really bump it up.
But after I created 4 pages by hands I clear understood that I should better start building a php script that will process all the pages.
even better if that script will be build by xenForo team. )
So @BT10 vote for amp and you will not care about a sitemap.

 

[arn]

I mean it's too early to say it's more than natural variation, but... it moved a little. Arrow is when change was made.

973,000 -> 1,020,590

but highest recorded back in April was 1,083,322

 

[briansol]

i changed mine to 10000 urls (just hard-coded it in the Builder.php file.

 

[arn]

coverage going up, but not helping impressions. Also noticed a big increase in crawl rate from google

 

[briansol]

i'm not seeing anything significant yet.

 

[Anatoliy]

noticed a big increase in crawl rate from google

it's possible that it actively re-crawls everything after algorithm change. mine is already back to usual.

 

[BT10]

@arn What did you change for this to happen?

 

[arn]

@arn What did you change for this to happen?

Lack of interest - Better Sitemap Magement

While it's great that XenForo includes the ability to build and submit forum's sitemap, I think improvements can be made. I recently came across this article by Oliver H.G. Mason, where he talks about optimizing his XML Sitemap. A great read from Oliver H.G. Mason on his findings on splitting...

xenforo.com

 

[arn]

so I think it did bump up our index numbers, but didn't help our traffic. I'm gonna end this experiment because I want to make more changes to our sitemap, and prune it more. But I do think it increased our indexing.