XF 2.3 How to exclude older pages from our sitemap?

[Stuart Wright]

One major finding by our commercial team is that Google is electing not to index half of all pages in our sitemap due to the age of those pages (nothing before 2010). This degrades Google's trust in the sitemap and wastes crawl budget.
Do you know of any sitemap tools within xenforo that can help me set conditional rules about what gets submitted in the sitemap?

 

[Stuart Wright]

Ok, so I found the option in forum admin Allow search engine indexing set to Index threads based on criteria.
And then set the criteria to Thread was replied to no more than X days ago.
Huge job to set this for all 200 forums, though.
I'd like to run this query:

update xf_forum set allow_index='criteria', index_criteria = '{"max_days_last_post":5400,"is_sticky":false,"is_article":true,"is_solved_question":true,"first_post_staff":false}';

Any reason not to? I'm guessing it would be ok.

 

[Mr Lucky]

This degrades Google's trust in the sitemap

How do you know this?

and wastes crawl budget.

That is possible, and I can understand it would be preferable if you had control over what is excluded due to crawl budget. But I find it odd that it's threads oler than 2010, was everything written on the internet back then now worthless (unless it got gobbled up by AI bots and vomited out again?

 

[Stuart Wright]

How do you know this?

That is possible, and I can understand it would be preferable if you had control over what is excluded due to crawl budget. But I find it odd that it's threads oler than 2010, was everything written on the internet back then now worthless (unless it got gobbled up by AI bots and vomited out again?

My expert told me.
1. Improve Google's trust in our sitemap (=more crawling, more urls indexed and ranking)
2. Increase Google's capacity to crawl the site (crawl budget) as it's not wasting crawls on what it considers junk

However, this issue seems to be happening to us, so we need to address it.

XF 2.2 Thread 'GSC only reads in the first 9 sitemap files...'

Oct 25, 2022

Bit of an odd one this, but it's basically as the title says.

My total sitemap size is about 730,000 pages - so this is split up into 15 sitemap files (sitemap-1.xml > sitemap-15.xml) however when Google search console reads in the main sitemap.xml index file, it only picks up the first 9 - totalling 450,000 pages.

There doesn't seem to be any rhyme or reason for this - just wondered if anyone else had come across this before or any ideas how to solve it?

I did try adding the sitemap again as the .php version just to see if that made a difference, but nope, still just the first 9...

• benFF
• Replies: 14
• Forum: Troubleshooting and problems

• 

 

[briansol]

90% of my SE traffic hits threads from 2003-2009. I don't think turning it off is a great idea, but i also don't have the thread count you do, so the use case is different.

un-sure on the query.

but old doesn't always mean junk.

your offtopic board is probably always junk....