XF 1.5 Sitemap :no individual pages ?

Weppa333

Active member
Hello

I just noticed on my install, but also on this website, that the sitemaps do not include individual pages (above the first page obviously).
When sitemap was an external add-on, it was the case (all the pages were included).

Basically, a lot of canonical urls are missing from XF sitemaps if they don't include pages ; why is it so ?
 

Chris D

XenForo developer
Staff member
Ultimately the benefit of including page URLs is limited. Once a search engine finds the thread, there's plenty of hints from there as to what other content should be crawled, including metadata in the source itself:
Code:
<link rel="next" href="threads/my-forums-getting-lots-of-spam.35195/page-2" />
 

Weppa333

Active member
Well it's your opinion :)
I'm not sure many people realized this.

Also, in the google sitemap tools, you see the difference between "submitted urls" and "crawled urls". With your way of doing things, this tools makes no sense at all anymore since you don't submit every URL.

I'll edit the sitemap deferred task to include all the pages, and I'm sure many people here expect this behaviour (they can like my post if they do :) )
 

Weppa333

Active member
I suggest we agree to disagree.

You absolutely do not know how google handles sitemaps. Neither do I.
What you describe over here is the way google bot works without sitemaps (it discovers page after page), or with a partial sitemap.

With a full sitemap officially submitted in GWT, it's likely google "jumps" directly to "not indexed content". I could prove this to you but I'm not in the business of creating forum software and I suggest you reproduce this in a lab to make a proper decision.

Also, as I said, you've got stats at GWT that make no sense with your way of doing sitemaps (which none else does this way)
And finally, your website looks less "active" to google GWT since your sitemap only changes when a new thread is submitted.

It's a bad decision.
 

Crazy-Achmet

Well-known member
I know this thread is very old. I've used the external Sitemap AddOn (with all pages of threads) for a long time. I've switched to use the internal sitemap create almost 2 years ago (February 2016). Around that time, i also changed a lot in my navigation.

I've around 65% of traffic in 3 weeks. I paniced and i've undone all the changes (except sitemap cause i've never ever thought this could have caused the problem cause i've thought the internal sitemap create works same as the AddOn) but nothing changed. In december 2016 (still very concerned about the lost traffic) i've stumbled around the sitemap and realized that the new sitemap includes way less URLs then the old sitemap from the backup. I disabled the internal sitemap creation and used the AddOn again. My traffic went up within weeks.

1517131978969.png
Screenshot of searchmetrics SEO visibility of that timeframe

Why am i posting this now? I would like to get rid of the AddOn. Less Addons are better, specially when you have the same functions build in. I would like to include a lot of other pages of AddOns in the sitemap (like Showcase,...) but that's not possible with the external AddOn. At least not for me.

So please consider adding the option to include all pages of a thread (and maybe also forum). In the meantime, maybe there is a developer out there who could create a custom modification for the XF1 sitemap for me. :) This is SOO important for me!
 

dutchbb

Well-known member
I suggest we agree to disagree.

You absolutely do not know how google handles sitemaps. Neither do I.
What you describe over here is the way google bot works without sitemaps (it discovers page after page), or with a partial sitemap.

With a full sitemap officially submitted in GWT, it's likely google "jumps" directly to "not indexed content". I could prove this to you but I'm not in the business of creating forum software and I suggest you reproduce this in a lab to make a proper decision.

Also, as I said, you've got stats at GWT that make no sense with your way of doing sitemaps (which none else does this way)
And finally, your website looks less "active" to google GWT since your sitemap only changes when a new thread is submitted.

It's a bad decision.
This would be a lot of pages to send with a 10mil posts forum for example. Also pages in between are not necessary then, only first and last page i think. Google just needs to detect when a thread has been updated.
 

dutchbb

Well-known member
I know this thread is very old. I've used the external Sitemap AddOn (with all pages of threads) for a long time. I've switched to use the internal sitemap create almost 2 years ago (February 2016). Around that time, i also changed a lot in my navigation.

I've around 65% of traffic in 3 weeks. I paniced and i've undone all the changes (except sitemap cause i've never ever thought this could have caused the problem cause i've thought the internal sitemap create works same as the AddOn) but nothing changed. In december 2016 (still very concerned about the lost traffic) i've stumbled around the sitemap and realized that the new sitemap includes way less URLs then the old sitemap from the backup. I disabled the internal sitemap creation and used the AddOn again. My traffic went up within weeks.

View attachment 167624
Screenshot of searchmetrics SEO visibility of that timeframe

Why am i posting this now? I would like to get rid of the AddOn. Less Addons are better, specially when you have the same functions build in. I would like to include a lot of other pages of AddOns in the sitemap (like Showcase,...) but that's not possible with the external AddOn. At least not for me.

So please consider adding the option to include all pages of a thread (and maybe also forum). In the meantime, maybe there is a developer out there who could create a custom modification for the XF1 sitemap for me. :)This is SOO important for me!
Which add-on are you using?
 

dutchbb

Well-known member
Last edited:

upnet

Active member
I know this thread is very old. I've used the external Sitemap AddOn (with all pages of threads) for a long time. I've switched to use the internal sitemap create almost 2 years ago (February 2016). Around that time, i also changed a lot in my navigation.

I've around 65% of traffic in 3 weeks. I paniced and i've undone all the changes (except sitemap cause i've never ever thought this could have caused the problem cause i've thought the internal sitemap create works same as the AddOn) but nothing changed. In december 2016 (still very concerned about the lost traffic) i've stumbled around the sitemap and realized that the new sitemap includes way less URLs then the old sitemap from the backup. I disabled the internal sitemap creation and used the AddOn again. My traffic went up within weeks.

View attachment 167624
Screenshot of searchmetrics SEO visibility of that timeframe

Why am i posting this now? I would like to get rid of the AddOn. Less Addons are better, specially when you have the same functions build in. I would like to include a lot of other pages of AddOns in the sitemap (like Showcase,...) but that's not possible with the external AddOn. At least not for me.

So please consider adding the option to include all pages of a thread (and maybe also forum). In the meantime, maybe there is a developer out there who could create a custom modification for the XF1 sitemap for me. :) This is SOO important for me!
Interesting. Did you decide on a solution?
 

JackieChun

Active member
I just discovered, in horror, that Google is not indexing a huge bulk (more than 90%) of my multi-page threads. These threads have LOTS of valuable content, you could write several technical books from everything written there, and yet all these years, they have been getting zero search traffic. That means they have only been read by the threads' followers. I went into the Google Search Console and confirmed that Google has crawled but not indexed a HUGE number of pages and seemingly has no intention of doing so, because they are not in the sitemap.

This is a serious oversight. XenForo basically assumed in designing this tool that most forums act like their own community, which mostly consists of a huge number of short threads. That's not the case with some forums! Some forums have only a few active threads with thousands of pages, and Google will NOT index that content without the webmaster's help.

Please include an option for multiple pages in the official sitemap generation tool. In the meantime, I'm going to need to write a script to make up for all the thread pages XenForo has missed. Damn.
 

snoopy5

Well-known member
+1

Now I finally understand why only a fraction of my pages are indexed by Google. I always thought eevrything would be included in The XF sitemap.

We really need a sitemap, that allows us as admins to decide how much detail/depth is included in that sitemap! I much prefer a sitemap that has all in it than the solution available at the moment.
 

Kerstin28

Member
I went into the Google Search Console and confirmed that Google has crawled but not indexed a HUGE number of pages and seemingly has no intention of doing so, because they are not in the sitemap.
If Googlebot has crawled the pages, it knows about them already. Putting them in a sitemap won't change anything.
 

JackieChun

Active member
If Googlebot has crawled the pages, it knows about them already. Putting them in a sitemap won't change anything.
That's absolutely false. Crawling and indexing are two different concepts. You signal to Google which pages you wish indexed by submitting them in a sitemap.
 

Kevin

Well-known member
That's absolutely false. Crawling and indexing are two different concepts. You signal to Google which pages you wish indexed by submitting them in a sitemap.
You shouldn't use words like "absolutely" unless you've read the actual Google docs & support pages.

Crawled - currently not indexed: The page was crawled by Google, but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling.
Do I need a sitemap?
If your site’s pages are properly linked, our web crawlers can usually discover most of your site. Even so, a sitemap can improve the crawling of your site, particularly if your site meets one of the following criteria:
 

JackieChun

Active member
You shouldn't use words like "absolutely"
I use the term "absolutely" when I know that something is absolutely false.

unless you've read the actual Google docs & support pages.
I've been doing that for almost two decades now, in addition to attending conferences and talking to Google employees. Google is no longer interested in revealing exactly what they do and do not index, because SEO is now a $100 billion industry and they don't want to be gamed by people looking to make easy money with poor content.

You signal pages you wish to be crawled. Google indexes pages based on their criteria and not on your wishes.
That's a falsedy falsehood. As a site owner, you absolutely have ways to tell Google what you want indexed (in addition to crawled), including literally asking Google to index individual pages. Screenshot directly from the Google Search Console.

 
Top