XF 1.5 Sitemap :no individual pages ?

Weppa333

Weppa333

Well-known member
Hello

I just noticed on my install, but also on this website, that the sitemaps do not include individual pages (above the first page obviously).
When sitemap was an external add-on, it was the case (all the pages were included).

Basically, a lot of canonical urls are missing from XF sitemaps if they don't include pages ; why is it so ?
 
Sort by date Sort by votes
Ultimately the benefit of including page URLs is limited. Once a search engine finds the thread, there's plenty of hints from there as to what other content should be crawled, including metadata in the source itself:
Code: 
<link rel="next" href="threads/my-forums-getting-lots-of-spam.35195/page-2" />
 
Upvote 0 Downvote
Well it's your opinion :)
I'm not sure many people realized this.

Also, in the google sitemap tools, you see the difference between "submitted urls" and "crawled urls". With your way of doing things, this tools makes no sense at all anymore since you don't submit every URL.

I'll edit the sitemap deferred task to include all the pages, and I'm sure many people here expect this behaviour (they can like my post if they do :) )
 
Upvote 0 Downvote
I suggest we agree to disagree.

You absolutely do not know how google handles sitemaps. Neither do I.
What you describe over here is the way google bot works without sitemaps (it discovers page after page), or with a partial sitemap.

With a full sitemap officially submitted in GWT, it's likely google "jumps" directly to "not indexed content". I could prove this to you but I'm not in the business of creating forum software and I suggest you reproduce this in a lab to make a proper decision.

Also, as I said, you've got stats at GWT that make no sense with your way of doing sitemaps (which none else does this way)
And finally, your website looks less "active" to google GWT since your sitemap only changes when a new thread is submitted.

It's a bad decision.
 
Upvote 0 Downvote
I know this thread is very old. I've used the external Sitemap AddOn (with all pages of threads) for a long time. I've switched to use the internal sitemap create almost 2 years ago (February 2016). Around that time, i also changed a lot in my navigation.

I've around 65% of traffic in 3 weeks. I paniced and i've undone all the changes (except sitemap cause i've never ever thought this could have caused the problem cause i've thought the internal sitemap create works same as the AddOn) but nothing changed. In december 2016 (still very concerned about the lost traffic) i've stumbled around the sitemap and realized that the new sitemap includes way less URLs then the old sitemap from the backup. I disabled the internal sitemap creation and used the AddOn again. My traffic went up within weeks.

1517131978969.webp
Screenshot of searchmetrics SEO visibility of that timeframe

Why am i posting this now? I would like to get rid of the AddOn. Less Addons are better, specially when you have the same functions build in. I would like to include a lot of other pages of AddOns in the sitemap (like Showcase,...) but that's not possible with the external AddOn. At least not for me.

So please consider adding the option to include all pages of a thread (and maybe also forum). In the meantime, maybe there is a developer out there who could create a custom modification for the XF1 sitemap for me. :) This is SOO important for me!
 
Upvote 0 Downvote
Weppa333 said:
I suggest we agree to disagree.

You absolutely do not know how google handles sitemaps. Neither do I.
What you describe over here is the way google bot works without sitemaps (it discovers page after page), or with a partial sitemap.

With a full sitemap officially submitted in GWT, it's likely google "jumps" directly to "not indexed content". I could prove this to you but I'm not in the business of creating forum software and I suggest you reproduce this in a lab to make a proper decision.

Also, as I said, you've got stats at GWT that make no sense with your way of doing sitemaps (which none else does this way)
And finally, your website looks less "active" to google GWT since your sitemap only changes when a new thread is submitted.

It's a bad decision.
Click to expand...

This would be a lot of pages to send with a 10mil posts forum for example. Also pages in between are not necessary then, only first and last page i think. Google just needs to detect when a thread has been updated.
 
Upvote 0 Downvote
Crazy-Achmet said:
I know this thread is very old. I've used the external Sitemap AddOn (with all pages of threads) for a long time. I've switched to use the internal sitemap create almost 2 years ago (February 2016). Around that time, i also changed a lot in my navigation.

I've around 65% of traffic in 3 weeks. I paniced and i've undone all the changes (except sitemap cause i've never ever thought this could have caused the problem cause i've thought the internal sitemap create works same as the AddOn) but nothing changed. In december 2016 (still very concerned about the lost traffic) i've stumbled around the sitemap and realized that the new sitemap includes way less URLs then the old sitemap from the backup. I disabled the internal sitemap creation and used the AddOn again. My traffic went up within weeks.

View attachment 167624
Screenshot of searchmetrics SEO visibility of that timeframe

Why am i posting this now? I would like to get rid of the AddOn. Less Addons are better, specially when you have the same functions build in. I would like to include a lot of other pages of AddOns in the sitemap (like Showcase,...) but that's not possible with the external AddOn. At least not for me.

So please consider adding the option to include all pages of a thread (and maybe also forum). In the meantime, maybe there is a developer out there who could create a custom modification for the XF1 sitemap for me. :)This is SOO important for me!
Click to expand...

Which add-on are you using?
 
Upvote 0 Downvote
Last edited:
Upvote 0 Downvote
Crazy-Achmet said:
I know this thread is very old. I've used the external Sitemap AddOn (with all pages of threads) for a long time. I've switched to use the internal sitemap create almost 2 years ago (February 2016). Around that time, i also changed a lot in my navigation.

I've around 65% of traffic in 3 weeks. I paniced and i've undone all the changes (except sitemap cause i've never ever thought this could have caused the problem cause i've thought the internal sitemap create works same as the AddOn) but nothing changed. In december 2016 (still very concerned about the lost traffic) i've stumbled around the sitemap and realized that the new sitemap includes way less URLs then the old sitemap from the backup. I disabled the internal sitemap creation and used the AddOn again. My traffic went up within weeks.

View attachment 167624
Screenshot of searchmetrics SEO visibility of that timeframe

Why am i posting this now? I would like to get rid of the AddOn. Less Addons are better, specially when you have the same functions build in. I would like to include a lot of other pages of AddOns in the sitemap (like Showcase,...) but that's not possible with the external AddOn. At least not for me.

So please consider adding the option to include all pages of a thread (and maybe also forum). In the meantime, maybe there is a developer out there who could create a custom modification for the XF1 sitemap for me. :) This is SOO important for me!
Click to expand...
Interesting. Did you decide on a solution?
 
Upvote 0 Downvote
I just discovered, in horror, that Google is not indexing a huge bulk (more than 90%) of my multi-page threads. These threads have LOTS of valuable content, you could write several technical books from everything written there, and yet all these years, they have been getting zero search traffic. That means they have only been read by the threads' followers. I went into the Google Search Console and confirmed that Google has crawled but not indexed a HUGE number of pages and seemingly has no intention of doing so, because they are not in the sitemap.

This is a serious oversight. XenForo basically assumed in designing this tool that most forums act like their own community, which mostly consists of a huge number of short threads. That's not the case with some forums! Some forums have only a few active threads with thousands of pages, and Google will NOT index that content without the webmaster's help.

Please include an option for multiple pages in the official sitemap generation tool. In the meantime, I'm going to need to write a script to make up for all the thread pages XenForo has missed. Damn.
 
Upvote 0 Downvote
+1

Now I finally understand why only a fraction of my pages are indexed by Google. I always thought eevrything would be included in The XF sitemap.

We really need a sitemap, that allows us as admins to decide how much detail/depth is included in that sitemap! I much prefer a sitemap that has all in it than the solution available at the moment.
 
Upvote 0 Downvote
JackieChun said:
I went into the Google Search Console and confirmed that Google has crawled but not indexed a HUGE number of pages and seemingly has no intention of doing so, because they are not in the sitemap.
Click to expand...

If Googlebot has crawled the pages, it knows about them already. Putting them in a sitemap won't change anything.
 
Upvote 0 Downvote
JackieChun said:
That's absolutely false. Crawling and indexing are two different concepts. You signal to Google which pages you wish indexed by submitting them in a sitemap.
Click to expand...
You shouldn't use words like "absolutely" unless you've read the actual Google docs & support pages.

Page Indexing report - Search Console Help

See which pages Google can find and index on your site, and learn about any indexing problems encountered. Open Page Indexing report
support.google.com
Crawled - currently not indexed: The page was crawled by Google, but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling.
Click to expand...

support.google.com

What Is a Sitemap | Google Search Central | Documentation | Google for Developers

A sitemap provides information that helps Google more intelligently crawl your site. Discover how a sitemap works and determine if you need one.
support.google.com support.google.com
Do I need a sitemap?
If your site’s pages are properly linked, our web crawlers can usually discover most of your site. Even so, a sitemap can improve the crawling of your site, particularly if your site meets one of the following criteria:
Click to expand...
 
Upvote 0 Downvote
Kevin said:
You shouldn't use words like "absolutely"
Click to expand...

I use the term "absolutely" when I know that something is absolutely false.

Kevin said:
unless you've read the actual Google docs & support pages.
Click to expand...

I've been doing that for almost two decades now, in addition to attending conferences and talking to Google employees. Google is no longer interested in revealing exactly what they do and do not index, because SEO is now a $100 billion industry and they don't want to be gamed by people looking to make easy money with poor content.

Kerstin28 said:
You signal pages you wish to be crawled. Google indexes pages based on their criteria and not on your wishes.
Click to expand...

That's a falsedy falsehood. As a site owner, you absolutely have ways to tell Google what you want indexed (in addition to crawled), including literally asking Google to index individual pages. Screenshot directly from the Google Search Console.

8zsQ5U4.png
 
Upvote 0 Downvote
You must log in or register to reply here.

Similar threads

Sal Collaziano
  • Question Question
XF 2.2 New reason prevents pages in a sitemap from being indexed on Google
Replies
5
Views
96
Miri
M
bzcomputers
Not a bug Incorrect og:url and lack of sitemap inclusion of canonical url for secondary thread pages - affecting SEO
Replies
1
Views
644
Mike
Mike
rickhz
  • Question Question
XF 2.1 Sitemap not creating split pages
Replies
5
Views
675
rickhz
rickhz
tomwood
  • Question Question
XF 2.1 Set Canonical URL for individual threads in Xenforo that point to content posted on Medium?
Replies
3
Views
699
Mr Lucky
Mr Lucky
RobinHood
  • Suggestion Suggestion
Lack of interest Allow display conditions or user group permissions for individual help pages
Replies
0
Views
469
RobinHood
RobinHood
Back
Top Bottom