• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

XF 1.4 sitemap major issues

Echelon

Active member
#1
Is there somebody who can check my sitemap consistency on our platform?

got more then 800 urls but the sitemap only "sees" 190

this is too difficult for me to get a grip on it so it would be nice if someone can check this for me
 

Chris D

XenForo developer
Staff member
#2
It's quite simple, really, there's not much that can go wrong.

How are you calculating 800 URLs?

What content types do you have checked to include in the Sitemap in Admin CP > Options > XML Sitemap?

Do you have any forums that are not visible to guests?
 

Echelon

Active member
#3
It's quite simple, really, there's not much that can go wrong.

How are you calculating 800 URLs?
look here: https://www.google.com/search?filter=0&hl=en&q=site:www.socialmediatalk.nl

What content types do you have checked to include in the Sitemap in Admin CP > Options > XML Sitemap?
All options....5

Do you have any forums that are not visible to guests?
yes. and they are not indexed

extra
does this has something to do with route options because the normal routes are in English and i translated them into Dutch
i.e. see visual below

sitem.png
 

Echelon

Active member
#5
There are many mor URL's on your Forum as th system writes in the sitemap.xml - anything is OK.
So what you say is that the sitemap structure is in good order and i miss no important things?
i do not need to fill in extra urls and xenforo sitemap recognise the routechanges i made?
 

Chris D

XenForo developer
Staff member
#6
Ok.

This is where the confusion lies.

The idea of a sitemap isn't necessarily to tell a search engine about every single URL on your site down to the finest detail, and providing a sitemap doesn't necessarily say to the search engine that they are your ONLY URLs.

If your sitemap contains 100 URLs and Google indexes 1000; there's no problem.

There's a few factors to consider:
  • The Sitemap generation does not include every page of every thread (or every page of every other piece of content), it only includes the URL to the first page.
  • The Sitemap won't include content you have hidden from guests using permissions.
  • The Sitemap won't include URLs from add-ons unless those add-ons have specifically added support for the Sitemap feature, e.g. you have a lot of "tags" pages indexed in Google, but these won't appear in the Sitemap unless that add-on adds support for that.
  • Search engines are typically more than capable of finding their own way around. They will often find URLs you may not have explicitly included in the Sitemap, hence why in the example above it has indexed tags pages even though you don't have a sitemap for them.
  • Search engines may index pages blocked using robots.txt, and may crawl pages specified as "noindex" but exclude the content from search results.
So, with all that considered, your sitemap is normal, your presence in Google is already good.

So what you say is that the sitemap structure is in good order and i miss no important things?
i do not need to fill in extra urls and xenforo sitemap recognise the routechanges i made?
Correct. All is good and you do not need to do anything extra.
 

Chris D

XenForo developer
Staff member
#7
All that being said, your Sitemap does contain some duplicate URLs and I'm not sure why it would.

Code:
<url><loc>http://www.socialmediatalk.nl/databank/google-brand-page-audit.1/</loc><lastmod>2013-10-21T13:46:13+00:00</lastmod></url>
<url><loc>http://www.socialmediatalk.nl/resources/google-brand-page-audit.1/</loc><lastmod>2013-10-21T13:46:13+00:00</lastmod></url>
Those URLs both go to the same page but the only difference is one is the original route, and one is the route you've changed that original route to with a route filter.
 

Echelon

Active member
#8
Ok.

This is where the confusion lies.

The idea of a sitemap isn't necessarily to tell a search engine about every single URL on your site down to the finest detail, and providing a sitemap doesn't necessarily say to the search engine that they are your ONLY URLs.

If your sitemap contains 100 URLs and Google indexes 1000; there's no problem.

There's a few factors to consider:
  • The Sitemap generation does not include every page of every thread (or every page of every other piece of content), it only includes the URL to the first page.
  • The Sitemap won't include content you have hidden from guests using permissions.
  • The Sitemap won't include URLs from add-ons unless those add-ons have specifically added support for the Sitemap feature, e.g. you have a lot of "tags" pages indexed in Google, but these won't appear in the Sitemap unless that add-on adds support for that.
  • Search engines are typically more than capable of finding their own way around. They will often find URLs you may not have explicitly included in the Sitemap, hence why in the example above it has indexed tags pages even though you don't have a sitemap for them.
  • Search engines may index pages blocked using robots.txt, and may crawl pages specified as "noindex" but exclude the content from search results.
So, with all that considered, your sitemap is normal, your presence in Google is already good.


Correct. All is good and you do not need to do anything extra.

Wow! Thanks a lot Chris for debunking my confusion and uncertainty about this sitemap issue. Now I'm satisfied.
 

Echelon

Active member
#9
All that being said, your Sitemap does contain some duplicate URLs and I'm not sure why it would.

Code:
<url><loc>http://www.socialmediatalk.nl/databank/google-brand-page-audit.1/</loc><lastmod>2013-10-21T13:46:13+00:00</lastmod></url>
<url><loc>http://www.socialmediatalk.nl/resources/google-brand-page-audit.1/</loc><lastmod>2013-10-21T13:46:13+00:00</lastmod></url>
Those URLs both go to the same page but the only difference is one is the original route, and one is the route you've changed that original route to with a route filter.
Well...as it seems the first is the translated one from resources [US] to Databank [Dutch]
How to solve this?
 

Chris D

XenForo developer
Staff member
#10
I don't know, it's not a behaviour I would expect to see.

It's not a big problem because Google would understand what's going on as one 301 redirects to the other.

If you go to Admin CP > Tools > Rebuild Caches > Rebuild XML Sitemap, it will rebuild the Sitemap for you. Then I'll check again.

I can't reproduce this myself by adding a Route Filter...
 

Echelon

Active member
#11
I don't know, it's not a behaviour I would expect to see.

It's not a big problem because Google would understand what's going on as one 301 redirects to the other.

If you go to Admin CP > Tools > Rebuild Caches > Rebuild XML Sitemap, it will rebuild the Sitemap for you. Then I'll check again.

I can't reproduce this myself by adding a Route Filter...
Ok. I just rebuild my sitemap.
 

Chris D

XenForo developer
Staff member
#12
Still there...

@Mike might need to answer this because I don't think that's expected but also I can't see where in the code it would be duplicating that.
 

Mike

XenForo developer
Staff member
#14
Have you added URLs manually to the sitemap? The order seems completely inconsistent as well, which is odd given that the sitemap is built by content type and with increasing IDs.

Any other add-ons enabled? Try disabling them and rebuilding the sitemap. Can you list all of the route filters you have?
 

Echelon

Active member
#15
Have you added URLs manually to the sitemap? The order seems completely inconsistent as well, which is odd given that the sitemap is built by content type and with increasing IDs.

Any other add-ons enabled? Try disabling them and rebuilding the sitemap. Can you list all of the route filters you have?
No. I did not added any other url's. Before i used a sitemap addon but deinstalled it.

got 35 routefilters...how do i list them since i cannot select the text
 

Chris D

XenForo developer
Staff member
#16
Have you added URLs manually to the sitemap?
That crossed my mind too, but they have identical lastmod dates so I assumed they were coming from the same sitemap handler.

got 35 routefilters...how do i list them since i cannot select the text
Screenshot, or, alternatively, if you're comfortable running queries on the database, run this query and copy and paste (or screenshot) the output:

Code:
SELECT * FROM xf_route_filter WHERE enabled = 1
 

Echelon

Active member
#17
That crossed my mind too, but they have identical lastmod dates so I assumed they were coming from the same sitemap handler.


Screenshot, or, alternatively, if you're comfortable running queries on the database, run this query and copy and paste (or screenshot) the output:

Code:
SELECT * FROM xf_route_filter WHERE enabled = 1
2.png 1.png
 

Chris D

XenForo developer
Staff member
#18
One point you missed from Mike's post is the question about add-ons. Clearly you have a lot of add-ons.

As a temporary measure, please disable all add-ons except XenForo Resource Manager and generate the Sitemap again.

As soon as you have generated the sitemap, you can re-enable all add-ons.
 

Echelon

Active member
#19
One point you missed from Mike's post is the question about add-ons. Clearly you have a lot of add-ons.

As a temporary measure, please disable all add-ons except XenForo Resource Manager and generate the Sitemap again.

As soon as you have generated the sitemap, you can re-enable all add-ons.
yes....84 addons minus the ones we don't need anymore because it's integrated in xenforo 1.4 ;-)

Ok. i will disable all the addons except resource manager tonight,
then generate a new sitemap,
then activate all the addons again

sorry for the fuzz but im not technical
 

Echelon

Active member
#20
yes....84 addons minus the ones we don't need anymore because it's integrated in xenforo 1.4 ;-)

Ok. i will disable all the addons except resource manager tonight,
then generate a new sitemap,
then activate all the addons again

sorry for the fuzz but im not technical
at this moment i disabled all addons and created a new sitemap....what to do next?