• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

xenforo sitemap.xml ?

Kainzo

Active member
#1
So I'm trying to increase my SEO and whatnot I'm using Woorank.com and it's not able to find my sitemap.xml

Is this a big issue?
http://www.woorank.com/en/www/herocraftonline.com

Is there an easy way to fix this? I current am using the Xenforo Sitemap plugin but not sure if google is even getting my up to date information.

A little new to optimizing website related things so trying to find the best route to take.
 

Slavik

XenForo moderator
Staff member
#2
So I'm trying to increase my SEO and whatnot I'm using Woorank.com and it's not able to find my sitemap.xml

Is this a big issue?
http://www.woorank.com/en/www/herocraftonline.com

Is there an easy way to fix this? I current am using the Xenforo Sitemap plugin but not sure if google is even getting my up to date information.

A little new to optimizing website related things so trying to find the best route to take.

Does it generate the sitemaps correctly, and are you defining it to google?
 

Kainzo

Active member
#3
Does it generate the sitemaps correctly, and are you defining it to google?
Correct, I can see the sitemaps created on website/main/sitemap/files <-- there are several.

I sent it to google and woorank still didnt see it ... I had to push a single sitemap to website/sitemap.xml - and it seems woorank now sees it - but I'm not sure if thats correct.

We used to get 200-300 guests on at a time now we're only see 50-100 - so something may not be synching :(
 

Volion

Active member
#6
I was thinking something changed since the new year, because we also used to get 200-300 guests on at a time now we're only see 50-100. However it looks like your getting indexed. :confused:
 

Volion

Active member
#8
Yah... perhaps we're just not as popular haha.
I moved servers/dns, changed robots.txt and dropped a parked domain. Hoping its just taking Google time to catch up, I may however sign up to woorank, let me know if it helps much. ;)

Also; our new firewall seems to love to block msn crawlers that port scan. :whistle:
 

Kainzo

Active member
#9
I moved servers/dns, changed robots.txt and dropped a parked domain. Hoping its just taking Google time to catch up, I may however sign up to woorank, let me know if it helps much. ;)
Well, woorank is free just to see where you're missing things but it seems pretty good for overall SEO.

What should I have in my robots.txt and where should I have it?
I looked around but it seems different all the time hah...
 

AzzidReign

Well-known member
#11
This is what I have in my robots that will likely help you:

Code:
Sitemap: http://www.****.com/forums/sitemap/sitemap.xml.gz
 
User-agent: Mediapartners-Google
Disallow: /
 
User-agent: BoardReader
Disallow: /
 
User-agent: BoardTracker
Disallow: /
 
User-agent: Gigabot
Disallow: /
 
User-agent: Twiceler
Disallow: /
 
User-agent: dotbot
Disallow: /
 
User-Agent: *
Disallow: /cgi-bin/
Disallow: /_private/
Disallow: /_vti_bin/
Disallow: /_vti_cnf/
Disallow: /_vti_log/
Disallow: /_vti_pvt/
Disallow: /_vti_txt/
Disallow: /forums/find-new/
Disallow: /forums/account/
Disallow: /forums/attachments/
Disallow: /forums/goto/
Disallow: /forums/posts/
Disallow: /forums/login/
Disallow: /forums/admin.php
Disallow: /forums/search/
Disallow: /forums/search
Disallow: /forums/help/
Disallow: /forums/online/
Disallow: /forums/lost-password/
Allow: /
Have the robots in your root folder. Also, some of these SEO things are stupid and they look for the sitemap in the root as well. You can set up a 301 redirect if that is what is happening and have it point to your real sitemap. But in the end, it doesn't really matter bc it's these cheap SEO ranking things that are too stupid to find your sitemap when it has a defined path.
Take the results with a grain of salt.
 

Kainzo

Active member
#12
Here's what I have.... it may be too basic.

# robots.txt for Sites
# Do Not delete this file.

User-agent: *
Disallow: /vbulletin
Disallow: /drupal
Disallow: /img
Disallow: /smf
Disallow: /test
Disallow: /testforum
Allow: /main
Allow: /

User-agent: BecomeBot
Crawl-Delay: 10
 

Weppa333

Active member
#13
there is something weird with XF sitemap and google indeed.

On a new install I did, google perfectly sees the "root" sitemap (the one indexing threads.1.xml and forums.1.xml ) but google NEVER EVER craxls the content of the files referenced in sitemap.xml ( namely, googlebot never downloads sitemap.thread.1.xml.gz )

Even by adding threads.xml manually in the webmaster central, google does not re index it.

I have no idea why, but this is something I just whitnessed
 

Weppa333

Active member
#14
here's a grep for "sitemap" in my server logs.
Pretty clear,


66.249.76.156 - - [18/Feb/2013:02:53:51 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 471 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:02:53:52 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:02:53:52 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:02:53:53 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:03:04:23 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 469 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:03:04:24 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:03:04:24 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:03:04:24 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:07:50:00 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 469 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:07:50:00 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:07:50:01 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:07:50:01 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:08:04:15 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 471 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:08:04:16 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:08:04:17 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:08:04:17 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:09:01:00 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 469 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:09:01:00 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:09:01:01 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:09:01:01 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
 

AzzidReign

Well-known member
#16
there is something weird with XF sitemap and google indeed.

On a new install I did, google perfectly sees the "root" sitemap (the one indexing threads.1.xml and forums.1.xml ) but google NEVER EVER craxls the content of the files referenced in sitemap.xml ( namely, googlebot never downloads sitemap.thread.1.xml.gz )

Even by adding threads.xml manually in the webmaster central, google does not re index it.

I have no idea why, but this is something I just whitnessed
In webmaster tools, does it show the number of pages in sitemap and the number that have been indexed?
 

Weppa333

Active member
#19
Well I'm not sure, google is a very mysterious place... There is nothing wrong with XF ; it's just that "something" at google doesn't seem to like the embedded sitemaps used by the two Sitemap Mods that many people use here.
I encourage everyone to check if googlebot not ony fecthes the "root" sitemap, but also fetches the other sitemaps.

Maybe it's a delay/sandbox of somekind and that it solves itself after a while...

In the meantime, I manually resubmit the root sitemap every now and then.