xenforo sitemap.xml ?

Kainzo

Kainzo

Active member
So I'm trying to increase my SEO and whatnot I'm using Woorank.com and it's not able to find my sitemap.xml

Is this a big issue?
http://www.woorank.com/en/www/herocraftonline.com

Is there an easy way to fix this? I current am using the Xenforo Sitemap plugin but not sure if google is even getting my up to date information.

A little new to optimizing website related things so trying to find the best route to take.
 
Kainzo said:
So I'm trying to increase my SEO and whatnot I'm using Woorank.com and it's not able to find my sitemap.xml

Is this a big issue?
http://www.woorank.com/en/www/herocraftonline.com

Is there an easy way to fix this? I current am using the Xenforo Sitemap plugin but not sure if google is even getting my up to date information.

A little new to optimizing website related things so trying to find the best route to take.
Click to expand...


Does it generate the sitemaps correctly, and are you defining it to google?
 
Slavik said:
Does it generate the sitemaps correctly, and are you defining it to google?
Click to expand...
Correct, I can see the sitemaps created on website/main/sitemap/files <-- there are several.

I sent it to google and woorank still didnt see it ... I had to push a single sitemap to website/sitemap.xml - and it seems woorank now sees it - but I'm not sure if thats correct.

We used to get 200-300 guests on at a time now we're only see 50-100 - so something may not be synching :(
 
Volion said:
Check your crawl stats in webmaster tools, did it drop sharply in January/February ?
Click to expand...
SwBbKnv.png


WdfY0cB.png
 
I was thinking something changed since the new year, because we also used to get 200-300 guests on at a time now we're only see 50-100. However it looks like your getting indexed. :confused:
 
Volion said:
I was thinking something changed since the new year, because we also used to get 200-300 guests on at a time now we're only see 50-100. However it looks like your getting indexed. :confused:
Click to expand...
Yah... perhaps we're just not as popular haha.
 
Kainzo said:
Yah... perhaps we're just not as popular haha.
Click to expand...

I moved servers/dns, changed robots.txt and dropped a parked domain. Hoping its just taking Google time to catch up, I may however sign up to woorank, let me know if it helps much. ;)

Also; our new firewall seems to love to block msn crawlers that port scan. :whistle:
 
Volion said:
I moved servers/dns, changed robots.txt and dropped a parked domain. Hoping its just taking Google time to catch up, I may however sign up to woorank, let me know if it helps much. ;)
Click to expand...
Well, woorank is free just to see where you're missing things but it seems pretty good for overall SEO.

What should I have in my robots.txt and where should I have it?
I looked around but it seems different all the time hah...
 
Kainzo said:
Well, woorank is free just to see where you're missing things but it seems pretty good for overall SEO.

What should I have in my robots.txt and where should I have it?
I looked around but it seems different all the time hah...
Click to expand...

That is a good question, every thread/opinion on whats best is different. :rolleyes:

It should be in the root, ergo;

http://xenforo.com/robots.txt
 
This is what I have in my robots that will likely help you:

Code: 
Sitemap: http://www.****.com/forums/sitemap/sitemap.xml.gz
 
User-agent: Mediapartners-Google
Disallow: /
 
User-agent: BoardReader
Disallow: /
 
User-agent: BoardTracker
Disallow: /
 
User-agent: Gigabot
Disallow: /
 
User-agent: Twiceler
Disallow: /
 
User-agent: dotbot
Disallow: /
 
User-Agent: *
Disallow: /cgi-bin/
Disallow: /_private/
Disallow: /_vti_bin/
Disallow: /_vti_cnf/
Disallow: /_vti_log/
Disallow: /_vti_pvt/
Disallow: /_vti_txt/
Disallow: /forums/find-new/
Disallow: /forums/account/
Disallow: /forums/attachments/
Disallow: /forums/goto/
Disallow: /forums/posts/
Disallow: /forums/login/
Disallow: /forums/admin.php
Disallow: /forums/search/
Disallow: /forums/search
Disallow: /forums/help/
Disallow: /forums/online/
Disallow: /forums/lost-password/
Allow: /
Have the robots in your root folder. Also, some of these SEO things are stupid and they look for the sitemap in the root as well. You can set up a 301 redirect if that is what is happening and have it point to your real sitemap. But in the end, it doesn't really matter bc it's these cheap SEO ranking things that are too stupid to find your sitemap when it has a defined path.
Take the results with a grain of salt.
 
Here's what I have.... it may be too basic.

# robots.txt for Sites
# Do Not delete this file.

User-agent: *
Disallow: /vbulletin
Disallow: /drupal
Disallow: /img
Disallow: /smf
Disallow: /test
Disallow: /testforum
Allow: /main
Allow: /

User-agent: BecomeBot
Crawl-Delay: 10
Click to expand...
 
there is something weird with XF sitemap and google indeed.

On a new install I did, google perfectly sees the "root" sitemap (the one indexing threads.1.xml and forums.1.xml ) but google NEVER EVER craxls the content of the files referenced in sitemap.xml ( namely, googlebot never downloads sitemap.thread.1.xml.gz )

Even by adding threads.xml manually in the webmaster central, google does not re index it.

I have no idea why, but this is something I just whitnessed
 
here's a grep for "sitemap" in my server logs.
Pretty clear,


66.249.76.156 - - [18/Feb/2013:02:53:51 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 471 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:02:53:52 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:02:53:52 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:02:53:53 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:03:04:23 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 469 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:03:04:24 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:03:04:24 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:03:04:24 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:07:50:00 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 469 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:07:50:00 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:07:50:01 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:07:50:01 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:08:04:15 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 471 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:08:04:16 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:08:04:17 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:08:04:17 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:09:01:00 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 469 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:09:01:00 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:09:01:01 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.76.156 - - [18/Feb/2013:09:01:01 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
 
Weppa333 said:
there is something weird with XF sitemap and google indeed.

On a new install I did, google perfectly sees the "root" sitemap (the one indexing threads.1.xml and forums.1.xml ) but google NEVER EVER craxls the content of the files referenced in sitemap.xml ( namely, googlebot never downloads sitemap.thread.1.xml.gz )

Even by adding threads.xml manually in the webmaster central, google does not re index it.

I have no idea why, but this is something I just whitnessed
Click to expand...
In webmaster tools, does it show the number of pages in sitemap and the number that have been indexed?
 
Yes, if you add or refresh in webmaster tools, it fecthes all the xml.gz, but it never does it again
weird...
 
Well I'm not sure, google is a very mysterious place... There is nothing wrong with XF ; it's just that "something" at google doesn't seem to like the embedded sitemaps used by the two Sitemap Mods that many people use here.
I encourage everyone to check if googlebot not ony fecthes the "root" sitemap, but also fetches the other sitemaps.

Maybe it's a delay/sandbox of somekind and that it solves itself after a while...

In the meantime, I manually resubmit the root sitemap every now and then.
 
You must log in or register to reply here.

Similar threads

S
  • Solved
XF 2.3 Same Two Sitemap.xml
Replies
3
Views
78
SelamT
S
JoyFreak
Duplicate Sitemap.xml & Rebuilding sitemap
Replies
1
Views
114
Chromaniac
Chromaniac
Seeker-Smith
  • Question Question
XF 2.3 2 xenforo sites sharing userbase
Replies
3
Views
73
CedricV
CedricV
Garfield™
Not a bug XenForo Sitemap File
Replies
9
Views
1K
MySiteGuy
MySiteGuy
VisEntities
How to Prevent Crawling or Scraping on My xenForo Site?
Replies
6
Views
138
VisEntities
VisEntities
Back
Top Bottom