1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

xenforo sitemap.xml ?

Discussion in 'Server Configuration and Hosting' started by Kainzo, Feb 11, 2013.

  1. Kainzo

    Kainzo Active Member

    So I'm trying to increase my SEO and whatnot I'm using Woorank.com and it's not able to find my sitemap.xml

    Is this a big issue?
    http://www.woorank.com/en/www/herocraftonline.com

    Is there an easy way to fix this? I current am using the Xenforo Sitemap plugin but not sure if google is even getting my up to date information.

    A little new to optimizing website related things so trying to find the best route to take.
     
  2. Slavik

    Slavik XenForo Moderator Staff Member


    Does it generate the sitemaps correctly, and are you defining it to google?
     
  3. Kainzo

    Kainzo Active Member

    Correct, I can see the sitemaps created on website/main/sitemap/files <-- there are several.

    I sent it to google and woorank still didnt see it ... I had to push a single sitemap to website/sitemap.xml - and it seems woorank now sees it - but I'm not sure if thats correct.

    We used to get 200-300 guests on at a time now we're only see 50-100 - so something may not be synching :(
     
  4. Volion

    Volion Active Member

    Check your crawl stats in webmaster tools, did it drop sharply in January/February ?
     
  5. Kainzo

    Kainzo Active Member

    [​IMG]

    [​IMG]
     
  6. Volion

    Volion Active Member

    I was thinking something changed since the new year, because we also used to get 200-300 guests on at a time now we're only see 50-100. However it looks like your getting indexed. :confused:
     
  7. Kainzo

    Kainzo Active Member

    Yah... perhaps we're just not as popular haha.
     
  8. Volion

    Volion Active Member

    I moved servers/dns, changed robots.txt and dropped a parked domain. Hoping its just taking Google time to catch up, I may however sign up to woorank, let me know if it helps much. ;)

    Also; our new firewall seems to love to block msn crawlers that port scan. :whistle:
     
  9. Kainzo

    Kainzo Active Member

    Well, woorank is free just to see where you're missing things but it seems pretty good for overall SEO.

    What should I have in my robots.txt and where should I have it?
    I looked around but it seems different all the time hah...
     
  10. Volion

    Volion Active Member

    That is a good question, every thread/opinion on whats best is different. :rolleyes:

    It should be in the root, ergo;

    http://xenforo.com/robots.txt
     
  11. AzzidReign

    AzzidReign Well-Known Member

    This is what I have in my robots that will likely help you:

    Code:
    Sitemap: http://www.****.com/forums/sitemap/sitemap.xml.gz
     
    User-agent: Mediapartners-Google
    Disallow: /
     
    User-agent: BoardReader
    Disallow: /
     
    User-agent: BoardTracker
    Disallow: /
     
    User-agent: Gigabot
    Disallow: /
     
    User-agent: Twiceler
    Disallow: /
     
    User-agent: dotbot
    Disallow: /
     
    User-Agent: *
    Disallow: /cgi-bin/
    Disallow: /_private/
    Disallow: /_vti_bin/
    Disallow: /_vti_cnf/
    Disallow: /_vti_log/
    Disallow: /_vti_pvt/
    Disallow: /_vti_txt/
    Disallow: /forums/find-new/
    Disallow: /forums/account/
    Disallow: /forums/attachments/
    Disallow: /forums/goto/
    Disallow: /forums/posts/
    Disallow: /forums/login/
    Disallow: /forums/admin.php
    Disallow: /forums/search/
    Disallow: /forums/search
    Disallow: /forums/help/
    Disallow: /forums/online/
    Disallow: /forums/lost-password/
    Allow: /
    
    Have the robots in your root folder. Also, some of these SEO things are stupid and they look for the sitemap in the root as well. You can set up a 301 redirect if that is what is happening and have it point to your real sitemap. But in the end, it doesn't really matter bc it's these cheap SEO ranking things that are too stupid to find your sitemap when it has a defined path.
    Take the results with a grain of salt.
     
  12. Kainzo

    Kainzo Active Member

    Here's what I have.... it may be too basic.

     
  13. Weppa333

    Weppa333 Active Member

    there is something weird with XF sitemap and google indeed.

    On a new install I did, google perfectly sees the "root" sitemap (the one indexing threads.1.xml and forums.1.xml ) but google NEVER EVER craxls the content of the files referenced in sitemap.xml ( namely, googlebot never downloads sitemap.thread.1.xml.gz )

    Even by adding threads.xml manually in the webmaster central, google does not re index it.

    I have no idea why, but this is something I just whitnessed
     
  14. Weppa333

    Weppa333 Active Member

    here's a grep for "sitemap" in my server logs.
    Pretty clear,


    66.249.76.156 - - [18/Feb/2013:02:53:51 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 471 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.156 - - [18/Feb/2013:02:53:52 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.156 - - [18/Feb/2013:02:53:52 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.156 - - [18/Feb/2013:02:53:53 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.156 - - [18/Feb/2013:03:04:23 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 469 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.156 - - [18/Feb/2013:03:04:24 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.156 - - [18/Feb/2013:03:04:24 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.156 - - [18/Feb/2013:03:04:24 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.156 - - [18/Feb/2013:07:50:00 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 469 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.156 - - [18/Feb/2013:07:50:00 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.156 - - [18/Feb/2013:07:50:01 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.156 - - [18/Feb/2013:07:50:01 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.156 - - [18/Feb/2013:08:04:15 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 471 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.156 - - [18/Feb/2013:08:04:16 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.156 - - [18/Feb/2013:08:04:17 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.156 - - [18/Feb/2013:08:04:17 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.156 - - [18/Feb/2013:09:01:00 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 469 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.156 - - [18/Feb/2013:09:01:00 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.156 - - [18/Feb/2013:09:01:01 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    66.249.76.156 - - [18/Feb/2013:09:01:01 +0100] "GET /sitemap/sitemap.xml.gz HTTP/1.1" 200 468 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
     
  15. Kainzo

    Kainzo Active Member

    Right... not sure whats going on there.
     
  16. AzzidReign

    AzzidReign Well-Known Member

    In webmaster tools, does it show the number of pages in sitemap and the number that have been indexed?
     
  17. Weppa333

    Weppa333 Active Member

    Yes, if you add or refresh in webmaster tools, it fecthes all the xml.gz, but it never does it again
    weird...
     
  18. Kainzo

    Kainzo Active Member

    So whats the fix? should I refresh every few days?
     
  19. Weppa333

    Weppa333 Active Member

    Well I'm not sure, google is a very mysterious place... There is nothing wrong with XF ; it's just that "something" at google doesn't seem to like the embedded sitemaps used by the two Sitemap Mods that many people use here.
    I encourage everyone to check if googlebot not ony fecthes the "root" sitemap, but also fetches the other sitemaps.

    Maybe it's a delay/sandbox of somekind and that it solves itself after a while...

    In the meantime, I manually resubmit the root sitemap every now and then.
     
  20. Kainzo

    Kainzo Active Member

    yeah, thats what im going to do I guess.
     

Share This Page