XML Sitemap for XenForo 1.3 [Not needed, included in 1.4]

[Rigel Kentaurus]

Moved to the resources section:
http://xenforo.com/community/resources/sitemap-for-xenforo.67/

 

[Rigel Kentaurus]

v1.1

• Updated for XenForo 1.1
• Removed thread redirects from sitemap

v1.02

• Added configuration options in AdminCP (enable threads, forums, members)
• Added configuration options for total URLs per sitemap, and sitemap directory
• Added the option to include member profile pages in sitemap
• Added to option to ping Google and Bing automatically after generating the sitemap
• Enhanced security, to make sure that only the threads and forums that are public are included

v1.01

• Generates the Index file for the sitemap
• Generates the sitemap for forums
• Generates the sitemap for threads
• Supports gzip compression of the generated file

 

[Rigel Kentaurus]

There are several ways to make Google notice your Sitemap

One is going to Google WebMaster tools, registering your site, and adding your sitemap manually. This works.
http://www.google.com/webmasters/

The other one is to just update the robots.txt entry, and add a line containing Sitemap: url
for example, my robots.txt looks like this

Sitemap: http://fanficslandia.com/sitemap/sitemap.xml.gz

(I guess the other one is just making the Sitemap generator ping Google, but what would be the fun in that?)

 

[yavuz]

Thank God, finally someone made a sitemap addon. I'm not sure why anyone made one until now. Thank you a million Rigel. I'll try it out and get back with the results.

 

[yavuz]

These features, I belive would help improve the sitemap:

- Max URLs in Sitemap Files
- Include - Exlude Pages, Member Profiles* etc.
- Sitemap Update Frequency (For elements such as threads, pages, forums etc)
- Be Able to set up sitemap priorities
- Auto-Notify Search Engines
- Be Able to change the sitemap directory.
- Crawler Report. Be able to see the search engine & bot activity.


Some of them can be excluded via robots.txt but it would be nice if we could determin what we like to include or exlude inside de sitemap addon.

This is which I came up to in short notice. Thank you again for this addon Rigel.

 

[Romchik®]

These features, I belive would help improve the sitemap:

- Max URLs in Sitemap Files
- Include - Exlude Pages, Member Profiles* etc.
- Sitemap Update Frequency (For elements such as threads, pages, forums etc)
- Be Able to set up sitemap priorities
- Auto-Notify Search Engines
- Be Able to change the sitemap directory.

Some of them can be excluded via robots.txt but it would be nice if we could determin what we like to include or exlude inside de sitemap addon.

This is which I came up to in short notice. Thank you again for this addon Rigel.

Try Unlimited Sitemap Generator.

 

[Kaiser]

I have always been confused of what a sitemap is. Anyone care to explain?

 

[Brogan]

I have always been confused of what a sitemap is. Anyone care to explain?

http://en.wikipedia.org/wiki/Site_map

 

[Rigel Kentaurus]

I have always been confused of what a sitemap is. Anyone care to explain?

You can either wait for Google to crawl your site, or you can submit a sitemap in advance with all the URLs that your site provides (all the threads, forums, etc). Doing a good sitemap can help you get more traffic from search engines. For example, Google had missing 6400 out of 10000 URLs in my sitemap, I can expect eventually to have all of them indexed

 

[DSF]

The minimun request for this nice add-on is to use the guest-setting.
Sorry, but I will not public all the hidden forums like as admin, mod and special users.

 

[Rigel Kentaurus]

The minimun request for this nice add-on is to use the guest-setting.
Sorry, but I will not public all the hidden forums like as admin, mod and special users.

This should be taken care of now, it kind of was in my to-do list. The sitemap will only include threads and forums that are visible as a guest.

 

[Rigel Kentaurus]

These features, I belive would help improve the sitemap:

- Max URLs in Sitemap Files
- Include - Exlude Pages, Member Profiles* etc.
- Sitemap Update Frequency (For elements such as threads, pages, forums etc)
- Be Able to set up sitemap priorities
- Auto-Notify Search Engines
- Be Able to change the sitemap directory.
- Crawler Report. Be able to see the search engine & bot activity.


Some of them can be excluded via robots.txt but it would be nice if we could determin what we like to include or exlude inside de sitemap addon.

This is which I came up to in short notice. Thank you again for this addon Rigel.

I added some of this

• Max URLs now defaults to 10,000 and can be configured
• There is now an option in the AdminCP to select whether to include threads, forums and members
• Auto-Notify search engines is ready
• Sitemap directory defaults to sitemap, and can be changed (it does not validate that it is writable as of yet, but I assume if someone changes that he knows what he's doing )

I am not sure about the crawler report .. it is way too much work. Especially because from the Google Webmaster Tools console you can see your sitemap, and see the last time it was accesed, parsed, and the number of links that are already in the index. It seems kind of redundant to have it again.

Setting up sitemap priorities is something that I want to look into, probably I'll check that next. I am still not sure whether to give priority to new content (that makes sense), or if for example forums should have higher priority than threads (and higher than member pages), or if all that should just be configurable. I guess I will have to do some research on the best practice.

 

[Kaiser]

You can either wait for Google to crawl your site, or you can submit a sitemap in advance with all the URLs that your site provides (all the threads, forums, etc). Doing a good sitemap can help you get more traffic from search engines. For example, Google had missing 6400 out of 10000 URLs in my sitemap, I can expect eventually to have all of them indexed

Oh ok, yea googles always crawling my site I have no need for it.

 

[DSF]

Is it a good idea to zip the sitemap's?

 

[Weiyan]

Its better to configure which forum should appear in the sitemap. For some post, I like the user can search it, but not allowed to view untill they registered.

 

[yavuz]

I added some of this

• Max URLs now defaults to 10,000 and can be configured
• There is now an option in the AdminCP to select whether to include threads, forums and members
• Auto-Notify search engines is ready
• Sitemap directory defaults to sitemap, and can be changed (it does not validate that it is writable as of yet, but I assume if someone changes that he knows what he's doing )

I am not sure about the crawler report .. it is way too much work. Especially because from the Google Webmaster Tools console you can see your sitemap, and see the last time it was accesed, parsed, and the number of links that are already in the index. It seems kind of redundant to have it again.

Setting up sitemap priorities is something that I want to look into, probably I'll check that next. I am still not sure whether to give priority to new content (that makes sense), or if for example forums should have higher priority than threads (and higher than member pages), or if all that should just be configurable. I guess I will have to do some research on the best practice.

Amazing, and it such a short notice, I've installed it and it works smoothly. Thanks Rigelfor the hard work.

 

[Puntocom]

Thanks for the mod. When I click Run I get this page (black rectangle added by me to hide url)



Anyway it works fine!

 

[fattony69]

I was waiting for this! Very awesome. Thank you very much!

 

[Weiyan]

After half day's work on finding bug of other mod, install this mod. Since other use tested it, I feel confidence to use it.

 

[Weiyan]

As I see the generating log, it excluded private forum. Does this means threads and subforums also excluded?
For public forums, does it iterate in the order of latest replied thread first? That means for a very old thread, if there is a reply, then it will appear at the top of the sitemap?