XML Sitemap for XenForo 1.3 [Not needed, included in 1.4]

Non Guru

Member
Hello..
On your robot.sample.txt
I put my site map, so it looks like this:

User-agent: *
Disallow: /account/
Disallow: /find-new/
Disallow: /help/
Disallow: /goto/
Disallow: /login/
Disallow: /lost-password/
Disallow: /misc/style/
Disallow: /online/
Disallow: /posts/
Disallow: /recent-activity/
Disallow: /register/
Disallow: /search/
Disallow: /admin.php
Disallow: /index.php?account/
Disallow: /index.php?find-new/
Disallow: /index.php?help/
Disallow: /index.php?goto/
Disallow: /index.php?login/
Disallow: /index.php?lost-password/
Disallow: /index.php?misc/style/
Disallow: /index.php?online/
Disallow: /index.php?posts/
Disallow: /index.php?recent-activity/
Disallow: /index.php?register/
Disallow: /index.php?search/
Disallow: /admin.php
Allow: /
Sitemap: http://myforum.com/sitemap/sitemap.xml.gz


Is that correct ?

or should I enter after Allow:/
So it will look like this:
Allow: /Sitemap: http://myforum.com/sitemap/sitemap.xml.gz

I just want to make sure so Search engine bot will crawl my site correctly.

Also I need to upload the robot.txt to my root or to sitemap directory ?
 

Rigel Kentaurus

Well-known member
Hello..
On your robot.sample.txt
I put my site map, so it looks like this:

User-agent: *
Disallow: /account/
Disallow: /find-new/
Disallow: /help/
Disallow: /goto/
Disallow: /login/
Disallow: /lost-password/
Disallow: /misc/style/
Disallow: /online/
Disallow: /posts/
Disallow: /recent-activity/
Disallow: /register/
Disallow: /search/
Disallow: /admin.php
Disallow: /index.php?account/
Disallow: /index.php?find-new/
Disallow: /index.php?help/
Disallow: /index.php?goto/
Disallow: /index.php?login/
Disallow: /index.php?lost-password/
Disallow: /index.php?misc/style/
Disallow: /index.php?online/
Disallow: /index.php?posts/
Disallow: /index.php?recent-activity/
Disallow: /index.php?register/
Disallow: /index.php?search/
Disallow: /admin.php
Allow: /
Sitemap: http://myforum.com/sitemap/sitemap.xml.gz


Is that correct ?

or should I enter after Allow:/
So it will look like this:
Allow: /Sitemap: http://myforum.com/sitemap/sitemap.xml.gz

I just want to make sure so Search engine bot will crawl my site correctly.

Also I need to upload the robot.txt to my root or to sitemap directory ?

That is correct, actually you don't need the "Allow" part, it wouldn't hurt, though

Upload it to the root, it won't be found if it's inside sitemap
 

Cats777

Member
After installing and running the cron entry, it goes to this plain-looking page. Is this normal?
 

Attachments

  • Untitled.png
    Untitled.png
    25.1 KB · Views: 11
T

tictocchoc

Guest
This will be my first add-on but I have questions on the install process:
  • It is not clear to me where I should upload the upload folder? At the root of my XF folder? Or is there a particular folder for add-on?
  • Can I then change the name of this upload folder ?
  • Also, if I upload only the upload folder, how can I
    d) Select addon_cemzoo_sitemap.xml and click on Install
    as this addon_cemzoo_sitemap.xml file should not be uploaded (as it is not located in the upload folder)
 

Rigel Kentaurus

Well-known member
This will be my first add-on but I have questions on the install process:
  • It is not clear to me where I should upload the upload folder? At the root of my XF folder? Or is there a particular folder for add-on?
  • Can I then change the name of this upload folder ?
  • Also, if I upload only the upload folder, how can I as this addon_cemzoo_sitemap.xml file should not be uploaded (as it is not located in the upload folder)
Your question is better suited on the XenForo, Questions and Support forum, http://xenforo.com/community/forums/xenforo-questions-and-support.25/ about how you install an add-on, I want to keep this thread only for comments about the add-on itself.
 

electrogypsy

Well-known member
There are several ways to make Google notice your Sitemap

One is going to Google WebMaster tools, registering your site, and adding your sitemap manually. This works.
http://www.google.com/webmasters/

The other one is to just update the robots.txt entry, and add a line containing Sitemap: url
for example, my robots.txt looks like this

Code:
Sitemap: http://fanficslandia.com/sitemap/sitemap.xml.gz

(I guess the other one is just making the Sitemap generator ping Google, but what would be the fun in that?) :)

just curious, i have my xenforo install at /community, and a wordpress blog in the root. should i add this robots.txt line to the root or to my community folder? thanks!
 

Rigel Kentaurus

Well-known member
just curious, i have my xenforo install at /community, and a wordpress blog in the root. should i add this robots.txt line to the root or to my community folder? thanks!
You need to add it to the root. That is the only place the spiders look for the file, they don't look into subfolders at all.
It you already have a sitemap for your wordpress, just add the one for your xenforo, too, the robots.txt accepts multiple sitemaps, as specified in http://www.sitemaps.org/protocol.html#submit_robots

Of course, you would need to make some changes, instead of
Sitemap: http://yoursite.com/sitemap/sitemap.xml.gz
It would be
Sitemap: http://yoursite.com/community/sitemap/sitemap.xml.gz
 

electrogypsy

Well-known member
You need to add it to the root. That is the only place the spiders look for the file, they don't look into subfolders at all.
It you already have a sitemap for your wordpress, just add the one for your xenforo, too, the robots.txt accepts multiple sitemaps, as specified in http://www.sitemaps.org/protocol.html#submit_robots

Of course, you would need to make some changes, instead of
Sitemap: http://yoursite.com/sitemap/sitemap.xml.gz
It would be
Sitemap: http://yoursite.com/community/sitemap/sitemap.xml.gz

awesome, thanks for the help!!!
 

Mouth

Well-known member
Suggestion:
When sitemap runs, it should first read /robots.txt and not include within the sitemap any disallowed url's from robots.txt. Sitemap options ticked within ACP should be overruled by robots.txt if disallowed.

Scenario:
1. My /robots.txt contains "Disallow: /members/"
2. I forgot (or didn't think) to untick 'Include members' within /admin.php?options/list/xenforo_sitemap
3. Sitemap for Xenforo included all my 1,000's of /members/* urls within for (and submitted to) Google
4. Google Webmaster Tools gave me 115,000 errors with my Sitemap due to it containing url's disallowed within robots.txt
 

Rigel Kentaurus

Well-known member
Suggestion:
When sitemap runs, it should first read /robots.txt and not include within the sitemap any disallowed url's from robots.txt. Sitemap options ticked within ACP should be overruled by robots.txt if disallowed.

Scenario:
1. My /robots.txt contains "Disallow: /members/"
2. I forgot (or didn't think) to untick 'Include members' within /admin.php?options/list/xenforo_sitemap
3. Sitemap for Xenforo included all my 1,000's of /members/* urls within for (and submitted to) Google
4. Google Webmaster Tools gave me 115,000 errors with my Sitemap due to it containing url's disallowed within robots.txt
Well, this is easily fixed by you going and unchecking members :) I don't think it is worth the additional development time
 

mcadx

Member
Hello - Should there be a similar number in submitted pages and the number of threads on the forum? How does the sitemap generator determine the number of pages that it submits?
 

mcadx

Member
Also, any insight as to what this error is? Running v1.2 on XF 1.1.3

Server Error

Undefined index: forumsPagination
  1. XenForo_Application::handlePhpError() in XfAddOns/Sitemap/Model/Sitemap.php at line 80
  2. XfAddOns_Sitemap_Model_Sitemap->generate() in XfAddOns/Sitemap/CronEntry/RebuildSitemap.php at line 31
  3. XfAddOns_Sitemap_CronEntry_RebuildSitemap::run()
  4. call_user_func() in XenForo/Model/Cron.php at line 356
  5. XenForo_Model_Cron->runEntry() in XenForo/ControllerAdmin/Cron.php at line 204
  6. XenForo_ControllerAdmin_Cron->actionRun() in XenForo/FrontController.php at line 310
  7. XenForo_FrontController->dispatch() in XenForo/FrontController.php at line 132
  8. XenForo_FrontController->run() in /usr/home/...../admin.php at line 13
 

ManOnDaMoon

Well-known member
Would I still need this if I am using Xen Utilites?
There is probably no need for this addon if you use XenUtiles.

Also, I suspect the two would conflict because of Sitemap using "/sitemap/" as a directory and XenUtiles using "/sitemap" as an XML filename.
 

Robby

Well-known member
when i have a robots text on my main domain, do i need one for the forum also or can i just remove robots.sample.txt?
 

luutruong

Active member
i have gots bug
Code:
#0 [internal function]: XenForo_Application::handlePhpError(2, 'unlink(sitemap/...', '/home/***/...', 85, Array)
#1 /home/***/public_html/library/XfAddOns/Sitemap/Helper/Base.php(85): unlink('sitemap/sitemap...')
#2 /home/***/public_html/library/XfAddOns/Sitemap/Model/Sitemap.php(180): XfAddOns_Sitemap_Helper_Base->save('sitemap/sitemap...')
#3 /home/***/public_html/library/XfAddOns/Sitemap/Model/Sitemap.php(70): XfAddOns_Sitemap_Model_Sitemap->generateForums()
#4 /home/***/public_html/library/XfAddOns/Sitemap/CronEntry/RebuildSitemap.php(31): XfAddOns_Sitemap_Model_Sitemap->generate()
#5 [internal function]: XfAddOns_Sitemap_CronEntry_RebuildSitemap::run(Array)
#6 /home/***/public_html/library/XenForo/Model/Cron.php(356): call_user_func(Array, Array)
#7 /home/***/public_html/library/XenForo/Cron.php(29): XenForo_Model_Cron->runEntry(Array)
#8 /home/***/public_html/library/XenForo/Cron.php(64): XenForo_Cron->run()
#9 /home/***/public_html/cron.php(12): XenForo_Cron::runAndOutput()
#10 {main}

Code:
ErrorException: unlink(sitemap/sitemap.forums.1.xml) [<a href='function.unlink'>function.unlink</a>]: No such file or directory - library/XfAddOns/Sitemap/Helper/Base.php:85
 
Generated By: Unknown Account, 52 mins ago
 
Top