XML Sitemap for XenForo 1.3 [Not needed, included in 1.4]

Hello..
On your robot.sample.txt
I put my site map, so it looks like this:

User-agent: *
Disallow: /account/
Disallow: /find-new/
Disallow: /help/
Disallow: /goto/
Disallow: /login/
Disallow: /lost-password/
Disallow: /misc/style/
Disallow: /online/
Disallow: /posts/
Disallow: /recent-activity/
Disallow: /register/
Disallow: /search/
Disallow: /admin.php
Disallow: /index.php?account/
Disallow: /index.php?find-new/
Disallow: /index.php?help/
Disallow: /index.php?goto/
Disallow: /index.php?login/
Disallow: /index.php?lost-password/
Disallow: /index.php?misc/style/
Disallow: /index.php?online/
Disallow: /index.php?posts/
Disallow: /index.php?recent-activity/
Disallow: /index.php?register/
Disallow: /index.php?search/
Disallow: /admin.php
Allow: /
Sitemap: http://myforum.com/sitemap/sitemap.xml.gz


Is that correct ?

or should I enter after Allow:/
So it will look like this:
Allow: /Sitemap: http://myforum.com/sitemap/sitemap.xml.gz

I just want to make sure so Search engine bot will crawl my site correctly.

Also I need to upload the robot.txt to my root or to sitemap directory ?
 
Non Guru said:
Hello..
On your robot.sample.txt
I put my site map, so it looks like this:

User-agent: *
Disallow: /account/
Disallow: /find-new/
Disallow: /help/
Disallow: /goto/
Disallow: /login/
Disallow: /lost-password/
Disallow: /misc/style/
Disallow: /online/
Disallow: /posts/
Disallow: /recent-activity/
Disallow: /register/
Disallow: /search/
Disallow: /admin.php
Disallow: /index.php?account/
Disallow: /index.php?find-new/
Disallow: /index.php?help/
Disallow: /index.php?goto/
Disallow: /index.php?login/
Disallow: /index.php?lost-password/
Disallow: /index.php?misc/style/
Disallow: /index.php?online/
Disallow: /index.php?posts/
Disallow: /index.php?recent-activity/
Disallow: /index.php?register/
Disallow: /index.php?search/
Disallow: /admin.php
Allow: /
Sitemap: http://myforum.com/sitemap/sitemap.xml.gz


Is that correct ?

or should I enter after Allow:/
So it will look like this:
Allow: /Sitemap: http://myforum.com/sitemap/sitemap.xml.gz

I just want to make sure so Search engine bot will crawl my site correctly.

Also I need to upload the robot.txt to my root or to sitemap directory ?
Click to expand...

That is correct, actually you don't need the "Allow" part, it wouldn't hurt, though

Upload it to the root, it won't be found if it's inside sitemap
 
After installing and running the cron entry, it goes to this plain-looking page. Is this normal?
 

Attachments

  • Untitled.webp
    Untitled.webp
    7.2 KB · Views: 11
This will be my first add-on but I have questions on the install process:
  • It is not clear to me where I should upload the upload folder? At the root of my XF folder? Or is there a particular folder for add-on?
  • Can I then change the name of this upload folder ?
  • Also, if I upload only the upload folder, how can I
    d) Select addon_cemzoo_sitemap.xml and click on Install
    Click to expand...
    as this addon_cemzoo_sitemap.xml file should not be uploaded (as it is not located in the upload folder)
 
tictocchoc said:
This will be my first add-on but I have questions on the install process:
  • It is not clear to me where I should upload the upload folder? At the root of my XF folder? Or is there a particular folder for add-on?
  • Can I then change the name of this upload folder ?
  • Also, if I upload only the upload folder, how can I as this addon_cemzoo_sitemap.xml file should not be uploaded (as it is not located in the upload folder)
Click to expand...
Your question is better suited on the XenForo, Questions and Support forum, http://xenforo.com/community/forums/xenforo-questions-and-support.25/ about how you install an add-on, I want to keep this thread only for comments about the add-on itself.
 
Rigel Kentaurus said:
There are several ways to make Google notice your Sitemap

One is going to Google WebMaster tools, registering your site, and adding your sitemap manually. This works.
http://www.google.com/webmasters/

The other one is to just update the robots.txt entry, and add a line containing Sitemap: url
for example, my robots.txt looks like this

Code: 
Sitemap: http://fanficslandia.com/sitemap/sitemap.xml.gz

(I guess the other one is just making the Sitemap generator ping Google, but what would be the fun in that?) :)
Click to expand...

just curious, i have my xenforo install at /community, and a wordpress blog in the root. should i add this robots.txt line to the root or to my community folder? thanks!
 
electrogypsy said:
just curious, i have my xenforo install at /community, and a wordpress blog in the root. should i add this robots.txt line to the root or to my community folder? thanks!
Click to expand...
You need to add it to the root. That is the only place the spiders look for the file, they don't look into subfolders at all.
It you already have a sitemap for your wordpress, just add the one for your xenforo, too, the robots.txt accepts multiple sitemaps, as specified in http://www.sitemaps.org/protocol.html#submit_robots

Of course, you would need to make some changes, instead of
Sitemap: http://yoursite.com/sitemap/sitemap.xml.gz
It would be
Sitemap: http://yoursite.com/community/sitemap/sitemap.xml.gz
 
Rigel Kentaurus said:
You need to add it to the root. That is the only place the spiders look for the file, they don't look into subfolders at all.
It you already have a sitemap for your wordpress, just add the one for your xenforo, too, the robots.txt accepts multiple sitemaps, as specified in http://www.sitemaps.org/protocol.html#submit_robots

Of course, you would need to make some changes, instead of
Sitemap: http://yoursite.com/sitemap/sitemap.xml.gz
It would be
Sitemap: http://yoursite.com/community/sitemap/sitemap.xml.gz
Click to expand...

awesome, thanks for the help!!!
 
Suggestion:
When sitemap runs, it should first read /robots.txt and not include within the sitemap any disallowed url's from robots.txt. Sitemap options ticked within ACP should be overruled by robots.txt if disallowed.

Scenario:
1. My /robots.txt contains "Disallow: /members/"
2. I forgot (or didn't think) to untick 'Include members' within /admin.php?options/list/xenforo_sitemap
3. Sitemap for Xenforo included all my 1,000's of /members/* urls within for (and submitted to) Google
4. Google Webmaster Tools gave me 115,000 errors with my Sitemap due to it containing url's disallowed within robots.txt
 
Mouth said:
Suggestion:
When sitemap runs, it should first read /robots.txt and not include within the sitemap any disallowed url's from robots.txt. Sitemap options ticked within ACP should be overruled by robots.txt if disallowed.

Scenario:
1. My /robots.txt contains "Disallow: /members/"
2. I forgot (or didn't think) to untick 'Include members' within /admin.php?options/list/xenforo_sitemap
3. Sitemap for Xenforo included all my 1,000's of /members/* urls within for (and submitted to) Google
4. Google Webmaster Tools gave me 115,000 errors with my Sitemap due to it containing url's disallowed within robots.txt
Click to expand...
Well, this is easily fixed by you going and unchecking members :) I don't think it is worth the additional development time
 
Hello - Should there be a similar number in submitted pages and the number of threads on the forum? How does the sitemap generator determine the number of pages that it submits?
 
Also, any insight as to what this error is? Running v1.2 on XF 1.1.3

Server Error

Undefined index: forumsPagination
  1. XenForo_Application::handlePhpError() in XfAddOns/Sitemap/Model/Sitemap.php at line 80
  2. XfAddOns_Sitemap_Model_Sitemap->generate() in XfAddOns/Sitemap/CronEntry/RebuildSitemap.php at line 31
  3. XfAddOns_Sitemap_CronEntry_RebuildSitemap::run()
  4. call_user_func() in XenForo/Model/Cron.php at line 356
  5. XenForo_Model_Cron->runEntry() in XenForo/ControllerAdmin/Cron.php at line 204
  6. XenForo_ControllerAdmin_Cron->actionRun() in XenForo/FrontController.php at line 310
  7. XenForo_FrontController->dispatch() in XenForo/FrontController.php at line 132
  8. XenForo_FrontController->run() in /usr/home/...../admin.php at line 13
Click to expand...
 
surfsup said:
Would I still need this if I am using Xen Utilites?
Click to expand...
There is probably no need for this addon if you use XenUtiles.

Also, I suspect the two would conflict because of Sitemap using "/sitemap/" as a directory and XenUtiles using "/sitemap" as an XML filename.
 
when i have a robots text on my main domain, do i need one for the forum also or can i just remove robots.sample.txt?
 
i have gots bug
Code: 
#0 [internal function]: XenForo_Application::handlePhpError(2, 'unlink(sitemap/...', '/home/***/...', 85, Array)
#1 /home/***/public_html/library/XfAddOns/Sitemap/Helper/Base.php(85): unlink('sitemap/sitemap...')
#2 /home/***/public_html/library/XfAddOns/Sitemap/Model/Sitemap.php(180): XfAddOns_Sitemap_Helper_Base->save('sitemap/sitemap...')
#3 /home/***/public_html/library/XfAddOns/Sitemap/Model/Sitemap.php(70): XfAddOns_Sitemap_Model_Sitemap->generateForums()
#4 /home/***/public_html/library/XfAddOns/Sitemap/CronEntry/RebuildSitemap.php(31): XfAddOns_Sitemap_Model_Sitemap->generate()
#5 [internal function]: XfAddOns_Sitemap_CronEntry_RebuildSitemap::run(Array)
#6 /home/***/public_html/library/XenForo/Model/Cron.php(356): call_user_func(Array, Array)
#7 /home/***/public_html/library/XenForo/Cron.php(29): XenForo_Model_Cron->runEntry(Array)
#8 /home/***/public_html/library/XenForo/Cron.php(64): XenForo_Cron->run()
#9 /home/***/public_html/cron.php(12): XenForo_Cron::runAndOutput()
#10 {main}

Code: 
ErrorException: unlink(sitemap/sitemap.forums.1.xml) [<a href='function.unlink'>function.unlink</a>]: No such file or directory - library/XfAddOns/Sitemap/Helper/Base.php:85
 
Generated By: Unknown Account, 52 mins ago
 
You must log in or register to reply here.

Similar threads

Langtrees
XML Sitemap for Xenforo 1.2
Replies
1
Views
1K
m1ne
m1ne
F
  • Question Question
XF 1.1 File path for add-on "Sitemap for XenForo 1.2.0"
Replies
5
Views
686
Paul B
Paul B
Top Bottom