XML Sitemap for XenForo 1.3 [Not needed, included in 1.4]

I'm having trouble redirecting robots.txt to robots.php. I suspect it is a basic .htaccess issue. I get the following in my error log

[Mon Jan 13 13:02:38 2014] [alert] [client 50.84.48.70] /home2/amphicar/public_html/.htaccess: Invalid command 'robots.txt', perhaps misspelled or defined by a module not included in the server configuration

Here is my .htaccess. Any help appreciated. My xenforo forum is installed in root (public_html).

SetEnvIfNoCase User-agent “Baidu” spammer=yes
SetEnvIfNoCase User-agent “Yandex” spammer=yes
SetEnvIfNoCase User-agent “Sosospider” spammer=yes
SetEnvIfNoCase User-agent “Proximic” spammer=yes
SetEnvIfNoCase User-agent “Brandwatch” spammer=yes
<Limit GET PUT POST>
order deny,allow
deny from env=spammer
</Limit>

# Mod_security can interfere with uploading of content such as attachments. If you
# cannot attach files, remove the "#" from the lines below.
<IfModule mod_security.c>
SecFilterEngine Off
SecFilterScanPOST Off
</IfModule>
ErrorDocument 401 default
ErrorDocument 403 default
ErrorDocument 404 default
ErrorDocument 500 default
<IfModule mod_rewrite.c>
RewriteEngine On
# If you are having problems with the rewrite rules, remove the "#" from the
# line that begins "RewriteBase" below. You will also have to change the path
# of the rewrite to reflect the path to your XenForo installation.
RewriteBase /public_html

RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -l [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^.*$ - [NC,L]
RewriteRule ^(data|js|styles|install) - [NC,L]
RewriteRule ^.*$ index.php [NC,L]
</IfModule>
<IfModule mod_rewrite.c>

RewriteEngine On
robots.txt => index.php?xfa-robots/index

</IfModule>​
 
Sportsoutlaw said: ↑
I have this running on one forum without any issues.

Installed it on a second forum and I am getting an internal server error when trying to run the cron. Checked my files and it isn't creating the sitemap/sitemap.xml.gz file. The other gz files are being created.

Ive uninstalled it and reinstalled a couple times, even tried installing a previous version.
Click to expand...


I have the same problem

Sportsoutlaw said:
It wad an APC issue. Cleared the cache and the complete site map created properly
Click to expand...

What is APC? Which cache did you delete and where/how?
 
Hi,

I called my webhoster. It is a time out problem. I am on a shared server with 100MB RAM. I have 30sec CPU time to use these 100MB RAM. They tested it and said this addon tries to do the sitemap in one big step and within the first 30 Seconds the whole RAM is already full. THis is why it gives the internal server error. If the RAM would not be used full within the first 30 seconds, the script could go on around 180 seconds.

Most others shared hosting offers allow only 10 seconds by the way. A dedicated server is too expensive for me.

So I strongly recommend to code this sitemap procedure in the way like xenforo is doing it for sending out emails. Always little steps and there is almost never a time out isse even with 30.000 emails per newsletter. This mass mailing takes me always a lot longer than 30 or 180 seconds. But no problems with the memory.
 
I'd like to limit sitemap generation with those forums which are available only to unregistered user. Google bot generates the error because of authetification is needed.

Or admin can supply the list of forums to be EXCLUDED from sitemap.

Will you please add this option?
 
  • Like
Reactions: Cao
vkams said:
I'd like to limit sitemap generation with those forums which are available only to unregistered user. Google bot generates the error because of authetification is needed.

Or admin can supply the list of forums to be EXCLUDED from sitemap.

Will you please add this option?
Click to expand...

this plugin hasn't been updated in months and there has been no response from the author on this plugin or any others. it'd say it's abandoned. you should try jaxel's sitemap plugin (although it does not at this time have the features you're requesting).
 
vkams said:
I'd like to limit sitemap generation with those forums which are available only to unregistered user. Google bot generates the error because of authetification is needed.

Or admin can supply the list of forums to be EXCLUDED from sitemap.

Will you please add this option?
Click to expand...
You should already have this ability built in. It's called Robots.txt and is an option in the add-on. Just add the URL for the nodes in there like
/forums/###
where the ### is the forum number. Good search engines that honor the robots.txt should exclude them.
 
Rigel Kentaurus said:
20 million posts .. that is interesting.

How many forums and threads, though? The links are generated for threads and forums rather than single posts
Click to expand...
About 290 forums, 1.6 million threads. Thanks for replying, but with respect, where have you been?
A matter of choice, and a suggestion for you to consider, but I would rather pay for an addon (more than the usual paltry tens of £) and know that I'm going to get some support than have a free one with support not happening for weeks.
Anyhow sitemap generation is a very important element of SEO. Google asks for it. So this, as far as I'm concerned is top priority. The ability to extend it to include other content like Showcases, Hotspots, Events and most importantly our custom editorial system is also paramount.
I'd be very grateful if we could discuss this because one way or another, I need a fully functional sitemap in the next few weeks at the very most.
Thanks (y)
 
Stuart Wright said:
About 290 forums, 1.6 million threads. Thanks for replying, but with respect, where have you been?
A matter of choice, and a suggestion for you to consider, but I would rather pay for an addon (more than the usual paltry tens of £) and know that I'm going to get some support than have a free one with support not happening for weeks.
Anyhow sitemap generation is a very important element of SEO. Google asks for it. So this, as far as I'm concerned is top priority. The ability to extend it to include other content like Showcases, Hotspots, Events and most importantly our custom editorial system is also paramount.
I'd be very grateful if we could discuss this because one way or another, I need a fully functional sitemap in the next few weeks at the very most.
Thanks (y)
Click to expand...

I am not really up for developing what you want, but feel free to pick up the code and give it to another developer to improve upon, or have them code it from scratch. The add-on actually is extensible (extending the model to add additional types).

I'm actually more interested on the technical challenge, 1.6 million threads is interesting. The add-on ran through Cli would take some time, but it should be able to finish, since Cli does not have the request timeout limit that is usually present on an Http Request. The Cli support was already added on a previous release. I have thought about adding resume support, but it is really a bit too much for small sites.

I'll replicate it locally, the only issue is memory, but the add-on should already be running lightweight unless debug is on (sql logging spew) or any of the mysql libraries are leaking memory.
 
We are monitoring index-less and low queries at the moment, but I don't think we were when I ran this.
All I can tell you is that when run via the scheduled task system, it did not complete (far too few files) and when I ran the scheduled task manually, it timed out.
How can I help you identify the problem please?
 
Stuart Wright said:
We are monitoring index-less and low queries at the moment, but I don't think we were when I ran this.
All I can tell you is that when run via the scheduled task system, it did not complete (far too few files) and when I ran the scheduled task manually, it timed out.
How can I help you identify the problem please?
Click to expand...
You cannot run it through the scheduling system for a board that size, the php timeout limit would hit before it completes.

run on a terminal $ php library/XfAddOns/Sitemap/Cli.php
That should be more successful
Then you can disable the scheduled task and set it up as a cron job
 
Oh, I see. Yes, you need custom development for that. Assuming that is in your database you just need a custom sitemap type for each of those. The Sitemap by default includes Forums, Threads, Resources ... so you will need to code the extra types for you
 
Rigel Kentaurus said:
Oh, I see. Yes, you need custom development for that. Assuming that is in your database you just need a custom sitemap type for each of those. The Sitemap by default includes Forums, Threads, Resources ... so you will need to code the extra types for you
Click to expand...
Thanks. I'll have to get our coder John Warwick or maybe Chris Deeming to do it.
 
You must log in or register to reply here.

Similar threads

Langtrees
XML Sitemap for Xenforo 1.2
Replies
1
Views
1K
m1ne
m1ne
F
  • Question Question
XF 1.1 File path for add-on "Sitemap for XenForo 1.2.0"
Replies
5
Views
692
Paul B
Paul B
Top Bottom