XML Sitemap for XenForo 1.3 [Not needed, included in 1.4]

There is an .htaccess sample on the zip file, you could try using one of those
If you already have an .htaccess make a backup

Hi there, does this look right?

Code:
RewriteCond %{HTTP_HOST} !^www\.47r-squad\.com$
RewriteRule (.*) http://www.47r-squad.com/$1 [R=301,L]
#    Mod_security can interfere with uploading of content such as attachments. If you
#    cannot attach files, remove the "#" from the lines below.
#<IfModule mod_security.c>
#    SecFilterEngine Off
#    SecFilterScanPOST Off
#</IfModule>
 
ErrorDocument 401 default
ErrorDocument 403 default
ErrorDocument 404 default
ErrorDocument 500 default
 
<IfModule mod_rewrite.c>
        RewriteEngine On
 
    #    If you are having problems with the rewrite rules, remove the "#" from the
    #    line that begins "RewriteBase" below. You will also have to change the path
    #    of the rewrite to reflect the path to your XenForo installation.
    # RewriteBase /
 
    #    This line may be needed to enable WebDAV editing with PHP as a CGI.
    #RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
 
    RewriteCond %{REQUEST_FILENAME} -f [OR]
    RewriteCond %{REQUEST_FILENAME} -l [OR]
    RewriteCond %{REQUEST_FILENAME} -d
    RewriteRule ^.*$ - [NC,L]
    RewriteRule ^(data/|js/|styles/|install/|favicon\.ico|crossdomain\.xml) - [NC,L]
    RewriteRule (robots\.txt)$ robots.php [NC,L]
    RewriteRule ^.*$ index.php [NC,L]
</IfModule>
 
This mod is getting to complicated for me I fear with the addition of the robots thingy.

I already have a working robots.txt in my rootfolder. If I don't wish to use the robots.php what do i need to do? Basically I just want to run the sitemaps and nothing else. :)
 
This mod is getting to complicated for me I fear with the addition of the robots thingy.

I already have a working robots.txt in my rootfolder. If I don't wish to use the robots.php what do i need to do? Basically I just want to run the sitemaps and nothing else. :)
Basically don't use it. It's optional. Don't put the rewrites in :)
 
Getting this error when generating the sitemap:

PHP:
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 83 bytes)

How much memory does the script use? I've got ~350,000 threads on my site and I'm concerned about it gobbling up all the memory on my server.
 
Getting this error when generating the sitemap:

PHP:
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 83 bytes)

How much memory does the script use? I've got ~350,000 threads on my site and I'm concerned about it gobbling up all the memory on my server.
Could you tell me which sitemaps were generated? That will give me an idea of which section is causing the problem (forums, threads, members, etc).

As such, the script should not use that much memory (I would be impressed if it used more than 16Mb), the reason for this is that the threads are fetched using a cursor, not with fetchAll, and only 10,000 at a time (since that is the limit for a sitemap), the xml file is then flushed to disk. So there is no such thing as hogging the memory, unless it's a bug.
 
Thanks for the quick reply!

Running the Cli.php from the command line, the following files were generated before raising an error:

Code:
-rw-r--r--  1 niall  staff    433 16 May 20:59 sitemap.forums.1.xml.gz
-rw-r--r--  1 niall  staff  14998 16 May 21:00 sitemap.forums.pags.1.xml.gz
-rw-r--r--  1 niall  staff  117949 16 May 21:00 sitemap.members.1.xml.gz
-rw-r--r--  1 niall  staff  119745 16 May 21:00 sitemap.members.2.xml.gz
-rw-r--r--  1 niall  staff  118136 16 May 21:00 sitemap.members.3.xml.gz
-rw-r--r--  1 niall  staff  117520 16 May 21:00 sitemap.members.4.xml.gz
-rw-r--r--  1 niall  staff  88058 16 May 21:00 sitemap.members.5.xml.gz
-rw-r--r--  1 niall  staff  180648 16 May 20:59 sitemap.threads.1.xml.gz
-rw-r--r--  1 niall  staff  196035 16 May 20:59 sitemap.threads.10.xml.gz
-rw-r--r--  1 niall  staff  205495 16 May 20:59 sitemap.threads.11.xml.gz
-rw-r--r--  1 niall  staff  208874 16 May 21:00 sitemap.threads.12.xml.gz
-rw-r--r--  1 niall  staff  204428 16 May 21:00 sitemap.threads.13.xml.gz
-rw-r--r--  1 niall  staff  147489 16 May 21:00 sitemap.threads.14.xml.gz
-rw-r--r--  1 niall  staff  185741 16 May 20:59 sitemap.threads.2.xml.gz
-rw-r--r--  1 niall  staff  187455 16 May 20:59 sitemap.threads.3.xml.gz
-rw-r--r--  1 niall  staff  190304 16 May 20:59 sitemap.threads.4.xml.gz
-rw-r--r--  1 niall  staff  187589 16 May 20:59 sitemap.threads.5.xml.gz
-rw-r--r--  1 niall  staff  189651 16 May 20:59 sitemap.threads.6.xml.gz
-rw-r--r--  1 niall  staff  184743 16 May 20:59 sitemap.threads.7.xml.gz
-rw-r--r--  1 niall  staff  191696 16 May 20:59 sitemap.threads.8.xml.gz
-rw-r--r--  1 niall  staff  191796 16 May 20:59 sitemap.threads.9.xml.gz

Error was:

Code:
PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 87 bytes) in <snip>/library/XenForo/Model.php on line 513
 
I've been getting errors about memory running out on my website, I am wondering if this mod is causing the problems.
 
I've been getting errors about memory running out on my website, I am wondering if this mod is causing the problems.
Depends on whether you get it exactly at the time that the sitemap runs. If it is at any other time, it is most definitely something else :) It is configured to run after midnight, and it would not happen more than once a day.
 
I did a small trial run in my installation, and it reached a max of 49Mb, while trying with a forum with 300,000 threads

Screen Shot 2013-05-16 at 6.35.54 PM.webp

Screen Shot 2013-05-16 at 6.36.09 PM.webp


The process peaks at 50 Mb, but was most of the time at 43Mb, it does not seem to be linear, bigger or smaller forum it was using pretty much the same memory (as it should). I double checked that the mysql cursor for the thread is being closed, which was my only hint for a memory hog. The thread and forum models is already cached, other than caching the node permission ids, not much can be done.

Could you try editing the Cli.php file, and add the following, on the second line (after the opening php tag)

Code:
ini_set('memory_limit', '256M');

That way you modify the memory limit for the script only, and not for your whole php installation. Assumming it does not run in safe mode (or you are running suphp), it will work.

It is always possible that the PHP that you have compiled has a lot of extensions. Extensions are usually loaded in memory, and add to the PHP footprint. And if the script is also trying to allocate memory it might timeout. It could also be the gzip or xml extensions misbehaving. Hopefully, that will not be the case.
 
That worked, thanks!

Code:
-rw-r--r--  1 niall  staff     433 17 May 22:56 sitemap.forums.1.xml.gz
-rw-r--r--  1 niall  staff   14998 17 May 22:57 sitemap.forums.pags.1.xml.gz
-rw-r--r--  1 niall  staff  117949 17 May 22:57 sitemap.members.1.xml.gz
-rw-r--r--  1 niall  staff  119745 17 May 22:57 sitemap.members.2.xml.gz
-rw-r--r--  1 niall  staff  118136 17 May 22:57 sitemap.members.3.xml.gz
-rw-r--r--  1 niall  staff  117520 17 May 22:57 sitemap.members.4.xml.gz
-rw-r--r--  1 niall  staff   88058 17 May 22:57 sitemap.members.5.xml.gz
-rw-r--r--  1 niall  staff  180648 17 May 22:56 sitemap.threads.1.xml.gz
-rw-r--r--  1 niall  staff  196035 17 May 22:57 sitemap.threads.10.xml.gz
-rw-r--r--  1 niall  staff  205495 17 May 22:57 sitemap.threads.11.xml.gz
-rw-r--r--  1 niall  staff  208870 17 May 22:57 sitemap.threads.12.xml.gz
-rw-r--r--  1 niall  staff  204427 17 May 22:57 sitemap.threads.13.xml.gz
-rw-r--r--  1 niall  staff  147473 17 May 22:57 sitemap.threads.14.xml.gz
-rw-r--r--  1 niall  staff  185741 17 May 22:56 sitemap.threads.2.xml.gz
-rw-r--r--  1 niall  staff  187455 17 May 22:56 sitemap.threads.3.xml.gz
-rw-r--r--  1 niall  staff  190304 17 May 22:56 sitemap.threads.4.xml.gz
-rw-r--r--  1 niall  staff  187589 17 May 22:56 sitemap.threads.5.xml.gz
-rw-r--r--  1 niall  staff  189651 17 May 22:56 sitemap.threads.6.xml.gz
-rw-r--r--  1 niall  staff  184743 17 May 22:57 sitemap.threads.7.xml.gz
-rw-r--r--  1 niall  staff  191696 17 May 22:57 sitemap.threads.8.xml.gz
-rw-r--r--  1 niall  staff  191796 17 May 22:57 sitemap.threads.9.xml.gz
-rw-r--r--  1 niall  staff  145277 17 May 22:58 sitemap.threads.pags.1.xml.gz
-rw-r--r--  1 niall  staff   76302 17 May 23:09 sitemap.threads.pags.10.xml.gz
-rw-r--r--  1 niall  staff   81570 17 May 23:10 sitemap.threads.pags.11.xml.gz
-rw-r--r--  1 niall  staff   81534 17 May 23:10 sitemap.threads.pags.12.xml.gz
-rw-r--r--  1 niall  staff   21245 17 May 23:10 sitemap.threads.pags.13.xml.gz
-rw-r--r--  1 niall  staff  123745 17 May 22:59 sitemap.threads.pags.2.xml.gz
-rw-r--r--  1 niall  staff  104630 17 May 23:00 sitemap.threads.pags.3.xml.gz
-rw-r--r--  1 niall  staff  102853 17 May 23:01 sitemap.threads.pags.4.xml.gz
-rw-r--r--  1 niall  staff   97854 17 May 23:02 sitemap.threads.pags.5.xml.gz
-rw-r--r--  1 niall  staff   91594 17 May 23:03 sitemap.threads.pags.6.xml.gz
-rw-r--r--  1 niall  staff   89189 17 May 23:04 sitemap.threads.pags.7.xml.gz
-rw-r--r--  1 niall  staff   83890 17 May 23:07 sitemap.threads.pags.8.xml.gz
-rw-r--r--  1 niall  staff   80284 17 May 23:08 sitemap.threads.pags.9.xml.gz
-rw-r--r--  1 niall  staff     319 17 May 23:10 sitemap.xml.gz


I'll experiment with the memory limit to see if I can get it lower.

For what it's worth, attached is my PHP info. It's the standard installation that comes with OS X Mountain Lion.
 

Attachments

Disabled the addon for 2 days and have not gotten the out of memory error since.
Should I try to do the same by eiting the cli.php file?
 
I tried the above, but unfortunately the website is still going over memory, not sure what else to do honestly. :(
I had it disabled for a day and no issues, when I re-enabled this morning, I've started having issues again.
 
I am getting "Googlebot can't access your site" on Webmaster tools after i installed this addon, any ideas whats wrong? there is no Robots.txt file, just Robots.php
 
You have an empty robots.txt here
http://www.razeflux.com/robots.txt

That could be invalid rewrites. Test your rewrites, and test with and without a robots.txt present in the directory

This one works: http://www.razeflux.com/robots.php


http://www.razeflux.com/robots.php works yes, but googlebot is searching for robots.txt and not robots.php...

I created robots.txt myself, is it necessary? i mean since i have robots.php....

2XzMh.png
 
You need to either put a rewrite in your .htaccess file (if using apache) to make robots.txt point to robots.php
Or, don't put any rewrite at all and provide your own robots.txt file

Code:
#    Mod_security can interfere with uploading of content such as attachments. If you
#    cannot attach files, remove the "#" from the lines below.
#<IfModule mod_security.c>
#    SecFilterEngine Off
#    SecFilterScanPOST Off
#</IfModule>
 
ErrorDocument 401 default
ErrorDocument 403 default
ErrorDocument 404 default
ErrorDocument 500 default
 
<IfModule mod_rewrite.c>
    RewriteEngine On
 
    #    If you are having problems with the rewrite rules, remove the "#" from the
    #    line that begins "RewriteBase" below. You will also have to change the path
    #    of the rewrite to reflect the path to your XenForo installation.
    #RewriteBase /xenforo
 
    #    This line may be needed to enable WebDAV editing with PHP as a CGI.
    #RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
 
    RewriteCond %{REQUEST_FILENAME} -f [OR]
    RewriteCond %{REQUEST_FILENAME} -l [OR]
    RewriteCond %{REQUEST_FILENAME} -d
    RewriteRule ^.*$ - [NC,L]
    RewriteRule ^(data/|js/|styles/|install/|favicon\.ico|crossdomain\.xml|robots\.txt) - [NC,L] 
      RewriteRule (robots\.txt)$ robots.php [NC,L]
    RewriteRule ^.*$ index.php [NC,L]
</IfModule>
 
RewriteEngine On
 
RewriteCond %{HTTP_HOST} ^razeflux\.com$ [OR]
RewriteCond %{HTTP_HOST} ^www\.razeflux\.com$
RewriteRule ^/?$ "http\:\/\/www\.razeflux\.com\/" [R=301,L]

Would that work out?
 
Top Bottom