XML Sitemap for XenForo 1.3 [Not needed, included in 1.4]

[Regina Ortega]

Thank you.

 

[Lee Armstrong]

Could you check "Server Errors" in the admin control panel and see if you have an error logged there ?

View attachment 44479

Empty I'm afraid! (except some site map generation succeeded messages). Nothing about the robots script at all though!

Happy to run debug code if you want, running nginx and brand new forum as you can see!

 

[Lee Armstrong]

Just fixed with a full cache rebuild

 

[Rigel Kentaurus]

Just fixed with a full cache rebuild

Glad you fixed it .. There is an "xfa_robots" template that is used, most likely that one was empty before the cache rebuilding

 

[Liam W]

Works well

Love the robots.php feature (I made my own rewrite rule, robots.txt will rewrite to robots.php now).

However, how would I edit the text outputted? Mainly, so I can add custom rules etc?

I checked the robots.php file, however I can't see anything that would actually output text (I am new to xenForo).

Ok, I found the template that controls it but I can't get it to place the line break (when I stick the HTML in it just prints it as text)

EDIT: Done it I edited the file that has the rendering function, and added a line that converts HTML br tags to plain text line breaks

I also found the admin options to add more entries

EDIT2: Even better, it now detects the User-agent lines and prefixes them with a new line, and removed the line from the top of the file. A perfect robots.txt with the only issues being the wildcard's in the disallow entries

Liam

 

[Rigel Kentaurus]

Works well

Love the robots.php feature (I made my own rewrite rule, robots.txt will rewrite to robots.php now).

However, how would I edit the text outputted? Mainly, so I can add custom rules etc?

I checked the robots.php file, however I can't see anything that would actually output text (I am new to xenForo).

Ok, I found the template that controls it but I can't get it to place the line break (when I stick the HTML in it just prints it as text)

EDIT: Done it I edited the file that has the rendering function, and added a line that converts HTML br tags to plain text line breaks

I also found the admin options to add more entries

EDIT2: Even better, it now detects the User-agent lines and prefixes them with a new line, and removed the line from the top of the file. A perfect robots.txt with the only issues being the wildcard's in the disallow entries

Liam

It is actually specifically removing new lines, it was meant as a future since the new lines are not really needed.
If you don't want that to happen on library/XfAddOns/Sitemap/ViewPublic/Robots.php delete this line

$output = preg_replace("/[\n][\n]+/", "\n", trim($output));

The output would have several line breaks, and any that you specifically put in the template would work

 

[Liam W]

It is actually specifically removing new lines, it was meant as a future since the new lines are not really needed.
If you don't want that to happen on library/XfAddOns/Sitemap/ViewPublic/Robots.php delete this line

$output = preg_replace("/[\n][\n]+/", "\n", trim($output));

The output would have several line breaks, and any that you specifically put in the template would work

I only wanted line breaks before the user agent definitions as that's valid robots.text format - its classed as an error otherwise.

 

[Rigel Kentaurus]

I only wanted line breaks before the user agent definitions as that's valid robots.text format - its classed as an error otherwise.

Do you used any particular validator for that? I checked the grammar at http://www.robotstxt.org/norobots-rfc.txt and I don't see anything that would mark the lack of line breaks to be a problem.

I also tested online with the Google Webmaster Tools and it does not complain about the lack of line breaks

 

[OakleyForum]

When my sitemap tries to run, I get this error:

Undefined variable: sitemaps

1. XenForo_Application::handlePhpError() in XfAddOns/Sitemap/Model/Sitemap.php at line 139
2. XfAddOns_Sitemap_Model_Sitemap->runAllAvailableSiteMaps() in XfAddOns/Sitemap/CronEntry/RebuildSitemap.php at line 31
3. XfAddOns_Sitemap_CronEntry_RebuildSitemap::run()
4. call_user_func() in XenForo/Model/Cron.php at line 356
5. XenForo_Model_Cron->runEntry() in XenForo/ControllerAdmin/Cron.php at line 204
6. XenForo_ControllerAdmin_Cron->actionRun() in XenForo/FrontController.php at line 310
7. XenForo_FrontController->dispatch() in XenForo/FrontController.php at line 132
8. XenForo_FrontController->run() in /home/oakleyfo/public_html/admin.php at line 13

 

[Rigel Kentaurus]

When my sitemap tries to run, I get this error:

Since it is not ocurring for everyone seems to be for your specific PHP configuration
I'll fix it in the next update, but in the meantime, on the file library/XfAddOns/Sitemap/Model/Sitemap

Change this

        foreach ($classes as $klass)
        {

To this

        foreach ($classes as $klass)
        {
                $sitemaps = null;

 

[OakleyForum]

Fixed it! Thanks! Though I had to chmod777 the sitemap.gz files inside the sitemap folder before it worked. Is this normal?

 

[Rigel Kentaurus]

Fixed it! Thanks! Though I had to chmod777 the sitemap.gz files inside the sitemap folder before it worked. Is this normal?

When they are created, they are chmod'ed 666 (but even 600 would work), I do not recommend 777 as they do not need to be executable

It is not necessary normal, I would rather say delete them and let the sitemap create them again. But it won't hurt anything.

Could you also check your "Server Error Logs" in the admincp to make sure there is nothing there ? That error you experience could be symptom or something else failing

 

[OakleyForum]

Ok, I deleted and rebuilt them.

 

[Rigel Kentaurus]

Rigel Kentaurus updated Sitemap for XenForo with a new update entry:

1.2.7b Extra Robots.txt options

If you like this add-on, please post a review. It takes 1 minute and I enjoy the feedback!

A small maintenance release, you could skip this one

• Fixed a bug when the sitemap variable does not get initialized, and later the code references it. (For @OakleyForum)
• Added whitespace before the User-Agent to keep some validators a little happier. Google still validated fine before (For @Liam)
• Added an "additional robots.txt" section in the AdminCP options, so you can extend robots.txt with any other rules

 

[usAdultAds]

Rigel,

How about adding sitemap.html for forum display?
but the html should be created in the forum directory,
all other files in the sitemap directory....i guess it could
be in the sitemap directory as index.html rather then
having 0 bytes, it could be for public view of what the
sitemap looks like as it is growing....

 

[Rigel Kentaurus]

Rigel,

How about adding sitemap.html for forum display?
but the html should be created in the forum directory,
all other files in the sitemap directory....i guess it could
be in the sitemap directory as index.html rather then
having 0 bytes, it could be for public view of what the
sitemap looks like as it is growing....

The sitemap is really just for the robots I don't see anyone voluntarily reading a list of 10,000 links

 

[usAdultAds]

fair enough...
installed, appears to work fine....

The sitemap is really just for the robots I don't see anyone voluntarily reading a list of 10,000 links

 

[my1es]

Getting a 500 Error on trying to Run Rebuild after first install .. doesn't leave the Cron Entries screen. Any ideas? It did generate a few files in my sitemap folder however

 

[Rigel Kentaurus]

Getting a 500 Error on trying to Run Rebuild after first install .. doesn't leave the Cron Entries screen. Any ideas? It did generate a few files in my sitemap folder however

sounds like a timeout. How big is your forum?

 

[my1es]

Discussions: 106,881
Messages: 1,884,339
Members: 11,095