Resource icon

MetaMirror 1.6.1

No permission to download
Overview Updates (17) Reviews (24) History Discussion
Is there any reason this would degrade in performance? I have a site with over 170k attachments and the Tester tool with Commit checked just sits there forever and never completes when doing just 5 posts at a time.

On other forums it is speedy with as many as 100 posts at a time.
 
I finally get this:

Code: 
An exception occurred: Mysqli prepare error: MySQL server has gone away in /home/sportgfx/public_html/library/Zend/Db/Statement/Mysqli.php on line 77

Zend_Db_Statement_Mysqli->_prepare() in Zend/Db/Statement.php at line 115
Zend_Db_Statement->__construct() in Zend/Db/Adapter/Mysqli.php at line 381
Zend_Db_Adapter_Mysqli->prepare() in Zend/Db/Adapter/Abstract.php at line 478
Zend_Db_Adapter_Abstract->query() in Zend/Db/Adapter/Abstract.php at line 825
Zend_Db_Adapter_Abstract->fetchOne() in XenForo/Error.php at line 48
XenForo_Error::unexpectedException() in XenForo/Application.php at line 362
XenForo_Application::handleException()

That is just doing 1 post at a time.

I can do 100 posts pulling 166 images on another site on the same server with no problem.
 
BamaStangGuy said:
Is there any reason this would degrade in performance? I have a site with over 170k attachments and the Tester tool with Commit checked just sits there forever and never completes when doing just 5 posts at a time.

On other forums it is speedy with as many as 100 posts at a time.
Click to expand...

So that looks like the SQL server is timing out. How do the sites compare in terms of number of posts? Do you exclude forums in the site that times out?

I'm no SQL expert. The query I use (with no excludes) expands to
Code: 
                SELECT * FROM xf_post
                INNER JOIN xf_thread ON (xf_thread.thread_id = xf_post.thread_id)
                WHERE message LIKE '%[/IMG]%'
                AND  MetaMirror_Processed != 1008
               
                LIMIT 0, 10

Maybe you could try running that query in phpmyadmin, see how long it takes. Then try without the INNER JOIN line, and also try as:

Code: 
                SELECT * FROM xf_post
                INNER JOIN xf_thread ON (xf_thread.thread_id = xf_post.thread_id)
                WHERE  MetaMirror_Processed != 1008
                AND message LIKE '%[/IMG]%'               
                LIMIT 0, 10
 
Thinking about that, I'm pretty sure the query is highly sub-optimal, probably putting quite a load on the database if it's too large to fit in memory. I shall use the wisdom of the internet to make it better.
 
The site is my largest. 900k+ posts. It worked fine for awhile but now is struggling. I guess this is because it is getting higher up in the post count range? It is the only site where this problem occurs.

Coupled with all of this, this is also a graphics design website. So it isn't unheard of to have over 50 attachments in a single post linked to.

Which leads me to I do not have an attachment limit per post. It is unlimited. So this could also be trying to insert 50-100 attachments to each post.

For sites that do have an attachment limit, how does this addon handle that? Does it respect it?What happen if the post has more linked images in the post than the attachment limit?

Just some more info that might help you.
 
I suggest limiting each post to a maximum of 5 attachments. This way each page could have a maximum of 5x15 assuming 15 posts per page so that would be 75 attachments per page.
 
Last edited:
BamaStangGuy said:
The site is my largest. 900k+ posts. It worked fine for awhile but now is struggling. I guess this is because it is getting higher up in the post count range? It is the only site where this problem occurs.

Coupled with all of this, this is also a graphics design website. So it isn't unheard of to have over 50 attachments in a single post linked to.

Which leads me to I do not have an attachment limit per post. It is unlimited. So this could also be trying to insert 50-100 attachments to each post.

For sites that do have an attachment limit, how does this addon handle that? Does it respect it?What happen if the post has more linked images in the post than the attachment limit?

Just some more info that might help you.
Click to expand...

I suspect it is to do with the ordering of items in the query as it's a SQL timeout, could you try the attached CronEntry.php, which adjusts the order or execution, and skips the INNER JOIN if not needed.

Regarding attachment limits, unless it takes longer than a minute to download all the images in a single post, then it's not going to be a problem. Potentially it might get stuck if there are a large number of dead image links in a post though (links to sites that time-out rather than returning not-found. Not happened yet, but quite possible. I'll think about it. There's no simple way without keeping a database of image links.
 

Attachments

Jolly good, I'll roll that into an update.

It makes perfect sense, as "message LIKE '%[/IMG]%'" would vastly slower than "MetaMirror_Processed != 1008". Both are very quick on a smaller database, but it adds up. The way it was it would have to churn through hundreds of thousands of "LIKE" checks. Now it never does the LIKE until it's found something with the much quicker flag check.
 
I get this every now and then as well

Code: 
Server Error Log
Error Info
ErrorException: copy(/home/sportgfx/public_html/data/MetaMirrorCache/ai241.photobucket.com_albums_ff57_bety_chelsea_berbatov.jpg): failed to open stream: No such file or directory - library/XenForo/Helper/File.php:195
Generated By: Brent W, A moment ago
Stack Trace
#0 [internal function]: XenForo_Application::handlePhpError(2, 'copy(/home/spor...', '/home/sportgfx/...', 195, Array)
#1 /home/sportgfx/public_html/library/XenForo/Helper/File.php(195): copy('/home/sportgfx/...', '/home/sportgfx/...')
#2 /home/sportgfx/public_html/library/XenForo/DataWriter/AttachmentData.php(246): XenForo_Helper_File::safeRename('/home/sportgfx/...', '/home/sportgfx/...')
#3 /home/sportgfx/public_html/library/XenForo/DataWriter/AttachmentData.php(205): XenForo_DataWriter_AttachmentData->_moveFile('/home/sportgfx/...', '/home/sportgfx/...')
#4 /home/sportgfx/public_html/library/XenForo/DataWriter/AttachmentData.php(125): XenForo_DataWriter_AttachmentData->_writeAttachmentFile('/home/sportgfx/...', Array)
#5 /home/sportgfx/public_html/library/XenForo/DataWriter.php(1397): XenForo_DataWriter_AttachmentData->_postSave()
#6 /home/sportgfx/public_html/library/XenForo/Model/Attachment.php(524): XenForo_DataWriter->save()
#7 /home/sportgfx/public_html/library/MetaMirror/CronEntry.php(411): XenForo_Model_Attachment->insertUploadedAttachmentData(Object(XenForo_Upload), 1760, Array)
#8 /home/sportgfx/public_html/library/MetaMirror/CronEntry.php(885): MetaMirror_CronEntry::importPostAttachment(Object(Zend_Db_Adapter_Mysqli), 'ai241.photobuck...', '/home/sportgfx/...', 1760, 181021, 1201700324)
#9 /home/sportgfx/public_html/library/MetaMirror/CronEntry.php(1052): MetaMirror_CronEntry::updateSingleContent(Array)
#10 /home/sportgfx/public_html/library/MetaMirror/CronEntry.php(544): MetaMirror_CronEntry::runForContentType()
#11 /home/sportgfx/public_html/library/MetaMirror/ControllerAdmin/MetaMirror.php(80): MetaMirror_CronEntry::runMetaMirror(Array)
#12 /home/sportgfx/public_html/library/XenForo/FrontController.php(337): MetaMirror_ControllerAdmin_MetaMirror->actionRun()
#13 /home/sportgfx/public_html/library/XenForo/FrontController.php(134): XenForo_FrontController->dispatch(Object(XenForo_RouteMatch))
#14 /home/sportgfx/public_html/admin.php(13): XenForo_FrontController->run()
#15 {main}
Request State
array(3) {
  ["url"] => string(48) "http://www.sportgfx.com/admin.php?MetaMirror/Run"
  ["_GET"] => array(1) {
    ["MetaMirror/Run"] => string(0) ""
  }
  ["_POST"] => array(3) {
    ["override"] => string(0) ""
    ["commit"] => string(1) "1"
    ["_xfToken"] => string(8) "********"
  }
}
 
BamaStangGuy said:
I get this every now and then as well

Code: 
Server Error Log
Error Info
ErrorException: copy(/home/sportgfx/public_html/data/MetaMirrorCache/ai241.photobucket.com_albums_ff57_bety_chelsea_berbatov.jpg): failed to open stream: No such ...
Click to expand...

Could be that the tester tool is conflicting with the cron job - i.e. both running at the same time.
 
How much effort would it be to change the save folder, I find the file path rather ugly and would like to pretty it up a little.
 
been waiting for something like this on xenforo. used one on my vbulletin forum and its work great.
 
Autonamus said:
How much effort would it be to change the save folder, I find the file path rather ugly and would like to pretty it up a little.
Click to expand...

Depends how much you want to change. There's the path name itself
PHP: 
        // Folder where we put all the cached images, in the external data folder
        self::$metamirror_folder = '/MetaMirrorCache';

which is combined with the external data path:
PHP: 
        $root = XenForo_Helper_File::getExternalDataPath();
        // gives, for example: /Users/mick/Sites/metabunk.org/public_html/data

        // $path is the absolute path to the cach directory
        self::$externalDataPath = $root.self::$metamirror_folder;

Then there's a bit of logic to get the URL, accounting of off-site pull CDNs for the external data.
 
Code: 
[error] File does not exist: /home/mysite/public_html/data/MetaMirrorCache/xbiglongfilenameofactualfile.jpg.pagespeed.ic.nmYKtH4b0N.jpg
Found a bug. If you are running mod_pagespeed then MetaMirror will ad the pagespeed info to the image name. Now if you turn off pagespeed it becomes a not found issue. It should ignore pagespeed info in the url.
 
Last edited:
weird thing is the image is there and displays. So maybe that was just a problem right after I turned of mod pagespeed.

Going by error logs
 
You must log in or register to reply here.

Similar threads

Alpha1
Add-on Metamirror for XF2 (rehosting of linked files, documents as attachments)
2
Replies
31
Views
3K
rdn
R
Back
Top Bottom