[bd] Data Storage [Deleted]

also having XFMG issues with thumbnail image creation

Code: 
ErrorException: getimagesize(): Read error! - library/XenGallery/Helper/Image.php:84
Generated By: pete, 7 minutes ago
Stack Trace
#0 [internal function]: XenForo_Application::handlePhpError(8, 'getimagesize():...', '/home/thumped/p...', 84, Array)
#1 /home/thumped/public_html/bbs/library/XenGallery/Helper/Image.php(84): getimagesize('ds0:///xengalle...')
#2 /home/thumped/public_html/bbs/library/XenGallery/Helper/Image.php(54): XenGallery_Helper_Image->_readImageInfo()
#3 /home/thumped/public_html/bbs/library/XenGallery/Thumbnail/Abstract.php(136): XenGallery_Helper_Image->__construct('ds0:///xengalle...')
#4 /home/thumped/public_html/bbs/library/XenGallery/Thumbnail/Abstract.php(151): XenGallery_Thumbnail_Abstract->saveThumbnail()
#5 /home/thumped/public_html/bbs/library/XenGallery/Thumbnail/YouTube.php(33): XenGallery_Thumbnail_Abstract->verifyThumbnailUrl('https://i.ytimg...')
#6 /home/thumped/public_html/bbs/library/XenGallery/Model/Media.php(3207): XenGallery_Thumbnail_YouTube->getThumbnailUrl('qOR0HxzxM6Q')
#7 /home/thumped/public_html/bbs/library/XenGallery/ControllerPublic/Media.php(2441): XenGallery_Model_Media->getVideoThumbnailUrlFromParts(Array)
#8 /home/thumped/public_html/bbs/library/XenForo/FrontController.php(347): XenGallery_ControllerPublic_Media->actionPreviewVideo()
#9 /home/thumped/public_html/bbs/library/XenForo/FrontController.php(134): XenForo_FrontController->dispatch(Object(XenForo_RouteMatch))
#10 /home/thumped/public_html/bbs/index.php(13): XenForo_FrontController->run()
#11 {main}
Request State
array(3) {
  ["url"] => string(98) "https://thumped.com/bbs/index.php?xengallery/preview-video&_xfResponseType=json&media_id=undefined"
  ["_GET"] => array(3) {
    ["xengallery/preview-video"] => string(0) ""
    ["_xfResponseType"] => string(4) "json"
    ["media_id"] => string(9) "undefined"
  }
  ["_POST"] => array(5) {
    ["embed_url"] => string(43) "https://www.youtube.com/watch?v=qOR0HxzxM6Q"
    ["_xfRequestUri"] => string(14) "/bbs/media/add"
    ["_xfNoRedirect"] => string(1) "1"
    ["_xfToken"] => string(8) "********"
    ["_xfResponseType"] => string(4) "json"
  }
}
Code: 
ErrorException: stat(): stat failed for /home/thumped/public_html/bbs/internal_data/temp/bddss3df_50o0NQ - library/bdDataStorage/StreamWrapper/Abstract.php:182
Generated By: pete, 7 minutes ago
Stack Trace
#0 [internal function]: XenForo_Application::handlePhpError(2, 'stat(): stat fa...', '/home/thumped/p...', 182, Array)
#1 /home/thumped/public_html/bbs/library/bdDataStorage/StreamWrapper/Abstract.php(182): stat('/home/thumped/p...')
#2 /home/thumped/public_html/bbs/library/bdDataStorage/StreamWrapper/Abstract.php(483): bdDataStorage_StreamWrapper_Abstract->_cacheSaveFile(Array, 'xengallery/yout...', '/home/thumped/p...')
#3 [internal function]: bdDataStorage_StreamWrapper_Abstract->stream_open('ds0:///xengalle...', 'rb', 16, NULL)
#4 /home/thumped/public_html/bbs/library/XenGallery/Helper/Image.php(84): getimagesize('ds0:///xengalle...')
#5 /home/thumped/public_html/bbs/library/XenGallery/Helper/Image.php(1077): XenGallery_Helper_Image->_readImageInfo()
#6 /home/thumped/public_html/bbs/library/XenGallery/Helper/Image.php(196): XenGallery_Helper_Image->getRatioWidthToHeight()
#7 /home/thumped/public_html/bbs/library/XenGallery/Thumbnail/Abstract.php(137): XenGallery_Helper_Image->resize('300', '169', 'crop')
#8 /home/thumped/public_html/bbs/library/XenGallery/Thumbnail/Abstract.php(151): XenGallery_Thumbnail_Abstract->saveThumbnail()
#9 /home/thumped/public_html/bbs/library/XenGallery/Thumbnail/YouTube.php(33): XenGallery_Thumbnail_Abstract->verifyThumbnailUrl('https://i.ytimg...')
#10 /home/thumped/public_html/bbs/library/XenGallery/Model/Media.php(3207): XenGallery_Thumbnail_YouTube->getThumbnailUrl('qOR0HxzxM6Q')
#11 /home/thumped/public_html/bbs/library/XenGallery/ControllerPublic/Media.php(2441): XenGallery_Model_Media->getVideoThumbnailUrlFromParts(Array)
#12 /home/thumped/public_html/bbs/library/XenForo/FrontController.php(347): XenGallery_ControllerPublic_Media->actionPreviewVideo()
#13 /home/thumped/public_html/bbs/library/XenForo/FrontController.php(134): XenForo_FrontController->dispatch(Object(XenForo_RouteMatch))
#14 /home/thumped/public_html/bbs/index.php(13): XenForo_FrontController->run()
#15 {main}
Request State
array(3) {
  ["url"] => string(98) "https://thumped.com/bbs/index.php?xengallery/preview-video&_xfResponseType=json&media_id=undefined"
  ["_GET"] => array(3) {
    ["xengallery/preview-video"] => string(0) ""
    ["_xfResponseType"] => string(4) "json"
    ["media_id"] => string(9) "undefined"
  }
  ["_POST"] => array(5) {
    ["embed_url"] => string(43) "https://www.youtube.com/watch?v=qOR0HxzxM6Q"
    ["_xfRequestUri"] => string(14) "/bbs/media/add"
    ["_xfNoRedirect"] => string(1) "1"
    ["_xfToken"] => string(8) "********"
    ["_xfResponseType"] => string(4) "json"
  }
}
 
This method in bdDataStorage_StreamWrapper_S3 is problematic:

PHP: 
protected function _downloadFile(array $config, $path)
{
    $connection = $this->_openConnection($config);

    $tempFile = tempnam(XenForo_Helper_File::getTempDir(), 'bddss3df_');
    self::$_downloadedFiles[$config['_protocol']][$path] = $tempFile;

    $object = $connection->getObject($config['bucket'] . '/' . $path);

    if (!empty($object)) {
        file_put_contents($tempFile, $object);

        $this->_log('_downloadFile/s3 ok', $path, $tempFile);
        return $tempFile;
    } else {
        $this->_log('_downloadFile/s3 failed', $path);
        return false;
    }
}

If $path is cached in $_downloadedFiles, the function won't return false when it's supposed to; it will always return a path. This causes url_stat in bdDataStorage_StreamWrapper_Abstract to erroneously report that directories are files. This, in turn, confuses XenForo_Helper_File::createDirectory(), which returns false because it thinks the directory already exists as a file. As a result, XenForo_Model_Avatar::_writeAvatar() returns false without calling XenForo_Helper_File::safeRename().

Whether this bug results in noticeable problems seems to depend on the environment. It randomly started preventing all avatar uploads on our development site, but wasn't causing any problems in production.

Here's a patched version of the method:

PHP: 
protected function _downloadFile(array $config, $path)
{
    $connection = $this->_openConnection($config);
    $object = $connection->getObject($config['bucket'] . '/' . $path);

    if (empty($object)) {
        $this->_log('_downloadFile/s3 failed', $path);
        return false;
    }

    $tempFile = tempnam(XenForo_Helper_File::getTempDir(), 'bddss3df_');
    file_put_contents($tempFile, $object);
    self::$_downloadedFiles[$config['_protocol']][$path] = $tempFile;

    $this->_log('_downloadFile/s3 ok', $path, $tempFile);

    return $tempFile;
}
 
Last edited:
more XFMG error action

An exception occurred: Cannot save file /youtube_E03tbipoDkE_thumb.jpg in /home/thumped/public_html/bbs/library/XenGallery/Helper/Image.php on line 833
XenGallery_Helper_Image->save() in XenGallery/Thumbnail/Abstract.php at line 139
XenGallery_Thumbnail_Abstract->saveThumbnail() in XenGallery/Thumbnail/Abstract.php at line 151
XenGallery_Thumbnail_Abstract->verifyThumbnailUrl() in XenGallery/Thumbnail/YouTube.php at line 33
XenGallery_Thumbnail_YouTube->getThumbnailUrl() in XenGallery/Model/Media.php at line 3207
XenGallery_Model_Media->getVideoThumbnailUrlFromParts() in XenGallery/Model/Media.php at line 1067
XenGallery_Model_Media->prepareMedia() in XenGallery/Model/Media.php at line 1130
XenGallery_Model_Media->prepareMediaItems() in XenGallery/Callback.php at line 76
XenGallery_Callback::getMediaForBlock()
call_user_func() in XenForo/Template/Abstract.php at line 330
XenForo_Template_Abstract->callTemplateCallback() in /home/thumped/public_html/bbs/internal_data/templates/S.12,L.1,xengallery_media_block_sidebar.php at line 41
include() in XenForo/Template/Abstract.php at line 260
XenForo_Template_Abstract->_renderInternal() in XenForo/Template/Abstract.php at line 191
XenForo_Template_Abstract->render() in XenForo/Template/Public.php at line 110
XenForo_Template_Public->render() in WidgetFramework/WidgetRenderer/Template.php at line 57
WidgetFramework_WidgetRenderer_Template->_render() in WidgetFramework/WidgetRenderer.php at line 660
WidgetFramework_WidgetRenderer->render() in WidgetFramework/Core.php at line 549
WidgetFramework_Core->_renderWidgetsFor_renderWidgetsContainer() in WidgetFramework/Core.php at line 473
WidgetFramework_Core->_renderWidgetsFor() in WidgetFramework/Core.php at line 377
WidgetFramework_Core->renderWidgetsFor() in WidgetFramework/Listener.php at line 100
WidgetFramework_Listener::template_post_render()
call_user_func_array() in XenForo/CodeEvent.php at line 58
XenForo_CodeEvent::fire() in XenForo/Template/Abstract.php at line 195
XenForo_Template_Abstract->render() in XenForo/Template/Public.php at line 110
XenForo_Template_Public->render() in XenForo/Template/Abstract.php at line 528
XenForo_Template_Abstract->__toString() in /home/thumped/public_html/bbs/internal_data/templates/S.12,L.1,find_new_wrapper.php at line 995
include() in XenForo/Template/Abstract.php at line 260
XenForo_Template_Abstract->_renderInternal() in XenForo/Template/Abstract.php at line 191
XenForo_Template_Abstract->render() in XenForo/Template/Public.php at line 110
XenForo_Template_Public->render() in XenForo/ViewRenderer/HtmlPublic.php at line 123
XenForo_ViewRenderer_HtmlPublic->renderContainer() in XenForo/FrontController.php at line 618
XenForo_FrontController->renderView() in XenForo/FrontController.php at line 158
XenForo_FrontController->run() in /home/thumped/public_html/bbs/index.php at line 13
Click to expand...

i've had to disable this addon completely now because of this. if a fix isn't possible, is there any way it can be selectively disabled for certain addons?
 
featured thread error when promoting to slider / adding an image

Code: 
getimagesize(ds0:///featured_threads/icons/95/95700.jpg): failed to open stream: "bdDataStorage_StreamWrapper_S3::stream_open" call failed

XenForo_Application::handlePhpError()
getimagesize() in CTA/FeaturedThreads/Helper/Image.php at line 44
CTA_FeaturedThreads_Helper_Image->_readImageInfo() in CTA/FeaturedThreads/Helper/Image.php at line 80
CTA_FeaturedThreads_Helper_Image->_setNewMainImage() in CTA/FeaturedThreads/Helper/Image.php at line 273
CTA_FeaturedThreads_Helper_Image->saveToPath() in CTA/FeaturedThreads/Model/Featured.php at line 471
CTA_FeaturedThreads_Model_Featured->applyFeaturedThreadImage() in CTA/FeaturedThreads/Model/Featured.php at line 399
CTA_FeaturedThreads_Model_Featured->uploadUrlFeaturedThreadImage() in CTA/FeaturedThreads/ControllerPublic/Thread.php at line 997
CTA_FeaturedThreads_ControllerPublic_Thread->actionFeaturedThreadImages() in XenForo/FrontController.php at line 347
XenForo_FrontController->dispatch() in XenForo/FrontController.php at line 134
XenForo_FrontController->run() in /home/thumped/public_html/bbs/index.php at line 13

halp!
 
Updated the fix in my previous post. Storing false in the cache breaks bdDataStorage_StreamWrapper_Abstract#stream_open(), and it's less than ideal for servers that don't have a reliable connection to S3.

Here's a proper patch:

Code: 
87,90d86
<
<         $tempFile = tempnam(XenForo_Helper_File::getTempDir(), 'bddss3df_');
<         self::$_downloadedFiles[$config['_protocol']][$path] = $tempFile;
<
93,98c89
<         if (!empty($object)) {
<             file_put_contents($tempFile, $object);
<
<             $this->_log('_downloadFile/s3 ok', $path, $tempFile);
<             return $tempFile;
<         } else {
---
>         if (empty($object)) {
101a93,100
>
>         $tempFile = tempnam(XenForo_Helper_File::getTempDir(), 'bddss3df_');
>         file_put_contents($tempFile, $object);
>         self::$_downloadedFiles[$config['_protocol']][$path] = $tempFile;
>
>         $this->_log('_downloadFile/s3 ok', $path, $tempFile);
>
>         return $tempFile;

@thumped, see if this updated patch helps with your bug. We've been seeing similar errors at NamePros. I've updated the original post if you'd rather replace the code manually.
 
Zenexer said:
Updated the fix in my previous post. Storing false in the cache breaks bdDataStorage_StreamWrapper_Abstract#stream_open(), and it's less than ideal for servers that don't have a reliable connection to S3.

Here's a proper patch:

Code: 
87,90d86
<
<         $tempFile = tempnam(XenForo_Helper_File::getTempDir(), 'bddss3df_');
<         self::$_downloadedFiles[$config['_protocol']][$path] = $tempFile;
<
93,98c89
<         if (!empty($object)) {
<             file_put_contents($tempFile, $object);
<
<             $this->_log('_downloadFile/s3 ok', $path, $tempFile);
<             return $tempFile;
<         } else {
---
>         if (empty($object)) {
101a93,100
>
>         $tempFile = tempnam(XenForo_Helper_File::getTempDir(), 'bddss3df_');
>         file_put_contents($tempFile, $object);
>         self::$_downloadedFiles[$config['_protocol']][$path] = $tempFile;
>
>         $this->_log('_downloadFile/s3 ok', $path, $tempFile);
>
>         return $tempFile;

@thumped, see if this updated patch helps with your bug. We've been seeing similar errors at NamePros. I've updated the original post if you'd rather replace the code manually.
Click to expand...

definitely hasn't helped with XFMG thumbnail creation

I'll try competition images now.

edit: nope, no luck.
 
thumped said:
definitely hasn't helped with XFMG thumbnail creation

I'll try competition images now.

edit: nope, no luck.
Click to expand...
tl;dr: Try commenting out or deleting line 441 of bdDataStorage_StreamWrapper_Abstract related to unlinking the file, which is this: "@unlink($this->_openedFileTempFile);"

bdDataStorage_StreamWrapper_Abstract -> stream_open() calls _tryDownloadFile(), which checks the _downloadedFiles array for a "cached" version of the requested file; if one is found, it simply returns the path of that local file, or if not, it proceeds to download it (and add its path to the _downloadedFiles array) via the _downloadFil0e() function.
The problem arises when stream_close() is called, which (on line 441 of bdDataStorage_StreamWrapper_Abstract) unlinks (deletes) the file opened by steam_open() and nullifies associated variables. However, it does not remove the file reference from the _downloadedFiles cache array, so when an attempt is made to open that stream/access that file again, the _tryDownloadFile() function will think the file still exists and pass its path from the _downloadedFiles array, when in reality, the file no longer exists. Thus, in the same request/while running the same script, the initial attempt to access a file will always work, but any subsequent ones will not.
I assume the add-on developer added that unlink procedure at the end of stream_close() in order to "address issue with too many files left in temp directory" (1.1.4 update). In my humble opinion, if this actually worked, it would defeat the purpose of cache-ing the files and is somewhat unnecessary since cleanUp() is run at the end of each script. It does have merit in scripts where many different images are accessed and can thus clutter the storage space. In that latter case though, a storage economy is contrasted against an increase in the number of re-downloads.
The quickest way to fix this then is to comment out/delete line 441 of bdDataStorage_StreamWrapper_Abstract - the "@unlink($this->_openedFileTempFile);" call in the stream_close() function. Alternatively, file "caching" can be disabled - I think this can be simply achieved by always forcing _tryDownloadFile() to download the file.

Please let me know if this helped; and if so, can we patch this problem, please? :)
 
A S said:
Please let me know if this helped; and if so, can we patch this problem, please? :)
Click to expand...

Finally got a chance to look into this, but no luck sadly.

As soon as I re-enable the BDDataStorage addon, I get a lot of broken thumbnails from XFMediaGallery content I've recently uploaded. (so i'd need to do the "Upload existing files in external data directory (/data) to your remote server before enable the add-on." part again - fair enough)

But attempting to regenerate the thumbnail within XFMG throws this error:

Code: 
Server Error

copy(/youtube_GOW0YJ2RBSc_thumb.jpg): failed to open stream: Permission denied

XenForo_Application::handlePhpError()
copy() in XenGallery/Helper/Image.php at line 831
XenGallery_Helper_Image->save() in XenGallery/Thumbnail/Abstract.php at line 142
XenGallery_Thumbnail_Abstract->saveThumbnail() in XenGallery/Thumbnail/Abstract.php at line 154
XenGallery_Thumbnail_Abstract->verifyThumbnailUrl() in XenGallery/Thumbnail/YouTube.php at line 33
XenGallery_Thumbnail_YouTube->getThumbnailUrl() in XenGallery/Model/Media.php at line 3232
XenGallery_Model_Media->getVideoThumbnailUrlFromParts() in XenGallery/Model/Media.php at line 931
XenGallery_Model_Media->deleteMediaThumbnail() in XenGallery/ControllerPublic/Media.php at line 2387
XenGallery_ControllerPublic_Media->actionThumbnailChange() in XenForo/FrontController.php at line 347
XenForo_FrontController->dispatch() in XenForo/FrontController.php at line 134
XenForo_FrontController->run() in /home/MYSITE/public_html/bbs/index.php at line 13

could this be an issue with permissions on the amazon side? I have no clue.
 
Last edited:
thumped said:
Finally got a chance to look into this, but no luck sadly.

As soon as I re-enable the BDDataStorage addon, I get a lot of broken thumbnails from XFMediaGallery content I've recently uploaded. (so i'd need to do the "Upload existing files in external data directory (/data) to your remote server before enable the add-on." part again - fair enough)

But attempting to regenerate the thumbnail within XFMG throws this error:

Code: 
Server Error

copy(/youtube_GOW0YJ2RBSc_thumb.jpg): failed to open stream: Permission denied

XenForo_Application::handlePhpError()
copy() in XenGallery/Helper/Image.php at line 831
XenGallery_Helper_Image->save() in XenGallery/Thumbnail/Abstract.php at line 142
XenGallery_Thumbnail_Abstract->saveThumbnail() in XenGallery/Thumbnail/Abstract.php at line 154
XenGallery_Thumbnail_Abstract->verifyThumbnailUrl() in XenGallery/Thumbnail/YouTube.php at line 33
XenGallery_Thumbnail_YouTube->getThumbnailUrl() in XenGallery/Model/Media.php at line 3232
XenGallery_Model_Media->getVideoThumbnailUrlFromParts() in XenGallery/Model/Media.php at line 931
XenGallery_Model_Media->deleteMediaThumbnail() in XenGallery/ControllerPublic/Media.php at line 2387
XenGallery_ControllerPublic_Media->actionThumbnailChange() in XenForo/FrontController.php at line 347
XenForo_FrontController->dispatch() in XenForo/FrontController.php at line 134
XenForo_FrontController->run() in /home/MYSITE/public_html/bbs/index.php at line 13

could this be an issue with permissions on the amazon side? I have no clue.
Click to expand...
Hmm, I failed to reproduce this on my servers. Switched between different S3 regions and it works fine. As long as the full file is in place, thumbnails can be rebuilt without issue. Are your attachment full files in the default location?
 
xfrocks said:
Hmm, I failed to reproduce this on my servers. Switched between different S3 regions and it works fine. As long as the full file is in place, thumbnails can be rebuilt without issue. Are your attachment full files in the default location?
Click to expand...
i'm using your attachment store addon to keep the attachments on S3 too

i tried again last night and copied all the existing /data/ files back up to s3 but still no luck with it. I freely admit I know next to nothing about working with s3.
 
thumped said:
i'm using your attachment store addon to keep the attachments on S3 too

i tried again last night and copied all the existing /data/ files back up to s3 but still no luck with it. I freely admit I know next to nothing about working with s3.
Click to expand...
Oh, are you on the latest version of [bd] Attachment Store? The error you got has more to do with [bd] Attachment Store than [bd] Data Storage.
 
thumped said:
1.2.1 of both

View attachment 126369
Click to expand...
Just ran the thumbnail rebuild with similar setup as yours and it went without issue. My guess is you have at least one missing attachment (local or S3, doesn't matter) and XFMG choked on that one as soon as it reached that particular file and stop running further. Please start a conversation with me, we will discuss further how to pin point that file and fix it.
 
You must log in or register to reply here.

Similar threads

K
  • Suggestion Suggestion
Duplicate Use presigned URLs for attachments when using external data storage
Replies
1
Views
1K
Chris D
Chris D
arn
  • Question Question
XF 2.2 Facebook integration data, Xenforo data storage
Replies
1
Views
580
Mike
Mike
Russ
  • Question Question
XF 2.1 Data-storage-type - cookie
Replies
1
Views
337
Xon
X
Alpha1
  • Suggestion Suggestion
Lack of interest Custom Field: Encrypted field & Maximum Data Storage Time
Replies
3
Views
781
Kirby
K
Back
Top Bottom