[bd] Data Storage [Deleted]

xfrocks · Jun 2, 2015

thumped said:
brand new competition with a brand new thumbnail image

Did you made the changes as mentioned here? https://xenforo.com/community/threads/competitions-for-xenforo-paid.59696/page-11#post-666866

thumped · Jun 2, 2015

xfrocks said:
Did you made the changes as mentioned here? https://xenforo.com/community/threads/competitions-for-xenforo-paid.59696/page-11#post-666866

i did indeed

thumped · Jun 2, 2015

also having XFMG issues with thumbnail image creation

Code:

ErrorException: getimagesize(): Read error! - library/XenGallery/Helper/Image.php:84
Generated By: pete, 7 minutes ago
Stack Trace
#0 [internal function]: XenForo_Application::handlePhpError(8, 'getimagesize():...', '/home/thumped/p...', 84, Array)
#1 /home/thumped/public_html/bbs/library/XenGallery/Helper/Image.php(84): getimagesize('ds0:///xengalle...')
#2 /home/thumped/public_html/bbs/library/XenGallery/Helper/Image.php(54): XenGallery_Helper_Image->_readImageInfo()
#3 /home/thumped/public_html/bbs/library/XenGallery/Thumbnail/Abstract.php(136): XenGallery_Helper_Image->__construct('ds0:///xengalle...')
#4 /home/thumped/public_html/bbs/library/XenGallery/Thumbnail/Abstract.php(151): XenGallery_Thumbnail_Abstract->saveThumbnail()
#5 /home/thumped/public_html/bbs/library/XenGallery/Thumbnail/YouTube.php(33): XenGallery_Thumbnail_Abstract->verifyThumbnailUrl('https://i.ytimg...')
#6 /home/thumped/public_html/bbs/library/XenGallery/Model/Media.php(3207): XenGallery_Thumbnail_YouTube->getThumbnailUrl('qOR0HxzxM6Q')
#7 /home/thumped/public_html/bbs/library/XenGallery/ControllerPublic/Media.php(2441): XenGallery_Model_Media->getVideoThumbnailUrlFromParts(Array)
#8 /home/thumped/public_html/bbs/library/XenForo/FrontController.php(347): XenGallery_ControllerPublic_Media->actionPreviewVideo()
#9 /home/thumped/public_html/bbs/library/XenForo/FrontController.php(134): XenForo_FrontController->dispatch(Object(XenForo_RouteMatch))
#10 /home/thumped/public_html/bbs/index.php(13): XenForo_FrontController->run()
#11 {main}
Request State
array(3) {
  ["url"] => string(98) "https://thumped.com/bbs/index.php?xengallery/preview-video&_xfResponseType=json&media_id=undefined"
  ["_GET"] => array(3) {
    ["xengallery/preview-video"] => string(0) ""
    ["_xfResponseType"] => string(4) "json"
    ["media_id"] => string(9) "undefined"
  }
  ["_POST"] => array(5) {
    ["embed_url"] => string(43) "https://www.youtube.com/watch?v=qOR0HxzxM6Q"
    ["_xfRequestUri"] => string(14) "/bbs/media/add"
    ["_xfNoRedirect"] => string(1) "1"
    ["_xfToken"] => string(8) "********"
    ["_xfResponseType"] => string(4) "json"
  }
}

Code:

ErrorException: stat(): stat failed for /home/thumped/public_html/bbs/internal_data/temp/bddss3df_50o0NQ - library/bdDataStorage/StreamWrapper/Abstract.php:182
Generated By: pete, 7 minutes ago
Stack Trace
#0 [internal function]: XenForo_Application::handlePhpError(2, 'stat(): stat fa...', '/home/thumped/p...', 182, Array)
#1 /home/thumped/public_html/bbs/library/bdDataStorage/StreamWrapper/Abstract.php(182): stat('/home/thumped/p...')
#2 /home/thumped/public_html/bbs/library/bdDataStorage/StreamWrapper/Abstract.php(483): bdDataStorage_StreamWrapper_Abstract->_cacheSaveFile(Array, 'xengallery/yout...', '/home/thumped/p...')
#3 [internal function]: bdDataStorage_StreamWrapper_Abstract->stream_open('ds0:///xengalle...', 'rb', 16, NULL)
#4 /home/thumped/public_html/bbs/library/XenGallery/Helper/Image.php(84): getimagesize('ds0:///xengalle...')
#5 /home/thumped/public_html/bbs/library/XenGallery/Helper/Image.php(1077): XenGallery_Helper_Image->_readImageInfo()
#6 /home/thumped/public_html/bbs/library/XenGallery/Helper/Image.php(196): XenGallery_Helper_Image->getRatioWidthToHeight()
#7 /home/thumped/public_html/bbs/library/XenGallery/Thumbnail/Abstract.php(137): XenGallery_Helper_Image->resize('300', '169', 'crop')
#8 /home/thumped/public_html/bbs/library/XenGallery/Thumbnail/Abstract.php(151): XenGallery_Thumbnail_Abstract->saveThumbnail()
#9 /home/thumped/public_html/bbs/library/XenGallery/Thumbnail/YouTube.php(33): XenGallery_Thumbnail_Abstract->verifyThumbnailUrl('https://i.ytimg...')
#10 /home/thumped/public_html/bbs/library/XenGallery/Model/Media.php(3207): XenGallery_Thumbnail_YouTube->getThumbnailUrl('qOR0HxzxM6Q')
#11 /home/thumped/public_html/bbs/library/XenGallery/ControllerPublic/Media.php(2441): XenGallery_Model_Media->getVideoThumbnailUrlFromParts(Array)
#12 /home/thumped/public_html/bbs/library/XenForo/FrontController.php(347): XenGallery_ControllerPublic_Media->actionPreviewVideo()
#13 /home/thumped/public_html/bbs/library/XenForo/FrontController.php(134): XenForo_FrontController->dispatch(Object(XenForo_RouteMatch))
#14 /home/thumped/public_html/bbs/index.php(13): XenForo_FrontController->run()
#15 {main}
Request State
array(3) {
  ["url"] => string(98) "https://thumped.com/bbs/index.php?xengallery/preview-video&_xfResponseType=json&media_id=undefined"
  ["_GET"] => array(3) {
    ["xengallery/preview-video"] => string(0) ""
    ["_xfResponseType"] => string(4) "json"
    ["media_id"] => string(9) "undefined"
  }
  ["_POST"] => array(5) {
    ["embed_url"] => string(43) "https://www.youtube.com/watch?v=qOR0HxzxM6Q"
    ["_xfRequestUri"] => string(14) "/bbs/media/add"
    ["_xfNoRedirect"] => string(1) "1"
    ["_xfToken"] => string(8) "********"
    ["_xfResponseType"] => string(4) "json"
  }
}

PaulB · Jun 4, 2015

This method in bdDataStorage_StreamWrapper_S3 is problematic:

PHP:

protected function _downloadFile(array $config, $path)
{
    $connection = $this->_openConnection($config);

    $tempFile = tempnam(XenForo_Helper_File::getTempDir(), 'bddss3df_');
    self::$_downloadedFiles[$config['_protocol']][$path] = $tempFile;

    $object = $connection->getObject($config['bucket'] . '/' . $path);

    if (!empty($object)) {
        file_put_contents($tempFile, $object);

        $this->_log('_downloadFile/s3 ok', $path, $tempFile);
        return $tempFile;
    } else {
        $this->_log('_downloadFile/s3 failed', $path);
        return false;
    }
}

If $path is cached in $_downloadedFiles, the function won't return false when it's supposed to; it will always return a path. This causes url_stat in bdDataStorage_StreamWrapper_Abstract to erroneously report that directories are files. This, in turn, confuses XenForo_Helper_File::createDirectory(), which returns false because it thinks the directory already exists as a file. As a result, XenForo_Model_Avatar::_writeAvatar() returns false without calling XenForo_Helper_File::safeRename().

Whether this bug results in noticeable problems seems to depend on the environment. It randomly started preventing all avatar uploads on our development site, but wasn't causing any problems in production.

Here's a patched version of the method:

PHP:

protected function _downloadFile(array $config, $path)
{
    $connection = $this->_openConnection($config);
    $object = $connection->getObject($config['bucket'] . '/' . $path);

    if (empty($object)) {
        $this->_log('_downloadFile/s3 failed', $path);
        return false;
    }

    $tempFile = tempnam(XenForo_Helper_File::getTempDir(), 'bddss3df_');
    file_put_contents($tempFile, $object);
    self::$_downloadedFiles[$config['_protocol']][$path] = $tempFile;

    $this->_log('_downloadFile/s3 ok', $path, $tempFile);

    return $tempFile;
}

thumped · Jun 5, 2015

more XFMG error action

An exception occurred: Cannot save file /youtube_E03tbipoDkE_thumb.jpg in /home/thumped/public_html/bbs/library/XenGallery/Helper/Image.php on line 833
XenGallery_Helper_Image->save() in XenGallery/Thumbnail/Abstract.php at line 139
XenGallery_Thumbnail_Abstract->saveThumbnail() in XenGallery/Thumbnail/Abstract.php at line 151
XenGallery_Thumbnail_Abstract->verifyThumbnailUrl() in XenGallery/Thumbnail/YouTube.php at line 33
XenGallery_Thumbnail_YouTube->getThumbnailUrl() in XenGallery/Model/Media.php at line 3207
XenGallery_Model_Media->getVideoThumbnailUrlFromParts() in XenGallery/Model/Media.php at line 1067
XenGallery_Model_Media->prepareMedia() in XenGallery/Model/Media.php at line 1130
XenGallery_Model_Media->prepareMediaItems() in XenGallery/Callback.php at line 76
XenGallery_Callback::getMediaForBlock()
call_user_func() in XenForo/Template/Abstract.php at line 330
XenForo_Template_Abstract->callTemplateCallback() in /home/thumped/public_html/bbs/internal_data/templates/S.12,L.1,xengallery_media_block_sidebar.php at line 41
include() in XenForo/Template/Abstract.php at line 260
XenForo_Template_Abstract->_renderInternal() in XenForo/Template/Abstract.php at line 191
XenForo_Template_Abstract->render() in XenForo/Template/Public.php at line 110
XenForo_Template_Public->render() in WidgetFramework/WidgetRenderer/Template.php at line 57
WidgetFramework_WidgetRenderer_Template->_render() in WidgetFramework/WidgetRenderer.php at line 660
WidgetFramework_WidgetRenderer->render() in WidgetFramework/Core.php at line 549
WidgetFramework_Core->_renderWidgetsFor_renderWidgetsContainer() in WidgetFramework/Core.php at line 473
WidgetFramework_Core->_renderWidgetsFor() in WidgetFramework/Core.php at line 377
WidgetFramework_Core->renderWidgetsFor() in WidgetFramework/Listener.php at line 100
WidgetFramework_Listener::template_post_render()
call_user_func_array() in XenForo/CodeEvent.php at line 58
XenForo_CodeEvent::fire() in XenForo/Template/Abstract.php at line 195
XenForo_Template_Abstract->render() in XenForo/Template/Public.php at line 110
XenForo_Template_Public->render() in XenForo/Template/Abstract.php at line 528
XenForo_Template_Abstract->__toString() in /home/thumped/public_html/bbs/internal_data/templates/S.12,L.1,find_new_wrapper.php at line 995
include() in XenForo/Template/Abstract.php at line 260
XenForo_Template_Abstract->_renderInternal() in XenForo/Template/Abstract.php at line 191
XenForo_Template_Abstract->render() in XenForo/Template/Public.php at line 110
XenForo_Template_Public->render() in XenForo/ViewRenderer/HtmlPublic.php at line 123
XenForo_ViewRenderer_HtmlPublic->renderContainer() in XenForo/FrontController.php at line 618
XenForo_FrontController->renderView() in XenForo/FrontController.php at line 158
XenForo_FrontController->run() in /home/thumped/public_html/bbs/index.php at line 13

i've had to disable this addon completely now because of this. if a fix isn't possible, is there any way it can be selectively disabled for certain addons?

thumped · Jun 10, 2015

featured thread error when promoting to slider / adding an image

Code:

getimagesize(ds0:///featured_threads/icons/95/95700.jpg): failed to open stream: "bdDataStorage_StreamWrapper_S3::stream_open" call failed

XenForo_Application::handlePhpError()
getimagesize() in CTA/FeaturedThreads/Helper/Image.php at line 44
CTA_FeaturedThreads_Helper_Image->_readImageInfo() in CTA/FeaturedThreads/Helper/Image.php at line 80
CTA_FeaturedThreads_Helper_Image->_setNewMainImage() in CTA/FeaturedThreads/Helper/Image.php at line 273
CTA_FeaturedThreads_Helper_Image->saveToPath() in CTA/FeaturedThreads/Model/Featured.php at line 471
CTA_FeaturedThreads_Model_Featured->applyFeaturedThreadImage() in CTA/FeaturedThreads/Model/Featured.php at line 399
CTA_FeaturedThreads_Model_Featured->uploadUrlFeaturedThreadImage() in CTA/FeaturedThreads/ControllerPublic/Thread.php at line 997
CTA_FeaturedThreads_ControllerPublic_Thread->actionFeaturedThreadImages() in XenForo/FrontController.php at line 347
XenForo_FrontController->dispatch() in XenForo/FrontController.php at line 134
XenForo_FrontController->run() in /home/thumped/public_html/bbs/index.php at line 13

halp!

PaulB · Jul 9, 2015

Updated the fix in my previous post. Storing false in the cache breaks bdDataStorage_StreamWrapper_Abstract#stream_open(), and it's less than ideal for servers that don't have a reliable connection to S3.

Here's a proper patch:

Code:

87,90d86
<
<         $tempFile = tempnam(XenForo_Helper_File::getTempDir(), 'bddss3df_');
<         self::$_downloadedFiles[$config['_protocol']][$path] = $tempFile;
<
93,98c89
<         if (!empty($object)) {
<             file_put_contents($tempFile, $object);
<
<             $this->_log('_downloadFile/s3 ok', $path, $tempFile);
<             return $tempFile;
<         } else {
---
>         if (empty($object)) {
101a93,100
>
>         $tempFile = tempnam(XenForo_Helper_File::getTempDir(), 'bddss3df_');
>         file_put_contents($tempFile, $object);
>         self::$_downloadedFiles[$config['_protocol']][$path] = $tempFile;
>
>         $this->_log('_downloadFile/s3 ok', $path, $tempFile);
>
>         return $tempFile;

@thumped, see if this updated patch helps with your bug. We've been seeing similar errors at NamePros. I've updated the original post if you'd rather replace the code manually.

thumped · Jul 10, 2015

Zenexer said:
Updated the fix in my previous post. Storing false in the cache breaks bdDataStorage_StreamWrapper_Abstract#stream_open(), and it's less than ideal for servers that don't have a reliable connection to S3.

Here's a proper patch:

Code:

87,90d86 < < $tempFile = tempnam(XenForo_Helper_File::getTempDir(), 'bddss3df_'); < self::$_downloadedFiles[$config['_protocol']][$path] = $tempFile; < 93,98c89 < if (!empty($object)) { < file_put_contents($tempFile, $object); < < $this->_log('_downloadFile/s3 ok', $path, $tempFile); < return $tempFile; < } else { --- > if (empty($object)) { 101a93,100 > > $tempFile = tempnam(XenForo_Helper_File::getTempDir(), 'bddss3df_'); > file_put_contents($tempFile, $object); > self::$_downloadedFiles[$config['_protocol']][$path] = $tempFile; > > $this->_log('_downloadFile/s3 ok', $path, $tempFile); > > return $tempFile;

@thumped, see if this updated patch helps with your bug. We've been seeing similar errors at NamePros. I've updated the original post if you'd rather replace the code manually.

definitely hasn't helped with XFMG thumbnail creation

I'll try competition images now.

edit: nope, no luck.

A S · Jul 23, 2015

thumped said:
definitely hasn't helped with XFMG thumbnail creation

I'll try competition images now.

edit: nope, no luck.

tl;dr: Try commenting out or deleting line 441 of bdDataStorage_StreamWrapper_Abstract related to unlinking the file, which is this: "@unlink($this->_openedFileTempFile);"

bdDataStorage_StreamWrapper_Abstract -> stream_open() calls _tryDownloadFile(), which checks the _downloadedFiles array for a "cached" version of the requested file; if one is found, it simply returns the path of that local file, or if not, it proceeds to download it (and add its path to the _downloadedFiles array) via the _downloadFil0e() function.
The problem arises when stream_close() is called, which (on line 441 of bdDataStorage_StreamWrapper_Abstract) unlinks (deletes) the file opened by steam_open() and nullifies associated variables. However, it does not remove the file reference from the _downloadedFiles cache array, so when an attempt is made to open that stream/access that file again, the _tryDownloadFile() function will think the file still exists and pass its path from the _downloadedFiles array, when in reality, the file no longer exists. Thus, in the same request/while running the same script, the initial attempt to access a file will always work, but any subsequent ones will not.
I assume the add-on developer added that unlink procedure at the end of stream_close() in order to "address issue with too many files left in temp directory" (1.1.4 update). In my humble opinion, if this actually worked, it would defeat the purpose of cache-ing the files and is somewhat unnecessary since cleanUp() is run at the end of each script. It does have merit in scripts where many different images are accessed and can thus clutter the storage space. In that latter case though, a storage economy is contrasted against an increase in the number of re-downloads.
The quickest way to fix this then is to comment out/delete line 441 of bdDataStorage_StreamWrapper_Abstract - the "@unlink($this->_openedFileTempFile);" call in the stream_close() function. Alternatively, file "caching" can be disabled - I think this can be simply achieved by always forcing _tryDownloadFile() to download the file.

Please let me know if this helped; and if so, can we patch this problem, please?

thumped · Aug 2, 2015

A S said:
Please let me know if this helped; and if so, can we patch this problem, please?

Finally got a chance to look into this, but no luck sadly.

As soon as I re-enable the BDDataStorage addon, I get a lot of broken thumbnails from XFMediaGallery content I've recently uploaded. (so i'd need to do the "Upload existing files in external data directory (/data) to your remote server before enable the add-on." part again - fair enough)

But attempting to regenerate the thumbnail within XFMG throws this error:

Code:

Server Error

copy(/youtube_GOW0YJ2RBSc_thumb.jpg): failed to open stream: Permission denied

XenForo_Application::handlePhpError()
copy() in XenGallery/Helper/Image.php at line 831
XenGallery_Helper_Image->save() in XenGallery/Thumbnail/Abstract.php at line 142
XenGallery_Thumbnail_Abstract->saveThumbnail() in XenGallery/Thumbnail/Abstract.php at line 154
XenGallery_Thumbnail_Abstract->verifyThumbnailUrl() in XenGallery/Thumbnail/YouTube.php at line 33
XenGallery_Thumbnail_YouTube->getThumbnailUrl() in XenGallery/Model/Media.php at line 3232
XenGallery_Model_Media->getVideoThumbnailUrlFromParts() in XenGallery/Model/Media.php at line 931
XenGallery_Model_Media->deleteMediaThumbnail() in XenGallery/ControllerPublic/Media.php at line 2387
XenGallery_ControllerPublic_Media->actionThumbnailChange() in XenForo/FrontController.php at line 347
XenForo_FrontController->dispatch() in XenForo/FrontController.php at line 134
XenForo_FrontController->run() in /home/MYSITE/public_html/bbs/index.php at line 13

could this be an issue with permissions on the amazon side? I have no clue.

xfrocks · Jan 11, 2016

xfrocks updated [bd] Data Storage with a new update entry:

Feature Update

Added support for custom meta data (S3)

Added support for XenForo built-in sitemap

Added daily version checking from xfrocks.com

Fixed issues with non-standard S3 regions

Read the rest of this update entry...

thumped · Jan 11, 2016

@xfrocks did you get a chance to look at the errors i reported up above? tried the latest version but it hasn't helped.

xfrocks · Jan 12, 2016

thumped said:
@xfrocks did you get a chance to look at the errors i reported up above? tried the latest version but it hasn't helped.

Tracking this issue right now. Hang on, and sorry for the inconvenience.

xfrocks · Jan 14, 2016

thumped said:
Finally got a chance to look into this, but no luck sadly.

As soon as I re-enable the BDDataStorage addon, I get a lot of broken thumbnails from XFMediaGallery content I've recently uploaded. (so i'd need to do the "Upload existing files in external data directory (/data) to your remote server before enable the add-on." part again - fair enough)

But attempting to regenerate the thumbnail within XFMG throws this error:

Code:

Server Error copy(/youtube_GOW0YJ2RBSc_thumb.jpg): failed to open stream: Permission denied XenForo_Application::handlePhpError() copy() in XenGallery/Helper/Image.php at line 831 XenGallery_Helper_Image->save() in XenGallery/Thumbnail/Abstract.php at line 142 XenGallery_Thumbnail_Abstract->saveThumbnail() in XenGallery/Thumbnail/Abstract.php at line 154 XenGallery_Thumbnail_Abstract->verifyThumbnailUrl() in XenGallery/Thumbnail/YouTube.php at line 33 XenGallery_Thumbnail_YouTube->getThumbnailUrl() in XenGallery/Model/Media.php at line 3232 XenGallery_Model_Media->getVideoThumbnailUrlFromParts() in XenGallery/Model/Media.php at line 931 XenGallery_Model_Media->deleteMediaThumbnail() in XenGallery/ControllerPublic/Media.php at line 2387 XenGallery_ControllerPublic_Media->actionThumbnailChange() in XenForo/FrontController.php at line 347 XenForo_FrontController->dispatch() in XenForo/FrontController.php at line 134 XenForo_FrontController->run() in /home/MYSITE/public_html/bbs/index.php at line 13

could this be an issue with permissions on the amazon side? I have no clue.

Hmm, I failed to reproduce this on my servers. Switched between different S3 regions and it works fine. As long as the full file is in place, thumbnails can be rebuilt without issue. Are your attachment full files in the default location?

thumped · Jan 14, 2016

xfrocks said:
Hmm, I failed to reproduce this on my servers. Switched between different S3 regions and it works fine. As long as the full file is in place, thumbnails can be rebuilt without issue. Are your attachment full files in the default location?

i'm using your attachment store addon to keep the attachments on S3 too

i tried again last night and copied all the existing /data/ files back up to s3 but still no luck with it. I freely admit I know next to nothing about working with s3.

xfrocks · Jan 15, 2016

thumped said:
i'm using your attachment store addon to keep the attachments on S3 too

i tried again last night and copied all the existing /data/ files back up to s3 but still no luck with it. I freely admit I know next to nothing about working with s3.

Oh, are you on the latest version of [bd] Attachment Store? The error you got has more to do with [bd] Attachment Store than [bd] Data Storage.

thumped · Jan 15, 2016

xfrocks said:
Oh, are you on the latest version of [bd] Attachment Store? The error you got has more to do with [bd] Attachment Store than [bd] Data Storage.

1.2.1 of both

xfrocks · Jan 18, 2016

thumped said:
1.2.1 of both

View attachment 126369

Just ran the thumbnail rebuild with similar setup as yours and it went without issue. My guess is you have at least one missing attachment (local or S3, doesn't matter) and XFMG choked on that one as soon as it reached that particular file and stop running further. Please start a conversation with me, we will discuss further how to pin point that file and fix it.

Rey Tran · Jan 22, 2016

What about full attachments? Will this only work with thumbnails?

eagle eyes · Apr 1, 2016

I need to disable (uninstall) this addon as to use the xenforo per default settings. How can i do that?

[bd] Data Storage [Deleted]

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

New member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

New member

Active member

Similar threads

We value your privacy