[bd] Attachment Store [Deleted]

I am currently attempting to disable this addon in order to be able to run a test upgrade to the XF2 Beta, however I am having some trouble moving the files back to use XF's default attachments system after using an external directory.

I am currently following these steps:
  1. Set storage options to 'Default' in Attachment Options
  2. Run move tool (selecting 'Store file in External Data directory')
  3. Disable addon (to confirm the move has been successful)
However, upon disabling the addon, I am encountering 404 errors whenever trying to grab any attachments anywhere on the site.
 
I am currently attempting to disable this addon in order to be able to run a test upgrade to the XF2 Beta, however I am having some trouble moving the files back to use XF's default attachments system after using an external directory.

I am currently following these steps:
  1. Set storage options to 'Default' in Attachment Options
  2. Run move tool (selecting 'Store file in External Data directory')
  3. Disable addon (to confirm the move has been successful)
However, upon disabling the addon, I am encountering 404 errors whenever trying to grab any attachments anywhere on the site.
After disabling it, what is your attachment URL? Please note that the attachment URL is updated only if user used ATTACH bb code in their post. If user copy pasted the URL from other posts or somewhere else, those URLs won't be updated.
 
Why use KeyCDN or AmazonS3 when BunnyCDN is CHEAPER :D
$0.02/GB/Mo for storage
$0.01/GB for transfer OUT to internet.
Free FTP upload bandwidth IN to BunnyCDN :)

Been using them and love it!
Here's the page for their pricing, very comparable with KeyCDN (Whom I have used but switched since then.)
https://bunnycdn.com/solutions/cdn-cloud-storage


Just updating this, but BunndyCDN now offers a "Large file" zone type for those hosting files and doesn't need files to be "at the closest server" to the user. Offers $0.005/GB for transfer OUT to the Internet, 50% savings from before! I LOVE THESE GUYS :D Perfect for anyone using this addon really and want something cheaper than Amazon S3
(I don't get paid by them or even affiliated with them, I just love them!)
 
Digital Ocean Storage Spaces is now available. Would love for this to be integrated into this addon.

https://blog.digitalocean.com/introducing-spaces-object-storage/
https://developers.digitalocean.com/documentation/spaces/

Spaces is available for a simple $5 per month price and includes 250GB of storage and 1TB of outbound bandwidth. There are no costs per request and additional storage is priced at the lowest rate available: $0.01 per GB transferred and $0.02 per GB stored. Uploads are free.
 
Is it possible to server images through this add-on to a different sub-domain than other CDN files configured in config.php? I.e data/js.
 
Sorry, I need urgent help, please (because my users are uploading new images already so the situation get mixed):

By mistake we did use the wrong Access Key ID, hurray, good job :(
With the wrong key in the options we used the tool to move all attachments and images in gallery to S3 (with CloudFlare). But of course they are not there.

We did keep local copy, all images are still in /data + /internal_data

What is the safest way to switch back? I assume the addon did change something in the database (S3 flag?), because just turning the addon off does not solve the 404.

Yes, I saw this https://xenforo.com/community/threads/bd-attachment-store-paid.49712/post-1174786, but since the images were not successfully moved I am unsure, what will work/is save. Try to avoid to delete local copy when moving back with "not existent" attachments from S3...

bdattachmentstore_engine and bdattachmentstore_options are all NULL in database in xf_attachment_data

@xfrocks @Jim Boy anybody, please?

As expected the file /internal_data/bdAttachmentStore_ShippableHelper_S3_makeRequest.log shows lots of errors because of the wrong authentification:


Code:
2017-11-29 07:35:28 PUT http://s3-eu-central-1.amazonaws.com:80/removed_for_post/2017/11/226125_fe1128c6822ee0c2fdd7b3e3e5467450.jpg array (
  'Content-Type' => 'image/jpeg',
  'x-amz-acl' => 'public-read',
  'Expect' => '100-continue',
  'x-amz-content-sha256' => '75aedef274c0eb19d22267fd9bb6c2d2f6c16ef81a27fa4bf4897e8f03648f83',
  'x-amz-date' => '20171129T073528Z',
  'Host' => 's3-eu-central-1.amazonaws.com',
  'Authorization' => 'AWS4-HMAC-SHA256 Credential=removed_for_post/20171129/eu-central-1/s3/aws4_request,SignedHeaders=content-type;expect;host;x-amz-acl;x-amz-content-sha256;x-amz-date,Signature=removed_for_post',
  'Content-type' => 'image/jpeg',
) -> 403 <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId>removed_for_post</AWSAccessKeyId><RequestId>ECC4FDCBEA7C6611</RequestId><HostId>/removed_for_post=</HostId></Error>
 
Last edited:
What does the "move" tool do?
  1. moving the files to new location & keep local file, if option is set?
  2. adding bdattachmentstore_engine and bdattachmentstore_options? NULL means "default", "s3" + BLOB means S3?
  3. anything more?

In my case, after moving all attachments from "default" to S3 (but with wrong key -> they did not go into S3) with local copy kept just shows NULL in bdattachmentstore_engine and bdattachmentstore_options for all attachments. For my taste this says "all attachments in default". But just switching back to "default" in options (without moving) OR disabling the addon does not show the old images.

Why, please?
Are the URLs to images somewhere written in database?
 
Last edited:
Thank you @graham_w , I really appreciate to get help and hints in such an situation, where hundreds of users are waitung for images to come back und to upload new. Thank you!

Yes, I would think "moveing back" should be the way too. But since the images never went to S3 (wrong key) I am unsure, if "moving" them (the non existant files) back will delete somehow the local copies. That's why I hesitate. Of course, later I have to try if now solution is found.

At the moment I try to understand, what happened to the /internal_data folders.
Lots of them are empty which were not in an earlier backup. But same time there are lots of new folders filled with images. At the moment, without checking in detail, I assume, that the local copies are not in their original folder, but in new folders. That would explain, why just turning the addon off or back to "default" does not bring back the images.

Can someone confirm that, that local copies are moved in newly created folders?
And... that moving back from "non existant in S3" won't delete the local copies?
 
Oha, just found 112000 temp files in /internal_data/temp - this is one for each attachment/media gallery image, from yesterday at the time I run the "move tool" with wrong key.

What is it @xfrocks, please?
The local copies or lost souls because the images could not go into S3?

And, in any case, my suggestion for next release: please validate authorization in addon before trying to move images. Of course, it was my typo, but... you know, it would have helped to avoid this ;)
 
So, It's as bad as thought. I just "moved" all attachments with rebuild caches after setting options to "Default".

The job did run very fast, around 3-5 minutes for 112000 images, so it did not copy files and afterwards the images are still not there. So I don't know to where it points, at least it does not use / copied the local copies to the correct places...
 
Sorry, I need urgent help, please (because my users are uploading new images already so the situation get mixed):

By mistake we did use the wrong Access Key ID, hurray, good job :(
With the wrong key in the options we used the tool to move all attachments and images in gallery to S3 (with CloudFlare). But of course they are not there.

We did keep local copy, all images are still in /data + /internal_data

What is the safest way to switch back? I assume the addon did change something in the database (S3 flag?), because just turning the addon off does not solve the 404.

Yes, I saw this https://xenforo.com/community/threads/bd-attachment-store-paid.49712/post-1174786, but since the images were not successfully moved I am unsure, what will work/is save. Try to avoid to delete local copy when moving back with "not existent" attachments from S3...

bdattachmentstore_engine and bdattachmentstore_options are all NULL in database in xf_attachment_data

@xfrocks @Jim Boy anybody, please?

As expected the file /internal_data/bdAttachmentStore_ShippableHelper_S3_makeRequest.log shows lots of errors because of the wrong authentification:


Code:
2017-11-29 07:35:28 PUT http://s3-eu-central-1.amazonaws.com:80/removed_for_post/2017/11/226125_fe1128c6822ee0c2fdd7b3e3e5467450.jpg array (
  'Content-Type' => 'image/jpeg',
  'x-amz-acl' => 'public-read',
  'Expect' => '100-continue',
  'x-amz-content-sha256' => '75aedef274c0eb19d22267fd9bb6c2d2f6c16ef81a27fa4bf4897e8f03648f83',
  'x-amz-date' => '20171129T073528Z',
  'Host' => 's3-eu-central-1.amazonaws.com',
  'Authorization' => 'AWS4-HMAC-SHA256 Credential=removed_for_post/20171129/eu-central-1/s3/aws4_request,SignedHeaders=content-type;expect;host;x-amz-acl;x-amz-content-sha256;x-amz-date,Signature=removed_for_post',
  'Content-type' => 'image/jpeg',
) -> 403 <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId>removed_for_post</AWSAccessKeyId><RequestId>ECC4FDCBEA7C6611</RequestId><HostId>/removed_for_post=</HostId></Error>
If your config was incorrect, the old files should not be changed at all. Just update the config and rerun the move tool.
 
Thank you, xfrocks,for hooking in.

It was like this:
  • Wrong key (auth)
  • Local copy checked
  • Move tool tried for 112k images though
  • The xml-file shows 112k auth errors at AWS
  • In internal_data/temp there are 112k temp renamed files
  • All 112k files in 113 dirs in internal_data/attachments were moved to newly created 113 dirs (so total 226) and renamed according the new dirs
  • The former dirs were emptied
Move back with options set to default and rerunning did not work.

Would have rerunning the tool have solved the 112k temp files, new dirs and renamed files?

I cannot test, I reverted back from backup and manually searched and renamed around 250 images, which were uploaded after the backup (half day).

Now the folders are original from 0 to 113, next is 266 with around 250 new manually renamed images. New images add now to this folder.
 
Again I need urgent help. Really, please.

Now I currently move all images to S3, it seems to work. In S3 there are new folders (year/month structure) with lots of images.
I did mark "local copy" before moving.

The move tool again creates new folders in internal_data/
So, folders 0-113 for 113k images are now empty, and the tool created folders 114-266, which contain the images. This might be correct, I don't know. Is this the expected correct behaviour?

But the thumbs in data/ are still in their folders 0-113 (not 114-266). So at the moment (still moving) all thumbs are broken, because URLs of thumbs now use folders 114-266 - which do not existing (yet??).

Are the thumbs / thumb folders synced at the end of moving? Or is this an error?

Any hint to calm me down and to make it possible to reopen my forums again is appreciated. Anybody, please?
 
Last edited:
Again I need urgent help. Really, please.

Now I currently move all images to S3, it seems to work. In S3 there are new folders (year/month structure) with lots of images.
I did mark "local copy" before moving.

The move tool again creates new folders in internal_data/
So, folders 0-113 for 113k images are now empty, and the tool created folders 114-266, which contain the images. This might be correct, I don't know. Is this the expected correct behaviour?
If you checked keep local copy, the tool will copy files from 0-113 to 114-266, basically it's making a duplicate of the old data, mirroring the S3 files.

But the thumbs in data/ are still in their folders 0-113 (not 114-266). So at the moment (still moving) all thumbs are broken, because URLs of thumbs now use folders 114-266 - which do not existing (yet??).

Are the thumbs / thumb folders synced at the end of moving? Or is this an error?

Any hint to calm me down and to make it possible to reopen my forums again is appreciated. Anybody, please?

Similarly, the thumbnail should be copied within data/ (from 0-133 to 114-266). You should see the new folders being created while moving. Also, the thumbnail URL even during the move, should start pointing to S3 instead of pointing to data/. The files in data/ and internal_data/ are for backup only, they are not used by the add-on.
 
thank you for your fast response.

I see, there are new folders 114-266 created in data/attachments.
No now folders are created in data/xengallery - and these folders are used in xen media gallery now, and they are not there.

I see thumbs laying in S3 - but thumbs are broken in Xen Media Gallery:

1512227513117.webp
 
so, the new folders are fine - but xengallery does not get correct copy folders and while moving ALL thumbs in Xnegallery are broken.

Can we save that or do I have to use a backup to revert everything. Sorry, it's really urgent since it's main season and forums is closed for a day already.
 
thank you for your fast response.

I see, there are new folders 114-266 created in data/attachments.
No now folders are created in data/xengallery - and these folders are used in xen media gallery now, and they are not there.

I see thumbs laying in S3 - but thumbs are broken in Xen Media Gallery:

View attachment 163082
You will need to rebuild XFMG thumbnails after the move. IIRC they have a rebuild tool for thumbnails in AdminCP > Tools > Rebuild caches.
 
Ahhh, that sounds logical. Recreate thumbs...
They will then be saved to folders 114-266 within data/xengallery?

Sorry, I am confused - are the thumbs used from S3 or data/ ? (I understood S3).
When I recreate thumbs in xengallery, will they be saved to S3? And what is with the thumbs already laying in S3?

But at the moment you say: all is fine/correct?
 
XFMG uses non standard size thumbnails (they are larger than normal forum attachment thumbnail) and they store it in data/xenmedia or similar (sorry I don't have an active installation to check). Those won't be uploaded to S3 though.
 
Top Bottom