[bd] Attachment Store [Deleted]

[xfrocks]

Also can you explain in more detail the following:

We have 62GB of attachments we would like to move to S3. Right now we are uploading attachments directly to S3 and would like it to stay that way.

I notice at the bottom of the cache rebuild page there are the following:

Move Attachment Data

and

Update Attachment Data Storage Options

I am not sure which one I need or do I need both?

To move, you only need "Move Attachment Data". The other tool is for changing configuration data for old attachments (which you don't need now).
If you want to move back from S3 to default, select default in the add-on option page then use the "Move Attachment Data" tool.

So, I changed my attachment store options from using s3 + cloudfront to just s3 (no cloudfront). All of my attachments look ok in the forums, but in @Chris D Xen Media Gallery, thumbnails for all NEW media uploads are broken... until I rebuild them, then they're fine again. Old media (pre-change) are all ok. Is there something I can do?

I will need to look at the site to understand the problem. Can you send me your site via conversation?

 

[Main Company]

Go to AdminCP > Tools > Rebuild Caches >Move Attachment Data

Okay, just so I am clear, I see 2 Rebuild options for attachments on that page - both with "Rebuild Now" options. Here's a screenshot:



So, what is the proper order of our next steps, after having setup FTP in ACP >> Options >> Attachments? (We used the FTP option.). Is it the following?

1. Select the options and input the information in 'Update Attachment Data Storage Options"
2. Then "Rebuild Now" in Move Attachment Data
3. Then "Rebuild Now" in 'Update Attachment Data Storage Options"

I'm still a little confused about the order of events, and I am trying to be cautious. Additionally, if there are directions available that lay this out in detail, please direct me.

 

[MattW]

Okay, so I asked our technical person to step-in to help implement this and he set it up to store the files on another drive, within the same server, using the FTP functionality (as shown in the image below).

My understanding is that before we enabled BD Attachment addon, the way XF handled an attachment was to:

1. Store the attachment in xenforo/data/accachments directory in xxxx.data format.
2. Add a hash record in XF DB, where this hash record keeps the mapping between the real file name and the xxxx.data file.

Now, when we add a new attachment, it will be directly put into folder: /data2/xenforo/data, with the filename: {id}_{filename}.{extension}, and a hash record in the XF DB will still be created.

My question is: how do we migrate the attachments that we uploaded to XF, before we enabled BD Attachments Addon, to the new location?

View attachment 79004

Why use FTP on the local host? If the directory you want to store the images in is writable by the webserver, just create a symlink from the xenforo/data/attachments directory on the new drive to the existing data location in the forum home.

This is how I serve mine via a different domain name

[root@host public]# ls -al
total 8
drwxr-xr-x 2 root root 4096 Aug  3 11:12 .
drwxr-xr-x 4 root root 4096 Aug  3 11:17 ..
lrwxrwxrwx 1 root root   34 Aug  3 11:12 data -> /home/www/z22se.co.uk/public/data/
lrwxrwxrwx 1 root root   32 Aug  3 11:12 js -> /home/www/z22se.co.uk/public/js/
lrwxrwxrwx 1 root root   36 Aug  3 11:12 styles -> /home/www/z22se.co.uk/public/styles/
[root@host public]#

 

[kontrabass]

To move, you only need "Move Attachment Data". The other tool is for changing configuration data for old attachments (which you don't need now).
If you want to move back from S3 to default, select default in the add-on option page then use the "Move Attachment Data" tool.


I will need to look at the site to understand the problem. Can you send me your site via conversation?

Thanks, I just decided to to move everything to local storage anyway for now. All good

 

[Main Company]

Why use FTP on the local host? If the directory you want to store the images in is writable by the webserver, just create a symlink from the xenforo/data/attachments directory on the new drive to the existing data location in the forum home.

This is how I serve mine via a different domain name

[root@host public]# ls -al
total 8
drwxr-xr-x 2 root root 4096 Aug  3 11:12 .
drwxr-xr-x 4 root root 4096 Aug  3 11:17 ..
lrwxrwxrwx 1 root root   34 Aug  3 11:12 data -> /home/www/z22se.co.uk/public/data/
lrwxrwxrwx 1 root root   32 Aug  3 11:12 js -> /home/www/z22se.co.uk/public/js/
lrwxrwxrwx 1 root root   36 Aug  3 11:12 styles -> /home/www/z22se.co.uk/public/styles/
[root@host public]#

yes, based on your earlier post I had made this suggestion to our tech and right now I can't recall why he went the ftp route.

 

[Main Company]

To move, you only need "Move Attachment Data". The other tool is for changing configuration data for old attachments (which you don't need now). If you want to move back from S3 to default, select default in the add-on option page then use the "Move Attachment Data" tool.

I could really use some feedback on this.

I tried "Move Attachment Data" an now I have approx 500 attachments that are not longer being displayed.

 

[xfrocks]

Okay, just so I am clear, I see 2 Rebuild options for attachments on that page - both with "Rebuild Now" options. Here's a screenshot:

View attachment 79960

So, what is the proper order of our next steps, after having setup FTP in ACP >> Options >> Attachments? (We used the FTP option.). Is it the following?

1. Select the options and input the information in 'Update Attachment Data Storage Options"
2. Then "Rebuild Now" in Move Attachment Data
3. Then "Rebuild Now" in 'Update Attachment Data Storage Options"

I'm still a little confused about the order of events, and I am trying to be cautious. Additionally, if there are directions available that lay this out in detail, please direct me.

If you have configured FTP in Options > Attachments. All you have to do is to click the "Rebuild Now" in "Move Attachment Data" to move existing attachments to FTP server.

 

[Main Company]

If you have configured FTP in Options > Attachments. All you have to do is to click the "Rebuild Now" in "Move Attachment Data" to move existing attachments to FTP server.

What, exactly, does "Update Attachment data storage option" do?

In our case we have FTP properly configured in Options > Attachments to upload all new attachments to Data2, which is another drive on our server. However, some attachments had been uploaded prior to the installation of this addon (July 27th), and so they resided in XF's default location ("xenforo/data/attachments") .

After using "Rebuild Now" in "Move Attachment Data", attachments uploaded prior to July 27th are displayed in the attachment browser, but are no longer accessible when I try to access them via the system. Hovering over the thumbnail display an URL from the original location ("xenforo/data/attachments") , not the new ("data2") location. I've got backups of all of the attachments, how do you suggest we proceed?

 

[xfrocks]

What, exactly, does "Update Attachment data storage option" do?

In our case we have FTP properly configured in Options > Attachments to upload all new attachments to Data2, which is another drive on our server. However, some attachments had been uploaded prior to the installation of this addon (July 27th), and so they resided in XF's default location ("xenforo/data/attachments") .

After using "Rebuild Now" in "Move Attachment Data", attachments uploaded prior to July 27th are displayed in the attachment browser, but are no longer accessible when I try to access them via the system. Hovering over the thumbnail display an URL from the original location ("xenforo/data/attachments") , not the new ("data2") location. I've got backups of all of the attachments, how do you suggest we proceed?

"Update Attachment Data Storage Option" is used when your server configuration needs to be updated (change port, change server address etc.). You don't need it now.

I will need access to your AdminCP to check. Please send me the information via conversation. Sorry for the inconvenience.

 

[Main Company]

"Update Attachment Data Storage Option" is used when your server configuration needs to be updated (change port, change server address etc.). You don't need it now.

I will need access to your AdminCP to check. Please send me the information via conversation. Sorry for the inconvenience.

I'll PM you.

 

[eagle eyes]

What do I need to do in order to go from external data directory to default and then uninstall (i want to uninstall temporarily due to compatibility with another addon). Please tell me the steps. Including what to disable etc.

• Store file in External Data directory

• to

• Default

 

[kontrabass]

What do I need to do in order to go from external data directory to default and then uninstall (i want to uninstall temporarily due to compatibility with another addon). Please tell me the steps. Including what to disable etc.

• Store file in External Data directory

• to

• Default

Wait for confirmation if you want, but I think all you need to do is this:

1. set the option in your admincp -> attachments to default storage method
2. go to your rebuild caches page and "move attachments" (i'd leave it at the default 50 per cycle)
3. uninstall the addon.

 

[xfrocks]

What do I need to do in order to go from external data directory to default and then uninstall (i want to uninstall temporarily due to compatibility with another addon). Please tell me the steps. Including what to disable etc.

• Store file in External Data directory

• to

• Default

What @kontrabass said is correct. If you uninstall the add-on, you have to move the files back to default first. And you cannot just move to external data because by default, XenForo doesn't serve attachments from external data.

 

[eagle eyes]

What @kontrabass said is correct. If you uninstall the add-on, you have to move the files back to default first. And you cannot just move to external data because by default, XenForo doesn't serve attachments from external data.

So i followed what he posted but i still see some images in data/attachements folder. Is that normal?

 

[eagle eyes]

^

Wait for confirmation if you want, but I think all you need to do is this:

1. set the option in your admincp -> attachments to default storage method
2. go to your rebuild caches page and "move attachments" (i'd leave it at the default 50 per cycle)
3. uninstall the addon.

Any idea how to tell if the movement (external to default) went through good and fine? I have checked the attachment browser, all pics are loading fine but there are so many that of course i can't go through all.

 

[Alan Hollis]

Hi Xfrocks:

Could you please clarify a few things for me

1) Enforce Attachment Permission with S3

If using "just" S3 without cloud-front what does this mean? Experimenting I seem to get the redirected to urls like http://s3.amazonaws.com/2014/11/78525_e05f46df57e4b21b07bf1e12305abb89.png is that what you'd expect?

2) Enforce attachment when using Amazon CloudFront to serve attachments?

If I tick this box I assume it doesn't matter if I've got the enforce attachment permissions set or not set?

3) "Auto-expired URLs will be used to enforce attachment" how does this work in practice?

Experimenting I get urls like this:

http://tff-xenforo-data.s3.amazonaw...YHNyGZRMSQ__&Key-Pair-Id=APKAI3FUQAMYQSG3SXKQ

But I get the error "Query-string authentication requires the Signature, Expires and AWSAccessKeyId parameters" .

Does the user always get redirected to cloud front this way?

4) Set private access impossible to go back?

I assume this just sets the access level in S3 to private? Why is it then impossible to turn off cloudfront? If I used s3cmd and set the ACL to public and then turned off cloud front I assume it would be fine?

5) Was there a reason for a change in directory structure?

I've noticed that the files get uploaded as /YYYY/MMMM/DATA_ID.xxxxxx.ext was there a reason for not keeping the structure the same way as Xenforo stores the data? I can't think of one, but I guess there must be?

Apologies for the many questions, would be really grateful for any response

Cheers

Alan

 

[xfrocks]

@Alan Hollis

1) If you enforce permission, the S3 url won't be used in XenForo pages. The original /attachments url will be used and when user click it, the add-on check for permission before actually redirect user to S3.

2) If you use CloudFront with signed url no need to enforce permission. CloudFront with signed url performs permission check already.

3) Basically a unique signed url will be generated whenever a user requests a file and the url will expire after a few minutes (depends on your configuration). Where did you get the quoted url? The add-on should generate a valid url and you should be able to see the file just like usual (before it expires of course).

4) Yes, if you can use tools to enable public access for all files you should be fine. Most people won't know enough to do it so I have to put a strong note.

5) XenForo stores 1000 files per directory and partition them using data_id but this add-on sometimes does not store the id in the filename so date time was used to partition the files instead.

 

[xfrocks]

^



Any idea how to tell if the movement (external to default) went through good and fine? I have checked the attachment browser, all pics are loading fine but there are so many that of course i can't go through all.

Wait a few days for the cron to run completely and run this query to check if any data is left behind

SELECT * FROM `xf_attachment_data` WHERE bdattachmentstore_engine IS NOT NULL

 

[Alan Hollis]

Thanks Xfrocks

Can you please clarify what you mean by "sometimes" does not store the id?

Thank you,

Alan

 

[xfrocks]

Thanks Xfrocks

Can you please clarify what you mean by "sometimes" does not store the id?

Thank you,

Alan

You can configure the filename template to include or exclude the id. It is an option when you configure the add-on in AdminCP.