[bd] Attachment Store [Deleted]

[sbj]

Thank you for your great help. Ah now I understand what CDN is.

1) I'm a bit lost here. So the purpose is to divide the backup? All the big data which is basically attachments will be moved away. And I only need to backup my database and the rest of the folders. But how is this different than normal backup via ftp to other server? Why should I divide it?

3)Don't you think the service of Amazon is a bit expensive compared to other services? I know of storage service that charge 10$ for 250 GB and 1 TB bandwidth.

4) If I used the data storage add-on, which is basically the same add-on as this, I could use it as a backup tool, right? I mean I could move my backups to this /data folder and done. But I assume all of us have TBs space in our personal computers. Wouldn't it be good to "let it download" to our own computers besides moving it to Amazon S3 or whatever? Much cheaper and extra backup?

 

[Jim Boy]

1) I'm a bit lost here. So the purpose is to divide the backup? All the big data which is basically attachments will be moved away. And I only need to backup my database and the rest of the folders. But how is this different than normal backup via ftp to other server? Why should I divide it?

This is not an add-on for doing backups, you could use it as a backup, but there are better ways of doing backups. This is an add-on for storing and serving user generated static assets in a way that is resilient allowing for multiple servers and remving some overhead from your webserver.

3)Don't you think the service of Amazon is a bit expensive compared to other services? I know of storage service that charge 10$ for 250 GB and 1 TB bandwidth.

Hmmm, I have always thought that $5 is less than $10. I agree that AWS isn't the cheapest on per GB level, but you only pay for what you use, that $10 you talk of is paying for capacity that you will probably never use.

4) If I used the data storage add-on, which is basically the same add-on as this, I could use it as a backup tool, right? I mean I could move my backups to this /data folder and done. But I assume all of us have TBs space in our personal computers. Wouldn't it be good to "let it download" to our own computers besides moving it to Amazon S3 or whatever? Much cheaper and extra backup?

Neither this addon nor the storage add-on is designed for backups, they are designed for working websites. If you are looking for backup solutions, look elsewhere.

 

[sbj]

I already have a backup solution with that said I am not looking for a backup solution. I try to understand how is this different than a normal backup?
It stores the attachments somewhere else. So my life site is on server A, and the attachments on server B. If something happens to server A, I need to restore server A. If something happens to server B, I need to restore server B.
BUT what is the purpose of splitting/dividing the live site in 2 seperate servers? I am a noob and don't understand the reason behind it.

 

[Jim Boy]

I already have a backup solution with that said I am not looking for a backup solution. I try to understand how is this different than a normal backup?
It stores the attachments somewhere else. So my life site is on server A, and the attachments on server B. If something happens to server A, I need to restore server A. If something happens to server B, I need to restore server B.
BUT what is the purpose of splitting/dividing the live site in 2 seperate servers? I am a noob and don't understand the reason behind it.

You split it up for a few reasons - the first is that if your site becomes so busy you want to add another server, or you use something like autoscaling, then you don't have to deal with moving large amounts of data around and risking inconsistencies. The second is that code is generally replacable directly from source - if you lose the box, you can download the code again from xenforo or from your own repo and get it back to how it was - without the need to go to backups. You can't do that with user generated content. The third reason is that it takes some pressure off your webserver, if your webserver is locked delivery large files, that means you need a bigger webserver.

S3 isn't something that breaks, AWS have an SLA of 11 9's. That means if you have a million objects on S3, you'll lose 1 item every 10,000 years. It's super-resilient. If a data-center hosting S3 gets blown up, you dont lose a thing as the data is replicated to other adat-centres. It's why dropbox uses it, its why Netflix use it.

This isn't a discussion about XF as such, its about how every professional web application should operate.

 

[AndreaMarucci]

@xfrocks I've a little question for you. Does your addon have XFMG support? Asking this since I'd like to implement Media Gallery but I'd like to host all the images in an external server (not a CDN) and since I'm one of your user I'd like to use this one also for the images...

 

[Blackbeard]

Yes it does. I've been using it since it was released and works with RM / XFMG / Showcase addons

@xfrocks I've a little question for you. Does your addon have XFMG support? Asking this since I'd like to implement Media Gallery but I'd like to host all the images in an external server (not a CDN) and since I'm one of your user I'd like to use this one also for the images...

Hopefully that answers your question.

 

[AndreaMarucci]

Fantastic

 

[AndreaMarucci]

Just a question. Since now I'm using AS to store all the attachments in an external server, I've setup a specific directory for these attach but if I install XFMG I'd like to store all the images in a different directory.

It's possible to do that?

 

[xfrocks]

Alright, deployed successfully after tweaking. Ran into a few problems:

The add-on is written such that images uploaded to S3 do not have Content-Disposition headers. However, when migrating data, each image takes on the Content-Disposition header of the last non-image in the batch. Images uploaded before any non-images will correctly lack a Content-Disposition header. For example, if a batch uploads alpha.png, beta.txt, and gamma.png, alpha.png and beta.txt will download as expected, but gamma.png will download with the filename beta.txt. This is actually a bug in Zend, and there is a comment in the Zend source code acknowledging the bug (Zend/Service/Amazon/S3.php, lines 609-610). However, the acknowledgment is actually mistaken: the bug is not in Zend_Http_Client, but at the location of the comment deferring blame. Zend_Http_Client has a resetParameters method that must be called between requests. In order for it to reset headers, true must be passed as the first parameter; however, Zend_Http_Client omits this. A simple fix would be to extend Zend_Service_Amazon_S3, override _makeRequest, and call self::getHttpClient()->resetParameters(true) prior to calling parent::_makeRequest(...). Example:

class bdAttachmentStore_Zend_Service_Amazon_S3 extends Zend_Service_Amazon_S3
{
    public function _makeRequest($method, $path = '', $params = null, $headers = array(), $data = null)
    {
        if (isset(self::$_httpClient)) {  // Don't bother if we're creating a new client
            self::getHttpClient()->resetParameters(true);
        }

        return parent::_makeRequest($method, $path, $params, $headers, $data);
    }
}

Note that this is also a potential issue with bdDataStorage, since bdAttachmentStore and bdDataStorage combined can result in multiple S3 uploads using the same Zend_Http_Client instance. The same fix can be applied there as well.


The migration tool has an SQL statement in bdAttachmentStore_CacheRebuilder_AttachmentData::rebuild that would benefit from an additional WHERE condition. Currently, the SELECT includes rows that are already on the current engine. I've added the following as a quick hack:

AND (
    attachment_data.bdattachmentstore_engine NOT LIKE ?
    ' . (empty($defaultEngine) ? 'AND attachment_data.bdattachmentstore_engine IS NOT NULL' : 'OR attachment_data.bdattachmentstore_engine IS NULL') . '
)

...

array(
    $position,
    empty($defaultEngine) ? '' : $defaultEngine,
    $options['batch']
)

$defaultEngine needs to be declared a bit earlier for this to work. This could probably be improved by excluding attachments with an attach_count of 0 (attachments that have already been migrated, but have yet to be cleaned up).

Also: both bdDataStorage and bdAttachmentStore appear to be saving attachment thumbnails. Is there an easy way to disable this for bdDataStorage? I haven't looked at that code as much.

Nice fix regarding the clear param. I have added fixes for both add-ons. Had reports similar to your problems before but couldn't reproduce. Thank you very much. About the improvement in SQL, it does make sense but it was intended to check with PHP later so maybe I won't incorporate your fix.

 

[xfrocks]

When we use this tool, attachments at /internal_data/ are deleted or I need to delete it manually after rebuilding? ("keep local file" not selected in options)

You don't need to delete the files manually. They are cleaned up by a cron hourly.

I have some beginner questions here.

1. Does it make sense to use this add-on for a forum from the start (so no members, no attachments at this moment)? Or should someone wait until one have GBs of attachments?
2. What is this CDN your are talking about? Some enable it, some dont. For what do I need it?
3. Let's say a forum with 30 gb attachments and 5000 users. The Amazon S3 would charge how much estimated?
4. Can we use this to save the attachments on our own personal computer which is not 24/7 online?

@Jim Boy has answered all of your questions nicely

 

[xfrocks]

Just a question. Since now I'm using AS to store all the attachments in an external server, I've setup a specific directory for these attach but if I install XFMG I'd like to store all the images in a different directory.

It's possible to do that?

No, you can't specify directories by add-on basis. Sorry.

I already have a backup solution with that said I am not looking for a backup solution. I try to understand how is this different than a normal backup?
It stores the attachments somewhere else. So my life site is on server A, and the attachments on server B. If something happens to server A, I need to restore server A. If something happens to server B, I need to restore server B.
BUT what is the purpose of splitting/dividing the live site in 2 seperate servers? I am a noob and don't understand the reason behind it.

In an idea world, I would have my web assets (php files, images, stylesheets etc.) in a private repo for easy version tracking and deploy, user generated data (database, attachments) should have their own mean of backup. That way I can scale horizontally whenever I like. I hope that clears your confusion.

 

[Weppa333]

I've got a fresh install of XF1.4.5 with XMG1.0.3, and attach store ssems to do absolutely nothing on XMG (it does work for the forum)
Is this normal ?
Is there a setting I missed somewhere ? Maybe a problem with 1.0.3 ?

 

[xfrocks]

I've got a fresh install of XF1.4.5 with XMG1.0.3, and attach store ssems to do absolutely nothing on XMG (it does work for the forum)
Is this normal ?
Is there a setting I missed somewhere ? Maybe a problem with 1.0.3 ?

I don't keep track of the version of XMG but this is the first report so far. I know many people running both add-ons together though. Can you share link to your site?

 

[Fred.]

I just noticed the internal_data/temp folder is 350+ MB

All files in there have the xf prefix.

The directory is taking up more disk space every day.
And it seems like they never get deleted. (Folder is chmod 777)

Does someone have an idea how I can fix this?

Thanks!

 

[xfrocks]

I just noticed the internal_data/temp folder is 350+ MB

All files in there have the xf prefix.

The directory is taking up more disk space every day.
And it seems like they never get deleted. (Folder is chmod 777)

Does someone have an idea how I can fix this?

Thanks!

For the time being, delete them manually. And 350+ MB is not very much. How heavy are your attachments?

 

[Fred.]

For the time being, delete them manually. And 350+ MB is not very much. How heavy are your attachments?

Ok, attachments are only 154 MB it's a small forum. 25k posts.

 

[3rd AnGle]

Hi @xfrocks

Getting a lot of server errors. I am assuming it could be this addon. Does this error related to your addon?

• unlink(/home/hippy/public_html/abc.com/internal_data/attachments/8/8001-b2daeee44d3bb0b7bfa057dbcabbddd8.data): Permission denied Yesterday at 3:11 PM - library/XenForo/DataWriter/AttachmentData.php:184

This error happens every hour... also the thumbnails on my XFMG are never working correctly for the old images (ones before i started using attachment store) inspite of repeatedly rebuilding them... Could you help?

 

[xfrocks]

Ok, attachments are only 154 MB it's a small forum. 25k posts.

Then please delete manually and keep an eye on them then.

Hi @xfrocks

Getting a lot of server errors. I am assuming it could be this addon. Does this error related to your addon?

• unlink(/home/hippy/public_html/abc.com/internal_data/attachments/8/8001-b2daeee44d3bb0b7bfa057dbcabbddd8.data): Permission denied Yesterday at 3:11 PM - library/XenForo/DataWriter/AttachmentData.php:184

This error happens every hour... also the thumbnails on my XFMG are never working correctly for the old images (ones before i started using attachment store) inspite of repeatedly rebuilding them... Could you help?

Can you check the file permission of the file now?

 

[MattW]

Hi @xfrocks

Getting a lot of server errors. I am assuming it could be this addon. Does this error related to your addon?

• unlink(/home/hippy/public_html/abc.com/internal_data/attachments/8/8001-b2daeee44d3bb0b7bfa057dbcabbddd8.data): Permission denied Yesterday at 3:11 PM - library/XenForo/DataWriter/AttachmentData.php:184

This error happens every hour... also the thumbnails on my XFMG are never working correctly for the old images (ones before i started using attachment store) inspite of repeatedly rebuilding them... Could you help?

That is due to the permissions specifically on that folder. It's not owned by the correct user / group (one of the old folders prior to moving to my server). I've corrected that now for you.

 

[3rd AnGle]

That is due to the permissions specifically on that folder. It's not owned by the correct user / group (one of the old folders prior to moving to my server). I've corrected that now for you.

thanks much... i wanted to approach you but for some reason i was shy

Can you check the file permission of the file now?

What about the thumbnails on XFMG?

also the thumbnails on my XFMG are never working correctly for the old images (ones before i started using attachment store) inspite of repeatedly rebuilding them... Could you help?

is it part of the same problem.. does the permission change fixed this issue?