XF 1.2 Resource Manager CDN (Large File Hosting)

intradox

intradox

Well-known member
Any ideas for how i could distribute files in excess over several hundred mb? Somewhere in the range of 50mb-3gb?

It would be legal material (games) so I am wondering if anyone has found a reliable way to use Amazon's S3 + Cloudfront to deliver resource manager files in a fast manager?

I know that the cdn, High Winds, is used by Steam and Humble Bundle to deliver digital games but I am not really sure how to go about implementing a pull and push with xenforo's RM.

*Would be willing to pay for an addon although if it's a simple change in the code perhaps that would be best.
 
Sort by date Sort by votes
intradox said:
Crap now I feel stupid. Are resource manager files considered attachments? Thought they were stored differently.

Thanks though, that looks like it fits the bill.
Click to expand...
Yeah, they are just normal attachments. I've just tested it with a PDF on my site, and it's pulled via the CDN I have set up for attachments using that add-on (y)
 
Upvote 0 Downvote
MattW said:
Yeah, they are just normal attachments. I've just tested it with a PDF on my site, and it's pulled via the CDN I have set up for attachments using that add-on (y)
Click to expand...
Slightly off-topic from xenforo but should I even consider Amazon for distributing large files? Only ever used them for assets (js/imgs).
 
Upvote 0 Downvote
Sorry for double posting but I was wondering if you or anyone might be able to enlighten me regarding the uploading of large files.

I'll most likely use xfrocks's attachment store addon to handle the download of large resources (500mb-1gb?)

But I am now wondering if Xenforo can even handle the uploading of files that large? How would I go about making sure it is a smooth process for my users and having it all happen as fast as possible? The site won't be handling many large uploads at a time. Maybe 2-10 max at a time but how would I handle this in PHP & Xenforo?
 
Upvote 0 Downvote
intradox said:
Sorry for double posting but I was wondering if you or anyone might be able to enlighten me regarding the uploading of large files.

I'll most likely use xfrocks's attachment store addon to handle the download of large resources (500mb-1gb?)

But I am now wondering if Xenforo can even handle the uploading of files that large? How would I go about making sure it is a smooth process for my users and having it all happen as fast as possible? The site won't be handling many large uploads at a time. Maybe 2-10 max at a time but how would I handle this in PHP & Xenforo?
Click to expand...
It's going to depend on how xfrocks add-on sends the files to S3
 
Upvote 0 Downvote
MattW said:
It's going to depend on how xfrocks add-on sends the files to S3
Click to expand...
I was planning on using s3+cloudfront with local mirroring (Keep Local File) incase I decide to use alternative storage or stop using aws.

I noticed you reviewed and I have not yet purchased it so I'm not really sure how he handles the upload process. If keeping a local file would not work well with large files then I'm fine with only using s3+cloudfront but I would ideally like a local copy.
 
Upvote 0 Downvote
@intradox if you plan to let user upload 500MB-1GB resources, it may not work. The add-on will take the upload from user (1GB of data to your server) and process it as needed. The process will subject to server maximum upload limitation (php.ini etc.) and XenForo size limitation (options in AdminCP). I assume you won't let user upload a 1GB JPEG because generating thumbnail for that will definitely kill the server... After that, the file will be uploaded to Amazon server (another 1GB outbound). Depending on user bandwidth and server resources (memory, bandwidth), uploading one file may take a long long time and a slight interruption will require another upload.

For your use case, probably use a client application with resume support to upload to Amazon and serve it from there?

If you don't have lots of file, you can upload fake file (a few KB) as the resource file. Then use the client application to upload the real file overwriting the fake one. That way your file will be protected with XenForo permission as usual.
 
Upvote 0 Downvote
xfrocks said:
@intradox if you plan to let user upload 500MB-1GB resources, it may not work. The add-on will take the upload from user (1GB of data to your server) and process it as needed. The process will subject to server maximum upload limitation (php.ini etc.) and XenForo size limitation (options in AdminCP). I assume you won't let user upload a 1GB JPEG because generating thumbnail for that will definitely kill the server... After that, the file will be uploaded to Amazon server (another 1GB outbound). Depending on user bandwidth and server resources (memory, bandwidth), uploading one file may take a long long time and a slight interruption will require another upload.

For your use case, probably use a client application with resume support to upload to Amazon and serve it from there?

If you don't have lots of file, you can upload fake file (a few KB) as the resource file. Then use the client application to upload the real file overwriting the fake one. That way your file will be protected with XenForo permission as usual.
Click to expand...

Would really like to avoid client application and have it entirely web-based as then it starts becoming quite confusing for the user. I was thinking I could look into Dropbox where user uploads it to their dropbox account and then I use something like Drop-In (https://www.dropbox.com/developers/dropins) or a similar service. I would then move it to s3/local. Will need to look into this it seems. I can definitely start with a cap of several hundred megs but i would need to support up to at least 500mb as otherwise it seems my marketplace idea would not work well.

Thanks again for your input, much appreciated!
 
Upvote 0 Downvote
Still trying to figure out the best way to allow files larger than ~10-20mb and up to 1gb.

Upload Large Files
Solution 1 - Chunking
Any idea if it would be possible to implement "chunked" uploading for large files with xenforo's resource center? http://en.wikipedia.org/wiki/Chunked_transfer_encoding

I'm using PHP5.3 with Nginx and Mysql so maybe custom xenforo upload module with having nginx handle the upload then relay it back to php (something like http://blog.martinfjordvald.com/2010/08/file-uploading-with-php-and-nginx/)

Solution 2 - 3rd Party Service (intermediary)
OR https://transloadit.com/ use a service like trans and have the file uploaded straight to s3/sftp and relay it back to resources? https://transloadit.com/demos/s3-store/store-uploaded-files (docs: https://transloadit.com/docs)

Downloading Large Files
Solution - nginx x-accel addon
I use nginx so I would need to implement x-accel/X-Accel-Redirect which is known as x-sendfile in apache.

Would @xfrocks or anyone be able to implement some paid solution to enable large file uploads AND downloads?

@MattW Any update on how it handles large files?
 
Last edited:
Upvote 0 Downvote
You must log in or register to reply here.
Back
Top Bottom