1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

XF 1.2 Resource Manager CDN (Large File Hosting)

Discussion in 'XenForo Questions and Support' started by intradox, Aug 17, 2013.

  1. intradox

    intradox Well-Known Member

    Any ideas for how i could distribute files in excess over several hundred mb? Somewhere in the range of 50mb-3gb?

    It would be legal material (games) so I am wondering if anyone has found a reliable way to use Amazon's S3 + Cloudfront to deliver resource manager files in a fast manager?

    I know that the cdn, High Winds, is used by Steam and Humble Bundle to deliver digital games but I am not really sure how to go about implementing a pull and push with xenforo's RM.

    *Would be willing to pay for an addon although if it's a simple change in the code perhaps that would be best.
  2. MattW

    MattW Well-Known Member

  3. intradox

    intradox Well-Known Member

  4. MattW

    MattW Well-Known Member

    Yeah, they are just normal attachments. I've just tested it with a PDF on my site, and it's pulled via the CDN I have set up for attachments using that add-on (y)
    intradox likes this.
  5. intradox

    intradox Well-Known Member

    Slightly off-topic from xenforo but should I even consider Amazon for distributing large files? Only ever used them for assets (js/imgs).
  6. MattW

    MattW Well-Known Member

    I've had no issues with using S3 before for large files. Should be pretty quick with the S3 bucket sat behind their Cloudfront CDN.
    intradox likes this.
  7. intradox

    intradox Well-Known Member

    Alright, awesome! Thanks again.
  8. intradox

    intradox Well-Known Member

    Sorry for double posting but I was wondering if you or anyone might be able to enlighten me regarding the uploading of large files.

    I'll most likely use xfrocks's attachment store addon to handle the download of large resources (500mb-1gb?)

    But I am now wondering if Xenforo can even handle the uploading of files that large? How would I go about making sure it is a smooth process for my users and having it all happen as fast as possible? The site won't be handling many large uploads at a time. Maybe 2-10 max at a time but how would I handle this in PHP & Xenforo?
  9. MattW

    MattW Well-Known Member

    It's going to depend on how xfrocks add-on sends the files to S3
  10. intradox

    intradox Well-Known Member

    I was planning on using s3+cloudfront with local mirroring (Keep Local File) incase I decide to use alternative storage or stop using aws.

    I noticed you reviewed and I have not yet purchased it so I'm not really sure how he handles the upload process. If keeping a local file would not work well with large files then I'm fine with only using s3+cloudfront but I would ideally like a local copy.
  11. MattW

    MattW Well-Known Member

    I can install a copy on my test server, and see how it handles large files uploading to S3 later on this evening.
    intradox likes this.
  12. xfrocks

    xfrocks Well-Known Member

    @intradox if you plan to let user upload 500MB-1GB resources, it may not work. The add-on will take the upload from user (1GB of data to your server) and process it as needed. The process will subject to server maximum upload limitation (php.ini etc.) and XenForo size limitation (options in AdminCP). I assume you won't let user upload a 1GB JPEG because generating thumbnail for that will definitely kill the server... After that, the file will be uploaded to Amazon server (another 1GB outbound). Depending on user bandwidth and server resources (memory, bandwidth), uploading one file may take a long long time and a slight interruption will require another upload.

    For your use case, probably use a client application with resume support to upload to Amazon and serve it from there?

    If you don't have lots of file, you can upload fake file (a few KB) as the resource file. Then use the client application to upload the real file overwriting the fake one. That way your file will be protected with XenForo permission as usual.
    intradox likes this.
  13. intradox

    intradox Well-Known Member

    Would really like to avoid client application and have it entirely web-based as then it starts becoming quite confusing for the user. I was thinking I could look into Dropbox where user uploads it to their dropbox account and then I use something like Drop-In (https://www.dropbox.com/developers/dropins) or a similar service. I would then move it to s3/local. Will need to look into this it seems. I can definitely start with a cap of several hundred megs but i would need to support up to at least 500mb as otherwise it seems my marketplace idea would not work well.

    Thanks again for your input, much appreciated!
  14. intradox

    intradox Well-Known Member

    Still trying to figure out the best way to allow files larger than ~10-20mb and up to 1gb.

    Upload Large Files
    Solution 1 - Chunking

    Any idea if it would be possible to implement "chunked" uploading for large files with xenforo's resource center? http://en.wikipedia.org/wiki/Chunked_transfer_encoding

    I'm using PHP5.3 with Nginx and Mysql so maybe custom xenforo upload module with having nginx handle the upload then relay it back to php (something like http://blog.martinfjordvald.com/2010/08/file-uploading-with-php-and-nginx/)

    Solution 2 - 3rd Party Service (intermediary)
    OR https://transloadit.com/ use a service like trans and have the file uploaded straight to s3/sftp and relay it back to resources? https://transloadit.com/demos/s3-store/store-uploaded-files (docs: https://transloadit.com/docs)

    Downloading Large Files
    Solution - nginx x-accel addon

    I use nginx so I would need to implement x-accel/X-Accel-Redirect which is known as x-sendfile in apache.

    Would @xfrocks or anyone be able to implement some paid solution to enable large file uploads AND downloads?

    @MattW Any update on how it handles large files?
    Last edited: Sep 6, 2013

Share This Page