1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Add-on Amazon S3 file uploads

Discussion in 'Resource and Add-on Requests' started by Mike Tougeron, Mar 25, 2013.

  1. Mike Tougeron

    Mike Tougeron Well-Known Member

    Has anyone written an add-on for xenforo that writes the image/file uploader to Amazon S3? I tried looking around the resource manager and didn't see anything but I wanted to check-in and make sure I didn't just miss it.
    Marcus likes this.
  2. Chris D

    Chris D XenForo Developer Staff Member

    I wouldn't know where to begin, unfortunately Mike, and I don't think there's an existing add-on.

    But for the benefit of yourself or anyone else who might want to give it a try, Zend Framework includes some Amazon S3 classes that may help as a starting point.

    Check out: http://framework.zend.com/manual/1.12/en/zend.service.amazon.s3.html

    The Zend classes that relate and are included in XenForo are:

    SneakyDave and Marcus like this.
  3. Mike Tougeron

    Mike Tougeron Well-Known Member

    Yeah, if there isn't one already written I'll have to write one myself. I just didn't want to re-invent it if someone already did it.
  4. Dinh Thanh

    Dinh Thanh Well-Known Member

    Will you release it?
  5. TheBigK

    TheBigK Well-Known Member

    Yeh, I'd want that too :)
    Dinh Thanh likes this.
  6. arms

    arms Active Member

  7. Mike Tougeron

    Mike Tougeron Well-Known Member

    Yeah, I will.
    Andy.N, Dinh Thanh and luutruong like this.
  8. Mike Tougeron

    Mike Tougeron Well-Known Member

    So hopefully my changes will get accepted into the core, but to do this it doesn't actually need a full fledged add-on; yippee!
    - Follow the changes mentioned in http://xenforo.com/community/threads/enable-streams-for-internal-external-data-paths.47011/
    - Change your config to point to your bucket. e.g.,
        'internalDataPath' => 's3://your-bucket-name/internal_data',
    'externalDataPath' => 's3://your-bucket-name/data',
    'externalDataUrl' => 'https://s3.amazonaws.com/your-bucket-name/data',
    - Add the following to your index.php and admin.php (I'm not sure but perhaps this could be done by extending the dependencies...)
    require_once 'Zend/Service/Amazon/S3.php';
    $s3 = new Zend_Service_Amazon_S3('your_aws_key''your_aws_secret');
    - Write a quick script to pre-create the sub-directories in S3. This is necessary so that XenForo doesn't try to create the directories itself in S3 -- the Zend_Service_Amazon_S3 doesn't support mkdir().
    require_once 'Zend/Service/Amazon/S3.php';
    $s3 = new Zend_Service_Amazon_S3('your_aws_key''your_aws_secret');
    file_put_contents('s3://your-bucket-name/internal_data/.htaccess',"Order deny,allow\nDeny from all");
    for (
    $i 0$i 1000$i++) {
    file_put_contents('s3://your-bucket-name/data/attachments/' $i '/index.html','');
    file_put_contents('s3://your-bucket-name/internal_data/attachments/' $i '/index.html','');
    That should be it! I'm still testing the edge cases but it feels pretty solid so far. Feel free to hit me up if you have any questions.
    Marcus and Andy.N like this.
  9. Dinh Thanh

    Dinh Thanh Well-Known Member

    Very Cool script.
    How about the current files? Re-upload first?
  10. Mike Tougeron

    Mike Tougeron Well-Known Member

    It'd be something like
    for (
    $i 0$i 1000$i++) {
    $files scandir($baseDir '/' $i '/');
        foreach (
    $files as $file) {
            if ( 
    is_file($baseDir '/' $i '/' $file) ) {
    copy($baseDir '/' $i '/' $file's3://your-bucket-name/data/attachments/' $i '/' $file);
    thedude likes this.
  11. Dinh Thanh

    Dinh Thanh Well-Known Member

    This will be very easy run into time out or die if we have network issue.
    We may download then upload manully
  12. Mike Tougeron

    Mike Tougeron Well-Known Member

    Yeah, that'd work too. I just threw that idea out there for you off the top of my head. I haven't written our actual script yet; that's something I'm going to try to do early next week.
  13. Dinh Thanh

    Dinh Thanh Well-Known Member

    Great, thank you
  14. Hlchia

    Hlchia Active Member

    I received following error:
    Fatal error: require_once() [function.require]: Failed opening required 'Zend/Service/Amazon/Abstract.php' (include_path='.:/usr/lib/php:/usr/local/lib/php') in /home/xxx/public_html/test/library/Zend/Service/Amazon/S3.php on line 26
  15. xfrocks

    xfrocks Well-Known Member

  16. Bill.D

    Bill.D Active Member

    Would it be easier to just add this to the config.php or will it not work from there?

  17. Mike Tougeron

    Mike Tougeron Well-Known Member

    It actually probably belongs inside the dependencies hooks.
  18. Bill.D

    Bill.D Active Member

    lol.. You just went over my head.. lol.. Is that in a file? Sorry.. I am only moderately experienced with this system.
  19. Mike Tougeron

    Mike Tougeron Well-Known Member

    One of the code event listeners (admin.php?code-event-listeners) you can use is init_dependencies. Inside the class/method you have that listener execute is where you would register the S3 stream wrapper.
  20. Mike Tougeron

    Mike Tougeron Well-Known Member

    BTW, FYI, I test this using the AWS php-sdk-2 and it is *way* slower than using ZF1's stream wrapper. Has something to do with the way Guzzle does the HEAD check for if the file_exists.
    Marcus likes this.

Share This Page