1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

How to decrease your attachments folder by 50-80%

Discussion in 'Forum Management' started by Moshe1010, Jan 19, 2015.

  1. Moshe1010

    Moshe1010 Well-Known Member

    I don't have super huge attachments folder, but it bothered me since every GB counts when you're on a SSD VPS. My initial attachments folder (not including avatars) was 7GB and I was able to decrease it to 4GB (about 40%) with a simple task. I initially wanted to build an add-on that does lossy compression by default, but I concluded it would be time demanding for a one time thing (if people would upload 1000+ images/day on my forum then I would do it). I could go even more extreme and compress it even more, but I was too lazy - you can get up to 80%, and in some rare cases even more.

    So how does it work?

    First let me show you the huge difference in size vs. no difference in quality. I'm heavy on HiDPI (e.g. Retina display), so keeping the quality and the original resolution of the pictures was the only option.

    File 1: 14.2 MB
    http://imagizer.imageshack.us/a/img908/7843/ABZQl9.jpg


    After compression: 3.5MB (80% less) - can you notice any difference? same exact crazy resolution.
    http://imagizer.imageshack.us/a/img538/4471/JM0l7E.jpg


    File 2: 7.4MB
    http://imageshack.com/a/img537/1594/MXF4XR.jpg



    After compression: 4.9MB (35% less)
    http://imageshack.com/a/img661/7687/81wjia.jpg



    What's the secret?
    Basically:
    http://www.jpegmini.com/

    They have a server, cloud and desktop solutions; I used the desktop version and uploaded everything manually to my FTP. For $20, you can use it as long as you want. If you want it to be automatically, they offer a cloud service ($40/month based on Amazon's services or a server based solution for $200/month). I've found these solutions not economical in the long run versus how much 1GB SSD costs.

    Even more compression can be achieved by:
    https://kraken.io/web-interface

    Choose LOSSLESS compression there, not the LOSSY option (LOSSY would screw up quality with kraken, and you already did LOSSY compression with JPEGmini [which does that the right way]). Their free service is up to 1MB images (no limit - drag and drop. PRO service (API included) is up to 16MB/image). I've found their interface and compression better than Yahoo! Smush.it (http://www.smushit.com/ysmush.it/).
    You can use Kraken for Wordpress as well (they have a plugin, but it require subscription which is $5/month the lowest).

    Just to note, the above method works only on JPEG files. for PNG/GIF compression I would suggest installing (if you have VPS - install via SSH) PHP extensions like optipng/and gifsicle. You can also install jpegtran to keep compression jpeg files without compromising quality.
     
    Last edited: Jan 19, 2015
  2. Jaxel

    Jaxel Well-Known Member

    I'm looking for an addon that will compress image uploads. I don't want to force specific dimensions on images, but I would still like proper compression. Most people on my forums take screenshots or uncompressed photos from their camera, and the file sizes are insane for the dimensions.

    For instance, this 1280x720 image is 1.8mb... It doesn't need to be.
    http://8wayrun.com/attachments/euro-1p-yoshi-3-png.42873/
     
  3. gfc

    gfc Active Member

    Dakis and thomas1 like this.
  4. Jim Boy

    Jim Boy Well-Known Member

    If you were that concerned about data storage usage, then you should get bdattachment and store attachments elsewhere, eg S3 and pay 3 cents per month per GB.
     
  5. gfc

    gfc Active Member

    It is not only storage. It is also bandwith for the users. 60% of my users / visits are using smartphones for browsing my forums. Huge pics makes it a pain because the loading is slow.
     
    StarArmy, D.O.A. and Xon like this.
  6. markku

    markku Well-Known Member

  7. Xon

    Xon Well-Known Member

    The metadata is stored in the database for the original filename, and detected mime-type.

    Any script would need to mark that that an attachment has been processed anyway.
     
    D.O.A. and Jim Boy like this.
  8. Jim Boy

    Jim Boy Well-Known Member

    Indeed, IMO the ideal scenario is to use a modified version of bdattachment that sends the file to Kraken.io with instructions to post to S3, although you could simplify by a curl call at the end of the bdattachment script that processes the non-optimised version of the image and replaces it with the optimised version, along with cache control headers that are currently lacking
     
    Xon likes this.
  9. Vitor Dos Reis

    Vitor Dos Reis Active Member

    How do you do this? You change .data files to ,jpg in your desktop and after compression change it again to original file extension?
     
  10. Moshe1010

    Moshe1010 Well-Known Member

    Vitor Dos Reis likes this.
  11. gfc

    gfc Active Member

    The first point is harder then the second, but should be solvable over tge mime type and the file command.

    The second point: i wouldnt mark a file, i would select the files based on modification date on a file level
     
  12. Xon

    Xon Well-Known Member

    Please don't use file dates as a stand in for the metadata describing if you have processed a file or not.

    This is a bad idea, as you now need to be careful that you don't run the script more than once, which leads into supporting multiple web nodes.

    Fundamentally, reusing some data which means something completely different for the rest of the system as your 'have I done stuff' flag is a bad idea.
     
  13. gfc

    gfc Active Member

    The other essy Options would be
    • Extended File Attributes (never used that before)
    • Checksums saved to a flat file (might be slow on checking)
    If i'm writing to a DB that would be an overkill.
     
  14. Xon

    Xon Well-Known Member

    My recommendation is to copy the XF deferred script for setting up a XF run-time instance, and then use XenForo_Application::getDb()

    This allows you to control settings of the script from XF itself, without requiring code edits.

    You can then log specific error messages to the XF error log via XenForo_Error::debug("my error message").
     
  15. gfc

    gfc Active Member

    yeah that I would do if I would sell the addon. But for my need i want to stay at bash only. But you're free to develop such an addon.
     
  16. Dakis

    Dakis Well-Known Member

    This jpegmini thing looks fantastic, I tried it on a few shots and the results are perfectly acceptable in terms of quality, even more if you only use it at a forum and not your professional shots.

    I'll try to compress my attachments folder and see how that goes :)

    Thanks @Moshe1010 !
     
  17. dethfire

    dethfire Well-Known Member

    XF needs to build in a compression function for uploads
     
    BoostN likes this.
  18. Mouth

    Mouth Well-Known Member

    If I make/map an SFTP connection to my server from my desktop, could I then use the desktop to mini the files on the server?
     
  19. dethfire

    dethfire Well-Known Member

    You downloaded 8GB of images off your server and then reuploaded them? How long did that take!? Download the software and drag and drop into program? You do this, what, like once a month?
     
  20. RoldanLT

    RoldanLT Well-Known Member

    Maybe he has 1GB connection at home :)
     
    dethfire likes this.

Share This Page