How to decrease your attachments folder by 50-80%

Moshe1010

Moshe1010

Well-known member
I don't have super huge attachments folder, but it bothered me since every GB counts when you're on a SSD VPS. My initial attachments folder (not including avatars) was 7GB and I was able to decrease it to 4GB (about 40%) with a simple task. I initially wanted to build an add-on that does lossy compression by default, but I concluded it would be time demanding for a one time thing (if people would upload 1000+ images/day on my forum then I would do it). I could go even more extreme and compress it even more, but I was too lazy - you can get up to 80%, and in some rare cases even more.

So how does it work?

First let me show you the huge difference in size vs. no difference in quality. I'm heavy on HiDPI (e.g. Retina display), so keeping the quality and the original resolution of the pictures was the only option.

File 1: 14.2 MB
http://imagizer.imageshack.us/a/img908/7843/ABZQl9.jpg


After compression: 3.5MB (80% less) - can you notice any difference? same exact crazy resolution.
http://imagizer.imageshack.us/a/img538/4471/JM0l7E.jpg


File 2: 7.4MB
http://imageshack.com/a/img537/1594/MXF4XR.jpg



After compression: 4.9MB (35% less)
http://imageshack.com/a/img661/7687/81wjia.jpg



What's the secret?
Basically:
http://www.jpegmini.com/

They have a server, cloud and desktop solutions; I used the desktop version and uploaded everything manually to my FTP. For $20, you can use it as long as you want. If you want it to be automatically, they offer a cloud service ($40/month based on Amazon's services or a server based solution for $200/month). I've found these solutions not economical in the long run versus how much 1GB SSD costs.

Even more compression can be achieved by:
https://kraken.io/web-interface

Choose LOSSLESS compression there, not the LOSSY option (LOSSY would screw up quality with kraken, and you already did LOSSY compression with JPEGmini [which does that the right way]). Their free service is up to 1MB images (no limit - drag and drop. PRO service (API included) is up to 16MB/image). I've found their interface and compression better than Yahoo! Smush.it (http://www.smushit.com/ysmush.it/).
You can use Kraken for Wordpress as well (they have a plugin, but it require subscription which is $5/month the lowest).

Just to note, the above method works only on JPEG files. for PNG/GIF compression I would suggest installing (if you have VPS - install via SSH) PHP extensions like optipng/and gifsicle. You can also install jpegtran to keep compression jpeg files without compromising quality.
 
Last edited:
I'm looking for an addon that will compress image uploads. I don't want to force specific dimensions on images, but I would still like proper compression. Most people on my forums take screenshots or uncompressed photos from their camera, and the file sizes are insane for the dimensions.

For instance, this 1280x720 image is 1.8mb... It doesn't need to be.
http://8wayrun.com/attachments/euro-1p-yoshi-3-png.42873/
 
If you were that concerned about data storage usage, then you should get bdattachment and store attachments elsewhere, eg S3 and pay 3 cents per month per GB.
 
It is not only storage. It is also bandwith for the users. 60% of my users / visits are using smartphones for browsing my forums. Huge pics makes it a pain because the loading is slow.
 
Xon said:
The metadata is stored in the database for the original filename, and detected mime-type.

Any script would need to mark that that an attachment has been processed anyway.
Click to expand...
Indeed, IMO the ideal scenario is to use a modified version of bdattachment that sends the file to Kraken.io with instructions to post to S3, although you could simplify by a curl call at the end of the bdattachment script that processes the non-optimised version of the image and replaces it with the optimised version, along with cache control headers that are currently lacking
 
  • Like
Reactions: Xon
Moshe1010 said:
They have a server, cloud and desktop solutions; I used the desktop version and uploaded everything manually to my FTP
Click to expand...
How do you do this? You change .data files to ,jpg in your desktop and after compression change it again to original file extension?
 
Xon said:
The metadata is stored in the database for the original filename, and detected mime-type.

Any script would need to mark that that an attachment has been processed anyway.
Click to expand...

The first point is harder then the second, but should be solvable over tge mime type and the file command.

The second point: i wouldnt mark a file, i would select the files based on modification date on a file level
 
gfc said:
The second point: i wouldnt mark a file, i would select the files based on modification date on a file level
Click to expand...
Please don't use file dates as a stand in for the metadata describing if you have processed a file or not.

This is a bad idea, as you now need to be careful that you don't run the script more than once, which leads into supporting multiple web nodes.

Fundamentally, reusing some data which means something completely different for the rest of the system as your 'have I done stuff' flag is a bad idea.
 
The other essy Options would be
  • Extended File Attributes (never used that before)
  • Checksums saved to a flat file (might be slow on checking)
If i'm writing to a DB that would be an overkill.
 
gfc said:
The other essy Options would be
  • Extended File Attributes (never used that before)
  • Checksums saved to a flat file (might be slow on checking)
If i'm writing to a DB that would be an overkill.
Click to expand...
My recommendation is to copy the XF deferred script for setting up a XF run-time instance, and then use XenForo_Application::getDb()

This allows you to control settings of the script from XF itself, without requiring code edits.

You can then log specific error messages to the XF error log via XenForo_Error::debug("my error message").
 
yeah that I would do if I would sell the addon. But for my need i want to stay at bash only. But you're free to develop such an addon.
 
This jpegmini thing looks fantastic, I tried it on a few shots and the results are perfectly acceptable in terms of quality, even more if you only use it at a forum and not your professional shots.

I'll try to compress my attachments folder and see how that goes :)

Thanks @Moshe1010 !
 
Moshe1010 said:
I used the desktop version and uploaded everything manually to my FTP
Click to expand...
You downloaded 8GB of images off your server and then reuploaded them? How long did that take!? Download the software and drag and drop into program? You do this, what, like once a month?
 
You must log in or register to reply here.
Back
Top Bottom