HUGE Space Size Savings On Images

[ProCom]

As I posted here, I'm in the process of converting a huge site to XF.

I also posted here asking for info about resizing uploaded pics.

Well, I have about 4 TB of images that I'm moving from my old host / site. A TON of the images are around 9mb in size.

We ran a bunch of tests on some big images and we’re going to run a script to resize and recompress all the huge files. This should save us a TON of space and bandwidth!

We’re using mogrify (script with ImageMagick) to do two things:

1. Resize images to a max of 2000 x 2000
2. Compress images to 85% quality

Check out the attached screenshot of 3 version of each file we tested:

a = full size original image
b = resized and default 92% compression
c = resized and 85% compression



So, we'll set the script to find all images over 3mb and resize them all down.

Incredible savings on sizes! Most went from 9mb down to around 500 kb!!!

 

[Mouth]

We’re using mogrify (script with ImageMagick)

Sounds great. Please share the script?

 

[Mouth]

Incredible savings on sizes! Most went from 9mb down to around 500 kb!!!

Do your images still work properly? I always understood there was a problem modifying/resizing images outside of XF since XF stores a checksum of the image file in the DB and after resizing the checksum changes and XF then has a mismatch and your have image display issues.

 

[ProCom]

Sounds great. Please share the script?

I sure will once we finish and fully test it.

Do your images still work properly?

Hmm... that's a fantastic question, and definitely something I should have thought about. We're applying this to the images from the previous platform that are not stored within a filesystem like XF, and we're making the changes before importing them into XF.

 

[Paul B]

If the images are modified before importing into XF, that should be fine.

 

[Mouth]

If the images are modified before importing into XF, that should be fine.

Is there a similiar script available that will also action the DB appropriately when the images are already listed within XF?

 

[Paul B]

I'm not aware of one.

 

[AndyB]

Is there a similiar script available that will also action the DB appropriately when the images are already listed within XF?

Yes there is.

https://xenforo.com/community/resources/image-resizer.2923/

 

[eva2000]

Well, I have about 4 TB of images that I'm moving from my old host / site. A TON of the images are around 9mb in size.

We ran a bunch of tests on some big images and we’re going to run a script to resize and recompress all the huge files. This should save us a TON of space and bandwidth!

nice, you inspired me to whip up optimise-images.sh for testing https://github.com/centminmod/optimise-images

• 24/04/17 Example
• 25/04/17 New Option: Compare Mode
• 26/04/17 New Option: WebP conversion support
• 26/04/17 New Option: WebP conversion + Compare Mode

 

[Mouth]

Yes there is.

https://xenforo.com/community/resources/image-resizer.2923/

Unfortunately, I don't ImageMagik within my PHP.

 

[MattW]

Do your images still work properly? I always understood there was a problem modifying/resizing images outside of XF since XF stores a checksum of the image file in the DB and after resizing the checksum changes and XF then has a mismatch and your have image display issues.

The images aren't going to be in the XF attachment system, they are going to be served directly via nginx using the previous cdn URL.

 

[eva2000]

The images aren't going to be in the XF attachment system, they are going to be served directly via nginx using the previous cdn URL.

ingenious - great way to keep url format and offload older attachments

 

[eva2000]

Oooh here comes Guetzli for Jpeg compression https://community.centminmod.com/th...-–-comparing-size-vs-optimization-time.11520/

 

[Anthony Parsons]

I personally like leaving the image optimisation process at cloudflare's server, mirage and polish handle it all (including webp), across all devices. I only downsize the image at my server on upload.

 

[eva2000]

I personally like leaving the image optimisation process at cloudflare's server, mirage and polish handle it all (including webp), across all devices. I only downsize the image at my server on upload.

i similarly do this via my Centmin Mod Nginx's ngx_pagespeed integration which does on the fly image optimisations including jpg/png to webp conversions for detected web browsers that support it https://community.centminmod.com/threads/benefits-of-ngx_pagespeed.1032/

Example of my Xenforo's attachments being on the fly converted to webp for web browser that supports it

auto optimised conversion of user attachments from PNG to WebP format where browser supports WebP. Almost 88KB size reduction for those 3 images combined by converting on the fly to WebP format.

hment-files/2014/05/198_upload_*.png' (24863 bytes) to `*attachmentlink*/2014/05/x198_upload_*.png.pagespeed.ic.3Exc-iTnt7.webp' (6268 bytes)
*threadlink*/:1241: Shrinking image `*attachmentlink*/2014/05/199_upload_2014-*.png' (42496 bytes) to `*attachmentlink*/2014/05/x199_upload_2014-*.png.pagespeed.ic.MtXR6WHd7d.webp' (12718 bytes)
*threadlink*/:1248: Shrinking image `*attachmentlink*/2014/05/200_upload_2014-*.png' (53259 bytes) to `*attachmentlink*/2014/05/x200_upload_2014-*.png.pagespeed.ic.QRsR_7kaYd.webp' (14484 bytes)

 

[Anthony Parsons]

Yer, used the page speed version. Does the same thing, absolutely. I simply like mine done at cloud flare, as it removes another burden from the server, and images resized on upload are reduced enough from a storage view for me. The cached version at cloud flare is then optimal and using their system to do it.

 

[MattW]

We need to shrink these down at source initially, as the old system stored full fat versions, along with multiple smaller ones as well. Will make managing them easier in the long run.

 

[eva2000]

We need to shrink these down at source initially, as the old system stored full fat versions, along with multiple smaller ones as well. Will make managing them easier in the long run.

Does that mean the old forum system actively referenced different sizes of the same image i.e. image srcset usage ? If you're keeping the old url format, does that mean you need to also make multiple sizes of same image but just optimise each set of images as well ? That would be alot of images !

 

[MattW]

I'm hoping @Slavik knows the answer to that, as he's doing the conversion from the previous software. Inspecting the source shows most are pulling the full CDN URL when they are embedded, and allowing them to pop out in the lightbox

Not sure how the thumbnails are going to work thorugh, as they look to be a mapped URL

content/type/61/id/7803445/width/350/height/700/flags/LL

 

[CvP]

Does that mean the old forum system actively referenced different sizes of the same image i.e. image srcset usage ? If you're keeping the old url format, does that mean you need to also make multiple sizes of same image but just optimise each set of images as well ? That would be alot of images !

Yes, to all.