XenForo need a revamped attachments system

[Kacee]

Currently XenForo have a very ineffective attachments management system, performance-wise.

First of all, it stores attachments (images) uploads as is, there is no optimization applied (that I know of).
Secondly, it serve attachments with "private, no-cache, max-age=0" so every request to view an attachment must go through XenForo before the browser cache can be used. This is needed to control attachment permission and view counts, though.

One PHP hit for every attachment view is simply ridiculous, it costs huge un-need latency, this latency can be optimized by caching view count in redis/memcached (I haven't inspected this yet) but still.

Users don't care about optimizing their images before upload, no one ever does, and smartphone these days are known for huge file size for the resolution their camera take.

My suggestions:

• Have an optimization process in place, either at client side (with javascript, for example pica https://github.com/nodeca/pica) or at server side (there are many) or both.
• Have an option to serve attachment directly with webserver, no view count or permission check needed, cache-control set properly.
• Optionally serve webp format for compatible browser (which are taking huge market share with Chrome and Edge)

Proof of concept:

I have implemented a custom solution to serve optimized attachment images to our guests, this is some proof of the huge improvements it made.

First of all attachments are served as is, no optimization applied. Note the Size and Time column.



This is with optimization applied, images are converted to webp.



This is statistics of the image re-compressor (pagespeed) - please disregard the percentage, instead of serving 424MB it saved 304MB and served only 120MB, that is 75% bandwidth saved!!! - this is for 30 minutes at 3AM, imagine how much it could save during peak hours.



Why is this important?

Speed!

In the screenshot, those value in time column are without PHP time, please add 50-100ms (depend on your server) to each of them. And this is on Gigabit Fiber connection, on 3G it could took 30 seconds+++ to load a page with 10 or more attachments. We have a review section with very high traffic and users are basically cry themselves out for revisiting a thread to comment.

Doing this could make that 30 seconds into 1-2 seconds, I'm not kidding.

Not to mention it could save your user tons of mobile data.

 

[Kacee]

This is an example of how many attachments can be in the first post on our site. This is first visit without cache.



That same thread refreshed, attachments are cached by browser, no need to go through XenForo to check for permission and register view count. Single digit ms in latency, not 50ms++ for each. Thread loaded instantly.

 

[Chromaniac]

the good thing is that apple has finally added support for webp in next updates of macos and ios. so a lot of apple users would have the ability to view webp by the end of the year (or early next). so for starters, it would be great if xenforo 2.2.x would treat webp at par with jpgs/pngs wrt attachment system and proxy system.

i would also love to get internal support for optimizations using open source tools like mozjpg and oxipng. there is a paid addon from theme house which offers some of these functionalities.

i have been considering implementing pagespeed module on my website for a while now. but of course, internal support would be great!

 

[🔥Iggy🔥]

anything improving image handling, serving, use etc is a big party time for us....we hope to see more as 2.2 grows into itself....

one thought about the optimization tho... i feel to disagree of in favor of a better solution....

the ideal here, for us, is to leave original images "as uploaded" but only viewable by click to lightbox type function....this maintains image integrity without slowing thread page load times....

providing that.......

xenforo can give us a medium size embed option...

at best full size images are such waste to embed only to be shrunken to the view port....at worst, its a real actual problem....it seems a clear logic to me but failed to gain support so far...

 

[RallyFan]

Older Apple users will not be able to see images if in webp format though.

 

[Chromaniac]

that's true. so i guess we would have to wait for a few more years before webp can be treated at par with jpegs and pngs. but it would be nice to have it as an opt-in so that forums where the userbase of unsupported platforms is minimal. in the end, there is always going to be some kind of traffic from such platforms.

the good thing is that apple device owners do update to latest builds rather quickly and apple is supporting 4-5 year old iphones with the latest ios update iirc.

 

[Kacee]

anything improving image handling, serving, use etc is a big party time for us....we hope to see more as 2.2 grows into itself....

one thought about the optimization tho... i feel to disagree of in favor of a better solution....

the ideal here, for us, is to leave original images "as uploaded" but only viewable by click to lightbox type function....this maintains image integrity without slowing thread page load times....

providing that.......

xenforo can give us a medium size embed option...

at best full size images are such waste to embed only to be shrunken to the view port....at worst, its a real actual problem....it seems a clear logic to me but failed to gain support so far...

if by integrity you mean quality then no need to worry, image optimization has come so far that size can be reduce up to 99% WITHOUT decreasing in quality, even lossless... of course that the best case scenario but 20-50% is normal... Google has already done that for a looooong time with Google Photos...

Older Apple users will not be able to see images if in webp format though.

that's true. so i guess we would have to wait for a few more years before webp can be treated at par with jpegs and pngs. but it would be nice to have it as an opt-in so that forums where the userbase of unsupported platforms is minimal. in the end, there is always going to be some kind of traffic from such platforms.

the good thing is that apple device owners do update to latest builds rather quickly and apple is supporting 4-5 year old iphones with the latest ios update iirc.

webp can be served conditionally, for only browser that accept image/webp, you don't have to choose one or the other.

 

[Chromaniac]

For that you would need to keep multiple copes of each image file or generate on the fly. As typically done by CDNs. Honestly looking forward to the time when we won't need to do this for Apple device users in coming years.

 

[Kacee]

For that you would need to keep multiple copes of each image file or generate on the fly. As typically done by CDNs. Honestly looking forward to the time when we won't need to do this for Apple device users in coming years.

well you can do both, frequently accessed images (like avatars) are kept with a WEBP copy permanently, attachments are generated on the fly and keep in memory with TTL of a week since last access...

 

[🔥Iggy🔥]

if by integrity you mean quality then no need to worry, image optimization has come so far that size can be reduce up to 99% WITHOUT decreasing in quality, even lossless... of course that the best case scenario but 20-50% is normal... Google has already done that for a looooong time with Google Photos...

yes i am aware.....however it remains less than ideal....what we wish to do is compress ONLY images served in content(threads), which would be thumbnail and "medium" size given the option....we may perhaps do a smaller compression to the full sized images but really i didnt see it as much concern as those will only ever be loaded one at a time....

even with compression the difference in dimension can be significant when adding up many photos.....a compressed 1600px image will be much smaller than a compressed 5000px....also, to leave originals untouched, its a sort of advertisement even for photo centric sites... its the same policy twitter adopted as i understand it...

the biggest difference and true optimization however will be in creating another size to embed....that is the only way to cut off all waste of data served how we see it...

 

[Kacee]

yes i am aware.....however it remains less than ideal....what we wish to do is compress ONLY images served in content(threads), which would be thumbnail and "medium" size given the option....we may perhaps do a smaller compression to the full sized images but really i didnt see it as much concern as those will only ever be loaded one at a time....

even with compression the difference in dimension can be significant when adding up many photos.....a compressed 1600px image will be much smaller than a compressed 5000px....also, to leave originals untouched, its a sort of advertisement even for photo centric sites... its the same policy twitter adopted as i understand it...

the biggest difference and true optimization however will be in creating another size to embed....that is the only way to cut off all waste of data served how we see it...

you have fair points, I haven't thought about that use case... XenForo already keeping several copies of avatars in various sizes so I think it's not a problem implement your suggestions.

 

[Karelke]

It would help if people insert thumbnails instead of the original, full size images. The sometimes endless scroll of large images is not really fine for the user experience IMO. The full size images are fine for the lightbox, and do not need to be compressed as they would only be downloaded when the lightbox opens.

 

[🔥Iggy🔥]

It would help if people insert thumbnails instead of the original, full size images. The sometimes endless scroll of large images is not really fine for the user experience IMO. The full size images are fine for the lightbox, and do not need to be compressed as they would only be downloaded when the lightbox opens.

yes that is what i mean for 2 sizes of thumbnail basically...thumbnails and full sized are served from different directories so its easy enough to compress only the former...in some cases photographers may actually use forums to sell original photos so its a good option to leave those separated imho...it is also possible to break up a long scroll by bbcodes like tabs and sliders or such, but any optimization is always a plus with such an increase in mobile use etc..

 

[Kirby]

Have an optimization process in place, either at client side (with javascript, for example pica https://github.com/nodeca/pica) or at server side (there are many) or both.

Hmm, I've never heard of any image optimization libary (apart from resizing which is the only thing Pica does) that would run on the client?

Having image attachmted serverd by the webserver directly is not difficult, I just did a PoC implementaiton for that.
But I wouldn't want to use this in production due to missing access permission checks.

 

[Kacee]

Hmm, I've never heard of any image optimization libary (apart from resizing with is the only thing Pica does) that would run on the client?

Having image attachmted serverd by the webserver directly is not difficult, I just did a PoC implementaiton for that.
But I wouldn't want to use this in production due to missing access permission checks.

Unfortunately I don't use any library except pica so I can not say, but quick search on npm came up with many.

But pica does have its use, except some niche boards where people share high quality full size photos for printing, pixel peeping... most only view photos/images on screen, so there's not much use for larger than 3000x2000 res (and remember this res is enough for high quality A4 size prints)

So pica step in and resize original images down to certain res, save on uploading size while ensuring high quality. It doesn't work as good as other server side optimization tools but it excels in speed and quality on client devices. I have one image upload site with pica and people have been uploading TB of images in one go, it's just resize all of them resulting in few hundreds KB images.

You can do further optimization server side as needed, I use pagespeed with in place resourse optimization to optimize/convert to WEBP on the fly.