XF 2.4 Chunked uploads

HYS 24 Chunked Uploads.webp
Welcome back to our Have you seen...? series for XenForo 2.4! The code reviews are coming in thick and fast as we complete our planned features so there should be plenty to show you over the coming weeks.

Part of our process for this and previous releases has involved looking at the amazing pool of developer talent that we have in this community and considering how we can work with those developers to bring those features to the core.

And that's the focus of this time where we'll be taking a look at the Chunked Uploads add-on by @JulianD which, starting with XenForo 2.4, will become a core feature.

But first...​

We have decided to accelerate our plans to modernise the minimum version of PHP that is supported by XenForo. While we had originally planned to increase the minimum version to PHP 7.4, in recent weeks we have decided that it might be time to leave PHP 7.x behind.

As such, XenForo 2.4 will require PHP 8.0.2 as a minimum. Around 90% of our customers running XF 2.3 are already running PHP 8.0 and above. If you're still on XF 2.2, you can get ahead of the game right now as we have supported PHP 8.0 for quite a number of years!

With PHP 8.0, as well as being able to finally use all of the new syntax stuff that we're usually deprived of, it also allows us to modernise some of the third party dependencies we use.

Chunkier uploads​


"Chunked uploads" is the process of splitting uploaded files into much smaller chunks which are then re-assembled on the server. The main reason you might want to do this is if you have very strict file upload limits imposed by your host, you want to upload much larger files, or you proxy your site behind services such as Cloudflare.

Cloudflare for example imposes a 100MB upload limit, regardless of what your server configuration may be. In some cases the default max upload size in PHP is as little as 2MB. Or, in some cases, your server config might be reasonable but you might occasionally need to upload larger files than allowed.

In all of these examples, you could pay Cloudflare money, or faff around with server configs or, alternatively, you can just enabled "Chunked uploads":

1738859856069.webp


As soon as you enable chunked uploads, the theoretical maximum file size you will be able to upload will now be a massive 128GB! We can do that by splitting a file into much smaller chunks. You can see from the "Chunk size" option, this determines the maximum size of each chunk that is uploaded. In the example above, my server limits the maximum file size to 2,048 KB so I should set the chunk size to a value below that. We can also customise the number of simultaneous chunks that can be uploaded at once. This is potentially useful for performance, theoretically allowing larger files with a large number of chunks to be uploaded more quickly.

And once enabled, as you can see, the normal "Maximum file size" options are now capable of being increased to a much larger size than previously allowed:

1738860234987.webp


We'd like to extend our thanks again to @JulianD for allowing us to not have to reinvent the wheel and giving us access to his code for the development of this feature.

For our next Have you seen...? we might have another feature or two from a prominent developer here or we might be rounding off some existing features with some new goodies. Which one? I'll let you know when I'm writing the HYS :)
 
Thank you @Chris D for biring up this to 2.4!

Also congratulations @JulianD

Will your add-on be free on 2.3? I don't have much time for 2.4 to install my addon on my site. Because I built my own addon for development.

Also I cannot buy your add-on without a Paypal account
 
Thank you @Chris D for biring up this to 2.4!

Also congratulations @JulianD

Will your add-on be free on 2.3? I don't have much time for 2.4 to install my addon on my site. Because I built my own addon for development.

Also I cannot buy your add-on without a Paypal account

I don’t think it’s necessarily reasonable to expect it for free. But that would be a question best asked directly to them or in the resource thread.
 

But first...​

We have decided to accelerate our plans to modernise the minimum version of PHP that is supported by XenForo. While we had originally planned to increase the minimum version to PHP 7.4, in recent weeks we have decided that it might be time to leave PHP 7.x behind.

As such, XenForo 2.4 will require PHP 8.0.2 as a minimum. Around 90% of our customers running XF 2.3 are already running PHP 8.0 and above. If you're still on XF 2.2, you can get ahead of the game right now as we have supported PHP 8.0 for quite a number of years!

With PHP 8.0, as well as being able to finally use all of the new syntax stuff that we're usually deprived of, it also allows us to modernise some of the third party dependencies we use.
Neat :O Are you able to share the current version breakdown? Want to figure out if I should bump my PHP version req again... 🤔
 
Great to see XF working with 3rd party devs to absorb some of their work into the core. In the end every one benefits from having a stronger feature set in the core product and free up dev time to work on other unrealised features.

As long as XF aren't pulling a Mullenweg and just commandeering entire resources via hostile takeover 😉
 
Core support for chunked uploads is nice :)

But this:
As such, XenForo 2.4 will require PHP 8.0.2 as a minimum.
is really appreciated.

Awesome! Would this also allow resuming interrupted uploads?
Unlikely at least at this point.
This is a bit disappointing as our inhouse chunked uploads Add-on (which we've been using for a few years by now with XF 2.1-2.3) does support this.
Would be nice if resuming could be implemented in core as well before 2.4 final release so I could drop another Add-on :)

Does the XenForo implementation handle uploads of large files (on object storage) without potentially running into timeouts?
Our Add-on unfortunately doesn't handle this too well as it requires rebuilding the original file from the chunks (parts) & saving that to VFS in one synchronous step after the last chunk (part) has been received.
Ideally chunked uploads should probably take advantage of the underlying storage (like S3) to handle multipart uploads
At least the chunks (parts) should (also) be kept locally to avoid having to download them again for rebuilding the original file.
 
I don't think there are very many add ons that are large, but I'd like to ask about this anyway... if you have $config['enableAddOnArchiveInstaller'] = true; set are add ons chunked with this too?

I only ever ran into this issue with Badges as it's above the 2M limit mentioned (easily gotten around by editing config files, but might be for consideration).
 
I don't know how to program, but I think it's easier to set up an option to send X emails every hour rather than making a system that splits the files and then reassembles them.
 
I'm pretty happy with this one as I'm using CloudFlare and voted for the suggestion. Not meant as criticism, but I'm wondering though though why a suggestion with only 29 votes (176th place of popularity) is considered when more popular ones can be implemented?
 
I'm pretty happy with this one as I'm using CloudFlare and voted for the suggestion. Not meant as criticism, but I'm wondering though though why a suggestion with only 29 votes (176th place of popularity) is considered when more popular ones can be implemented?
Because a lot of the work was already done, and a lot of 2.3 (and apparently 2.4 and 3.0) is focused on optimization and quality of life improvements.

I am actually happy to see some of the smaller suggestions that are more developer focused to be included, as for quite a while a lot of those suggestions were marked as "No interest", and it felt like making any technical suggestion was not worth the effort.
 
I'm pretty happy with this one as I'm using CloudFlare and voted for the suggestion. Not meant as criticism, but I'm wondering though though why a suggestion with only 29 votes (176th place of popularity) is considered when more popular ones can be implemented?
More popular ones will be implemented of course. It’s a little more niche but it’s a relatively quick win - especially when we didn’t conceptualise or write the majority of the code - that clearly benefits people so it’s certainly valid.
 
I'm pretty happy with this one as I'm using CloudFlare and voted for the suggestion. Not meant as criticism, but I'm wondering though though why a suggestion with only 29 votes (176th place of popularity) is considered when more popular ones can be implemented?

It probably also makes a huge amount of sense considering how many self hosted XF sites are now behind CDNs like cloudflare and the increased need to be able to upload large media files.

I have no idea because I haven't looked, but I wouldn't be surprised if XF Cloud also used a CDN, which would make all those sites also benefit from something like this. Cloudlfare or otherwise.

Either way, keep the easy wins and new features coming! It's a fantastic feature, regardless of the reason or how innovative it is.
 
I like this approach. XenForo certainly has quite a lot of popular addons by excellent 3rd party developers. After 2.4 is out, I assume that XF will be able see what the addons with the most installs are.
 
Php 8 as the minimum php version does mean that upgrading from xf1 can get tricky because xf1 has a maximum php version of 7.3
Probably best to go to xf2.3 first and when that's successful then go to xf2.4
If I have not made it before xf2.4 is out then I will need a way to stop the cli upgrade at 2.3 to prevent a php mismatch .
 
Back
Top Bottom