XF 2.4 Chunked uploads

HYS 24 Chunked Uploads.webp
Welcome back to our Have you seen...? series for XenForo 2.4! The code reviews are coming in thick and fast as we complete our planned features so there should be plenty to show you over the coming weeks.

Part of our process for this and previous releases has involved looking at the amazing pool of developer talent that we have in this community and considering how we can work with those developers to bring those features to the core.

And that's the focus of this time where we'll be taking a look at the Chunked Uploads add-on by @JulianD which, starting with XenForo 2.4, will become a core feature.

But first...​

We have decided to accelerate our plans to modernise the minimum version of PHP that is supported by XenForo. While we had originally planned to increase the minimum version to PHP 7.4, in recent weeks we have decided that it might be time to leave PHP 7.x behind.

As such, XenForo 2.4 will require PHP 8.0.2 as a minimum. Around 90% of our customers running XF 2.3 are already running PHP 8.0 and above. If you're still on XF 2.2, you can get ahead of the game right now as we have supported PHP 8.0 for quite a number of years!

With PHP 8.0, as well as being able to finally use all of the new syntax stuff that we're usually deprived of, it also allows us to modernise some of the third party dependencies we use.

Chunkier uploads​


"Chunked uploads" is the process of splitting uploaded files into much smaller chunks which are then re-assembled on the server. The main reason you might want to do this is if you have very strict file upload limits imposed by your host, you want to upload much larger files, or you proxy your site behind services such as Cloudflare.

Cloudflare for example imposes a 100MB upload limit, regardless of what your server configuration may be. In some cases the default max upload size in PHP is as little as 2MB. Or, in some cases, your server config might be reasonable but you might occasionally need to upload larger files than allowed.

In all of these examples, you could pay Cloudflare money, or faff around with server configs or, alternatively, you can just enabled "Chunked uploads":

1738859856069.webp


As soon as you enable chunked uploads, the theoretical maximum file size you will be able to upload will now be a massive 128GB! We can do that by splitting a file into much smaller chunks. You can see from the "Chunk size" option, this determines the maximum size of each chunk that is uploaded. In the example above, my server limits the maximum file size to 2,048 KB so I should set the chunk size to a value below that. We can also customise the number of simultaneous chunks that can be uploaded at once. This is potentially useful for performance, theoretically allowing larger files with a large number of chunks to be uploaded more quickly.

And once enabled, as you can see, the normal "Maximum file size" options are now capable of being increased to a much larger size than previously allowed:

1738860234987.webp


We'd like to extend our thanks again to @JulianD for allowing us to not have to reinvent the wheel and giving us access to his code for the development of this feature.

For our next Have you seen...? we might have another feature or two from a prominent developer here or we might be rounding off some existing features with some new goodies. Which one? I'll let you know when I'm writing the HYS :)
 
Php 8 as the minimum php version does mean that upgrading from xf1 can get tricky because xf1 has a maximum php version of 7.3
Probably best to go to xf2.3 first and when that's successful then go to xf2.4
If I have not made it before xf2.4 is out then I will need a way to stop the cli upgrade at 2.3 to prevent a php mismatch .
Why? Wouldn't you be able to simply upload a fresh copy of 2.4, then run the CLI upgrade? The upgrade script wouldn't touch the XF1 code, only the database and the data files 🤔
 
Why? Wouldn't you be able to simply upload a fresh copy of 2.4, then run the CLI upgrade? The upgrade script wouldn't touch the XF1 code, only the database and the data files 🤔
Because there is no guarantee that it will be successful this time. The live site will need to stay running until the upgrade is successful .The live site on xf1 has a maximum php version of 7.3 and centminmod allos one php version on a server.

Before starting the upgrade script from xf1 there are some steps to take on xf1 , like uninstalling addons and disabling addons. So the start or preparation of the upgrade is on a staging version of the live site on php7.
 
Because there is no guarantee that it will be successful this time. The live site will need to stay running until the upgrade is successful .The live site on xf1 has a maximum php version of 7.3 and centminmod allos one php version on a server.

Before starting the upgrade script from xf1 there are some steps to take on xf1 , like uninstalling addons and disabling addons. So the start or preparation of the upgrade is on a staging version of the live site on php7.
I mean, you can't exactly do an upgrade to any version of XF if the goal is to be able to instantly bring your old version back if it's unsuccessful, so you'd need to clone files/database no matter what, even if it was just 1.5 to 2.0 :P
 
Yeah sorry Alpha that doesn’t make sense.

You schedule the time for the upgrade, notifying your users. Upload the 2.4 files. Upgrade PHP. Run the upgrade.

If there’s a failure, you can revert everything.

But there won’t be a failure as no doubt you will do a test run first.

It’s really not complicated. There’s no correlation between the max version of XF1 and the minimum version of XF2.
 
Maybe the missing part of the information is that it concerns a big board with many major addons. The upgrade process alone takes a week. Some of the addons (social groups and wiki for example are very prone to upgrade issues )
I've ran many test runs and will continue to do that until one is successful .
 
You’re by far the only customer I’ve ever come across who alleges to have as many issues with either importing to XF, as it used to be, or upgrading to a newer version as it is now.

You’re not particularly unique, many customers have and continue to make the move and by comparison make it look effortless. Some with fewer add-ons, some with many.

I’m not sure what’s going wrong in the process but I’m confident you’ll make it to 2.x soon enough.
 
No but more seriously. Is there no ability to take a snapshot of your server @Alpha1 ? That's what I usually do before doing anything to a live site, just to be sure. This way, when anything goes wrong, you'll lose a few posts and a bit of time, that's all.
 
No but more seriously. Is there no ability to take a snapshot of your server @Alpha1 ? That's what I usually do before doing anything to a live site, just to be sure. This way, when anything goes wrong, you'll lose a few posts and a bit of time, that's all.
I use a copy of the live site. Try to upgrade that while I keep the live site active. I do not take it offline at all. If the upgrade is successful I will put the upgraded version live. It means that I will loose the posts made during the process, but that's fine. In the grand scheme of things its not a big deal.
We also have daily backups of sites and servers.
 
I use a copy of the live site. Try to upgrade that while I keep the live site active. I do not take it offline at all. If the upgrade is successful I will put the upgraded version live. It means that I will loose the posts made during the process, but that's fine. In the grand scheme of things its not a big deal.
We also have daily backups of sites and servers.

Definitely not the ideal way. If you are doing the upgrade process like this, I would do it late at night and use the setting in the admin panel to make the board "in active". Then go through that process you are talking about. Once you are done, re-activate the board.

In an "ideal" world, you would have a local copy on your machine to run the upgrades. If good, upload the files to production and run the upgrade. (of course making the board in-active before the upload/upgrade and re-activate after the upload/upgrade is complete)

In my world, we will push upgrades through our multiple environments through automated processes for testing and then kick off a final process for the production deployment process that will automatically run CLI scripts to complete the deployment. The whole upgrade process takes 2 minutes at most. (next step is blue/green deployments)
 
I use a copy of the live site. Try to upgrade that while I keep the live site active. I do not take it offline at all. If the upgrade is successful I will put the upgraded version live. It means that I will loose the posts made during the process, but that's fine. In the grand scheme of things its not a big deal.
We also have daily backups of sites and servers.
Have you ever contacted someone to help see why your upgrade is so difficult?

Even when I did a XF1 -> XF 2.2 migration from one custom plugin to another on a big board, as well as a large custom content add-on it did not take more than a day, and that was including the manual work I had to do.
 
Back
Top Bottom