XF 2.1 Large attachment size issue

[steveabraham]

Hey all,

So I've got a very edge-case issue and I'm wondering if anyone can perhaps just nudge me in the right direction.

I have an Xenforo (latest) install that's used internally to store and organise diagnostic information and tools for my work, I use it very much like a wiki, cross linked posts, downloads, etc, etc.

Some of these files are disk images, which can be very large, 3GB to 5.5GB is quite common. For this reason I have the PHP upload size limit set to 8GB and all the files up to around 4.5GB work flawlessly. Decent bandwidth here, so upload speed isn't an issue even at that size.

The problem I have is as soon as I hit 5GB files, they just fail. No error, no 'too big' message, the file just goes grey as soon as it completes. I can't see anything in the error logs, nothing is output to the browser console window. 4.5GB is fine, no problem, so this makes me think it's not a 'limit' or a 'timeout' problem, there's just something inherent about 5GB as a file size that's not able to be processed.

I appreciate this is an edge case problem, and it's really bending what the software (and even the protocols) are designed to do, but it's bugging me since I can sit and upload 4.5GB all day long without an issue. So I'm hoping maybe someone has an idea that gets me sorted.

Thank you!

 

[Kirby]

You've also increased post_max_size?
upload_tmp_dir has enough free space to hold files that large?

 

[steveabraham]

You've also increased post_max_size?
upload_tmp_dir has enough free space to hold files that large?

Thanks for the quick reply! post_max_size is matching the upload size at 8GB, however upload_tmp_dir is not set according to phpinfo(), what would that default to (Ubuntu 18.04, Apache, PHP FPM 7.2), do you know?

 

[steveabraham]

Found it, you were spot on. /tmp is only 5GB. So now to decide do I reconfigure the filesystem or just use a different temp location... anyway, that's my problem.

Thank you for the help, much appreciated.

 

[Kirby]

If unconfigured (default) it uses system /tmp as you already found out

I'd set a specific directory just for the XenForo installation (via its own PHP-FPM pool).
Storing large uploads that, unlike images, don't need to be processed, in memory seems a waste of resources to me. On the other hand having /tmp as tmpfs is a good thing for many cases (small stuff) so I'd keep that.

 

[steveabraham]

Thanks! Yep, that makes sense, I think that's exactly what I'll do, I just need to troubleshoot it a bit. I've got php_admin_value[upload_tmp_dir] set and it's showing in phpinfo() but for some reason it's still using /tmp for uploads.

 

[steveabraham]

OK, managed to get that sorted, so the system is now using a new location for temporary uploads where size isn't an issue (800+GB free). However I'm back to the original problem.

If I check the temp location as the upload hits 100%, I can confirm the entire filesize is uploaded into the temp location, it's not being truncated at all. A few seconds after that, it just drops back to a greyed out filename that's scored through and the temp file vanishes.

 

[Kirby]

Did you configure $config['tempDataPath'] in XenForo config.php?
By default this is internal_data/temp.

Do you have enough free space there as well?

 

[steveabraham]

I've not set it, but internal_data/temp is on the same volume as the new upload_tmp_dir, which has loads of space.

 

[Kirby]

Hmm, except for debugging the code step by step I can't think of anything else at the moment
Sorry.

 

[steveabraham]

No problem, thanks for helping.

Is there any additional logging/etc I can enable to see if anything shows up?

 

[Kirby]

You could enable debug mode, though I doubt that would bring much (if any) additional info for troubleshooting.

 

[steveabraham]

After playing around a bit more and not getting anywhere, I tried a little file upload script and the 5GB file has uploaded without any issues. So this looks like it's a limit within Xenforo, and I don't think there's anything further I can diagnose, but at least I know it's not environmental now.

 

[elisovs]

After playing around a bit more and not getting anywhere, I tried a little file upload script and the 5GB file has uploaded without any issues. So this looks like it's a limit within Xenforo, and I don't think there's anything further I can diagnose, but at least I know it's not environmental now.

Tell me, did you solve the problem? I have 6+ GB files

 

[steveabraham]

Yes, the support team gave me hack to do on one of the core files and said the issue would be resolved in a future update. I’m actually not 100% sure if that has happened yet, I’ve taken me eye off this project for a few weeks.

 

[steveabraham]

This is what they sent me, use at your own risk, etc.

Hopefully we can roll this out in XenForo 2.2 in full, but for now you can make the following changes.

Firstly please run this SQL query:

ALTER TABLE xf_attachment_data CHANGE file_size file_size bigint NOT NULL;

Then in Entity\AttachmentData.php. Find:

'file_size' => ['type' => self::UINT, 'required' => true],

and replace with

'file_size' => ['type' => self::UINT, 'required' => true, 'max' => PHP_INT_MAX],


That should get you going again for now!

 

[monir]

Hopefully we can roll this out in XenForo 2.2 in full, but for now you can make the following changes.

Firstly please run this SQL query:

ALTER TABLE xf_attachment_data CHANGE file_size file_size bigint NOT NULL;

Then in Entity\AttachmentData.php. Find:

'file_size' => ['type' => self::UINT, 'required' => true],

and replace with

'file_size' => ['type' => self::UINT, 'required' => true, 'max' => PHP_INT_MAX],


That should get you going again for now!

where to find this?

 

[monir]

above solution not work for me.