MG 2.2 Jumbo video size upload

[stromb0li]

I have a about 350GB worth of 4k videos from a recent event our community held, where some files are be up to 20GB-25GB each. What is the best way to pre-upload them to the server and reference via XFMG?

 

[Paul B]

There is no "pre-upload" function.

Just upload them to the media gallery and once transcoding is complete, you will receive a notification and it will be posted.

 

[stromb0li]

There is no "pre-upload" function.

Just upload them to the media gallery and once transcoding is complete, you will receive a notification and it will be posted.

The problem is XenForo doesn't support chunking. A single 20GB post is going to be a no-go through Cloudflare. If I host file my local machine to point directly to the webserver, both PHP and NGINX are going to say lol processing that request as well.

 

[Paul B]

I misread the G as M.

Can't the files be reduced in size/quality at all?
Or split into multiple pieces?

 

[stromb0li]

I'm already splitting what would be 100GB into 20GB chunks

 

[Ozzy47]

This paid addon supports chunked uploads, https://xenforo.com/community/resources/chunked-uploads-xf2.7133/ I’d verify with the author though before purchasing.

 

[stromb0li]

File upload works with the add-on, but NGINX times out and will start throwing 504s post upload on this step: https://my.domain.com/attachments/u..._id]=15&hash=90851c1a2d141b69297ba80295700833

Is there a way to debug what action is specifically dieing here? After upload, does XF serially process the video on the same request or should transcoding be async?

 

[Kirby]

Is there a way to debug what action is specifically dieing here?

Most likely reconstructing the original file or creating thumbnail / post image.

XFMG video handling is kinda inefficient and you will most certainly run into such issues when dealing with really large videos, especially if using cloud storage (S3, R2, etc.)

K

Thread 'Thumbnail / Poster handling seems inefficient'

Jul 10, 2022

\XFMG\Attachment\Media::onAssociation

if ($tempMedia->thumbnail_date)
{
    $tempThumbPath = $tempMedia->getAbstractedTempThumbnailPath();
    $mediaThumbPath = $container->getAbstractedThumbnailPath();

    $tempFile = \XF\Util\File::copyAbstractedPathToTempFile($tempThumbPath);
    \XF\Util\File::copyFileToAbstractedPath($tempFile, $mediaThumbPath);

    $container->fastUpdate('thumbnail_date', $tempMedia->thumbnail_date);

    \XF\Util\File::deleteFromAbstractedPath($tempThumbPath);
}
if ($tempMedia->poster_date)
{
    $tempPosterPath =...

• Kirby
• Replies: 0
• Forum: Media Gallery bug reports

K

Thread 'FFMPeg handling is very inefficient'

Jul 9, 2022

\XFMG\Attachment\Media::onNewAttachment

    public function onNewAttachment(Attachment $attachment, \XF\FileWrapper $file)
    {
        /** @var \XFMG\Service\Media\TempCreator $tempCreator */
        $tempCreator = \XF::app()->service('XFMG:Media\TempCreator');
        $tempCreator->setAttachment($attachment);
        if ($file->getExif())
        {
            $tempCreator->setExif($file->getExif());
        }
        $tempCreator->save();
    }

When this is called, the attachment upload has been completed, the file has been copied to data and the original uploaded...

• Kirby
• Replies: 1
• Forum: Media Gallery bug reports

• 

The whole process of uploading a large video to XFMG in chunks works like this:

1. Upload a chunk to the server
2. Process the uploaded chunk and store it FlySystem.
   If an object stroage is used this includes uploading the chunkd to that storage
3. Continue at step 1 if there are further chunks
4. Reconstructing the original file by serially reading in all chunks
   If an object storage is used this includes serially downloading each chunk
5. Feed the original file back to XenForo upload handler
6. Process the uploaded original file
   If an object storage is used this includes synchronally uploading the original file to that storage
7. Creating thumbnail and poster (if enabled)
   This is done by copying the whole file from abstracted storage to a temporary local file
   If an object storage is used this includes synchronally downloading the original from that storage
8. Determining if transcoding is necessary (if not force)
   This is done by (yet again) copying the whole file from abstracted storage to a temporary local file
   If an object storage is used this includes synchronally downloading the original from that storag
9. Queeuing a transcode job (if enabled and necessary or forced)

Steps 1)-3) happen in one request, for large files there will be dozens of such requests.
Steps 4)-9) happen in the request for the last chunk together with steps 1)-3).

Now think about this (assuming FFMPeg and transcoding is enabled):
For a 20 GB file in 100 MB chunks stored in an object storage this means

• ~ 200 GET requests to sequentially download the chunks = 20 GB download in Step 4)
• One POST request for a 20 GB object = 20 GB upload in Step 6)
• One GET request for a 20 GB object = 20 GB download in Step 7)
• Maybe another GET request for a 20 GB object = another 20 GB download in step 8)

Ignoring the overhead for the requests, that's 20-60 GB download + 20 GB upload + processing time.

Even with fullspeed GBit (I'd assume most dedicated servers and VPS don't get more) just the downloads would take well > 2 minutes at least.

Tl;Dr
Without significant changes to XenForo and XFMG it is not really possible to upload videos that large, at least not when using object storage - you might be lucky and it doesn't timeout with fast local (eg. NVME) storage.

 

[stromb0li]

Appreciate the super indepth response. Luckily the storage is local so we don't have the overhead of having to go across the internet. Is there any way to run step 4-5 manually? Or is there a way to trick the upload form to point to an already uploaded file on the server itself?

If this is truly looking for a copy of the file that already exists based on hash, potentially I could upload it to the right directory and run this API call? https://my.domain.com/attachments/u..._id]=15&hash=90851c1a2d141b69297ba80295700833

 

[Seeker-Smith]

I have a about 350GB worth of 4k videos from a recent event our community held, where some files are be up to 20GB-25GB each. What is the best way to pre-upload them to the server and reference via XFMG?

There is a addon called "chunk" or soemthing like that. Ity will herlp with your uploads.

Edit; link

Thread 'Chunked Uploads - XF2 [Paid]'

Apr 17, 2019

JulianD submitted a new resource:

Chunked Uploads - XF2 - Allows you to upload very large files even if your server doesn't allow it

This addon enables "chunked uploads" in XenForo and allows you to set a virtually higher maximum upload limit up to 128GB regardless of your server configuration.

Features

• Option to define the chunk size of each upload. This value cannot be higher than the maximum upload file size reported by your server.
• Virtually increase the maximum attachment size up to 128GB (see limitations below)
• Integrates seamlessly...

• JulianD
• Replies: 97
• Forum: Add-on releases [2.x]

 

[stromb0li]

There is a addon called "chunk" or soemthing like that. Ity will herlp with your uploads.

Edit; link

Thread 'Chunked Uploads - XF2 [Paid]'

Apr 17, 2019

JulianD submitted a new resource:

Chunked Uploads - XF2 - Allows you to upload very large files even if your server doesn't allow it

This addon enables "chunked uploads" in XenForo and allows you to set a virtually higher maximum upload limit up to 128GB regardless of your server configuration.

Features

• Option to define the chunk size of each upload. This value cannot be higher than the maximum upload file size reported by your server.
• Virtually increase the maximum attachment size up to 128GB (see limitations below)
• Integrates seamlessly...

• JulianD
• Replies: 97
• Forum: Add-on releases [2.x]

I'm already using this.

 

[Seeker-Smith]

I'm already using this.

I feel your pain but one by one is really the only way.