MG 2.2 How well does MG scale in terms of user-uploaded videos?

[Arnox]

Is it built to handle searching through and serving tens of thousands of videos without issue? Provided there was infinite storage, could MG theoretically handle YouTube levels of videos (~800 million videos on YouTube's servers as of this writing)?

 

[Arnox]

Apparently we're all laughing at this **** at the moment and not contributing anything, but I know a fair few people who would be very interested in such scaling, and the XF team has gone WAY above and beyond before with their software, so I would still really like an answer to this, preferably from one of the XF team themselves since they would obviously know the software's limits better than anyone else.

And besides. I think people are forgetting that the XF team makes software for enterprise use, not just personal use. So, in summary, I really don't know what the **** is so funny. To laugh at this question is, ironically enough, pretty much laughing at the entire XF team themselves and their capabilities.

 

[digitalpoint]

If you scaled it properly and built the proper infrastructure around it, it could do it. If you were handling YouTube loads, you would need more resources than a single server. The bottleneck isn't going to be how XenForo manages those videos, rather the backend resources needed to serve that much bandwidth and encode that much video as it's being uploaded. If you want to scale to YouTube size (YouTube gets around 720,000 hours of new video uploaded every day). You would need storage space, you would need many thousands of servers just to encode that much video being uploaded.

Again, the issue isn't what XenForo does (just manage the videos), the issue is the server resources to serve/process the underlying videos:

• You would need to encode 720,000 hours of videos being uploaded every day.
• You would need the bandwidth to serve up 4,500,000 hours of video being served every hour. So let's say an average of 4.5M concurrent video streams on average.

Once you have the server resources for that, anything you need XenForo to do on that infrastructure will be easy peazy.

The point is that if you have the server resources to serve and process videos, you have more than enough server resources for XenForo's part.

 

[briansol]

On top of all that,

youtube has extensive machine learning capabilities running in the background to create search indexes and related content. xfmg is incredibly simple, in that, it might look at the title of other text in a full text search concept.

The background architecture is surely managed by tools like Kafka and Airflow to orchestrate workflows, create and listen to data pipelines, and probably 10000s of other advanced toolings that extend well beyond a front end website running a single mysql instance.

YT has 10,000 employees.
xfmg was built by 1 as a side project and is now maintained by 3? 4 as an addon.


It simply can't compare. I'm sure it could be extended to scale, build out a more micro-service based architecture to process workloads at scale, and so forth with the API, but as it sits as a monolith, it doesn't have the capacity to scale itself. you'd need to throw multiple full instances of it behind a load balancer... ya know, what the enterprise did before microservice and lambda architectures. So, if you want 20-year old patterns, it can work, in theory. but back up the $$$ truck to pay for that non-ideal design pattern.

 

[VersoBit]

Honestly with our experience in XFMG relating to video uploading - while the platform works there is A LOT to be desired for customization:

• Controlling FFMPEG Threads
• Set Codecs, bitrates, resolutions, etc
• Ability to define "versions" of videos during encode output for dynamic resolution selection in playback
• Better Player Controls
• Video Upload Capped at 4GB (hard limit - this is regardless of your php/server environments "upload limit")
• Player has no resolution controls (so no dynamic bitrates)

You'd also be on the hook for implementing a DASH or HLS server to handle all the playback of your uploaded content, and frankly for the average user this is simply too much work for the scale you're looking at.

Typically one video will end up using around 70mbps (in bursts, when using HLS/DASH) with a 6k bitrate file at 1080p60. Being as these bursts typically happen in a waterfall fashion (one after another) say you had a few hundred users online and streaming. You're likely to saturate your entire i/o and bandwidth budget on just serving that pre-converted video.

It's also worth pointing out that scaling XenForo out horizontally is rather tough and keeping things synced will be a real ballache too.

In cases like this where Video is not the #1 source for uploaded content in XFMG, I could easily justify spending on Cloudflare Stream to accomplish my video upload/playback goals. Their solution is relatively cost effective when you consider that you'd be getting a VAST ready player and hosting/playback of your files for users (including your DRM).

XFMG is mature in a lot of ways, but also immature in a lot of ways too. YMMV.

 

[Arnox]

Thank you all for the responses! They are very appreciated. <3

If you scaled it properly and built the proper infrastructure around it, it could do it. If you were handling YouTube loads, you would need more resources than a single server. The bottleneck isn't going to be how XenForo manages those videos, rather the backend resources needed to serve that much bandwidth and encode that much video as it's being uploaded. If you want to scale to YouTube size (YouTube gets around 720,000 hours of new video uploaded every day). You would need storage space, you would need many thousands of servers just to encode that much video being uploaded.

Again, the issue isn't what XenForo does (just manage the videos), the issue is the server resources to serve/process the underlying videos:

• You would need to encode 720,000 hours of videos being uploaded every day.
• You would need the bandwidth to serve up 4,500,000 hours of video being served every hour. So let's say an average of 4.5M concurrent video streams on average.

Once you have the server resources for that, anything you need XenForo to do on that infrastructure will be easy peazy.

The point is that if you have the server resources to serve and process videos, you have more than enough server resources for XenForo's part.

Well, yeah. I don't think anyone expected XenForo to handle YouTube levels of activity on a Pentium 4. lol But yes.

On top of all that,

youtube has extensive machine learning capabilities running in the background to create search indexes and related content. xfmg is incredibly simple, in that, it might look at the title of other text in a full text search concept.

Keep in mind, XF already has a professional searching solution for sale on top of the regular XF search that fully utilizes ElasticSearch for its backend. Now, does it compare to YouTube's searching tools? Eh, maybe. Maybe not. But I will say this. YouTube has been caught many, many, MANY times promoting massive channels like Fox News over smaller channels just because. YouTube will also prioritize more recent and less relevant videos over videos that may be older but are more relevant and/or are more comprehensive. So YouTube search is far from this perfect untouchable thing.

Ability to define "versions" of videos during encode output for dynamic resolution selection in playback

Yeah, this is a feature that it seems only very few video hosters have that I would love for XF to have. With that said though, it would probably consume much more server resources. Much lighter and easier I imagine to serve just one version of a video as is and leave it at that.

It's also worth pointing out that scaling XenForo out horizontally is rather tough and keeping things synced will be a real ballache too.

Fair enough, but do we know that the XF team hasn't already done this?