Self edge caching content like Page rules cloudflare

[ShinLim]

I guess most of us are already use Page rules of cloudflare, I really like it because it can cache everything like images, css, js, etc.. (even video files, which actually is not allowed)
Currently I allow member upload videos in posts, up to 100mb in size, and every thread video getting viewed around 3000~5000 views per day, estimate that 10~20 videos upload per day. So the bandwidth is consuming around 2~3 TB per day
Is there any way to cache these files like cloudflare's page rules does or anyway to solve this with minimum cost (even no cost at all) ? Can it be cached and served from own server to save bandwidth ?
I'm so stressful right now

 

[Xon]

Cloudflare doesn't want to-do video caching like that (at least not without you paying them a lot).

You are likely going to need OVH or similar which offer fixed size but unlimited traffic with physical hardware rental. Virtual machines, and cloud stuff will be hilariously expensive or complete non-starters with provided quota

I have very little experience in setting up this sort solution, and I'm just aware of the pitfalls you can hit on the way.

 

[eva2000]

Can it be cached and served from own server to save bandwidth ?

Technically you could do nginx reverse proxy caching with byte-range caching and nginx slice module. But bandwidth costs can still be high for own hosted server if you're using gigabits per second in 10+ Gbps. You could be looking at 4 figure monthly own server hosting costs. But not sure how you'd do it integrated with xenforo member uploaded attachments with modifying xenforo ? Not something I've done as I don't allow video uploads and limit attachments to just 1MB in size.

see Nginx proxy cache and byte-range caching.

• Smart and Efficient Byte-Range Caching with NGINX
• A Guide to Caching with NGINX and NGINX Plus - NGINX
• NGINX Docs | NGINX Content Caching
• Module ngx_http_proxy_module
• Module ngx_http_slice_module

One of my Centmin Mod LEMP stack users hired me to setup Nginx proxy cache with sliced byte-range caching for his video streaming site (non-xenforo) but he had a budget of US$1,250+ per month to do this and required >10Gbps network connectivity https://community.centminmod.com/threads/help-with-a-solution-for-video-streaming.15538/

example of webm video nginx proxy cached slice byte-range caching at

• https://community.centminmod.com/threads/help-with-a-solution-for-video-streaming.15538/#post-66602
• and https://community.centminmod.com/th...n-for-video-streaming.15538/page-2#post-67218

Nginx vhost traffic stats reporting via Centmin Mod Nginx vhost traffic module for cache hits for nginx sliced byte-range caching of mp4 & webm videos



added conditional headers to track which sliced byte chunks are sliced = 1 and sliced-cache = HIT or MISS. It will only be from cache if you skip back or forth to point in a video's playback the first time and then cached (cache HIT) for subsequent requests for however long you configured cache expiry and cache inactive time. After cache size reached or expiry/inactive time is reached, the cache is cleared so again first time request is cache = MISS

 

[Xon]

But not sure how you'd do it integrated with xenforo member uploaded attachments with modifying xenforo ?

Attachment Improvements By Xon - Updates

xenforo.com

• Fix XFMG support (ie content stored in data rather than internal_data)
• Support for video stream via partial content requests
• Add "Partial content maximum chunk size" option

Already added support for slice-based Range header support

x-accel-redirect support also works with the Range header too.

 

[eva2000]

Attachment Improvements By Xon - Updates

xenforo.com

• Fix XFMG support (ie content stored in data rather than internal_data)
• Support for video stream via partial content requests
• Add "Partial content maximum chunk size" option

Already added support for slice-based Range header support

x-accel-redirect support also works with the Range header too.

sweet - love your work

 

[ShinLim]

Actually $1000 is enough for me with 100 TB per month, but it is not what I looking for or even $500 because I don't make that much to spend it
I only can spend $30~$50 per month, maximum $100, this issue is really tough for me

 

[eva2000]

I only can spend $30~$50 per month, maximum $100, this issue is really tough for me

If you can't afford the bandwidth costs, then only option is to stop serving videos or start charging members a $$ monthly fee for privilege of uploading videos i.e. a premium member user upgrade etc

 

[Ozzy47]

If you can't afford the bandwidth costs, then only option is to stop serving videos or start charging members a $$ monthly fee for privilege of uploading videos i.e. a premium member user upgrade etc

I’ve heard this before

Unfortunately if you can't afford it, then stop serving videos or start charging your members a fee for special uploading video privileges.

 

[RobinHood]

Or figure out a way to serve video ads before your videos, so you can at least monetise every play instance.

I mentioned that a bit in this transcoding suggestion here.

Q: But Robin, how are we supposed to pay for all this fancy extra transcoding?

A: I'm glad you asked, as someone who is experienced in extracting money from the rich, and giving it to the poor, we must apply the same principles to our video sharing endeavours.

We make the premium video features only available to the rich VIP, Premium, Platinum, Supporting, whatever you want to call them elite members of our websites, and take their cash in order to pay for all these fancy features and offer a basic core experience to everyone.

More and more high end phones are offering incredibly premium video options, including 4k and high FPS (slow motion) video options. These formats are very data heavy and will require more processing power to crunch.

One option is to enable 720 and 1080 uploading to everyone, but restrict 4k and high fps footage to those in premium user groups, to help subsidise the cost of the transcoding. Or some variation of that.


Another option is to offer pre and post roll video ads.

At its simplest form, we could have admin options to upload a short (5-30 second) video from a site sponsor (or our own ad) to play before, after, or both on any videos, tracking view counts in site stats.

This way we could enable a 'View Videos' permission for guests and sell some pre-roll ad spots to site sponsors, or just run our own ad to encourage users to sign up after watching.

We could even add a custom CTA link or button below the video from the sponsor or ourselves, definable from within the ACP.

The current guest attachment situation for videos is not great, you get presented with a simple text link, and cannot view it unless you register.

View attachment 194967

There's no pretty thumbnail, no big play button. And even if you're already a member and you log in, it doesn't always automatically play once logged in (at least on a mobile test I did a while back)

I imagine many guests will not bother to watch certain videos if linked to a website they've never heard of if they have to register to do so. However enabling the viewing of videos for guests, but with a pre or post roll ad we can get revenue from site sponsors to run their ads, making your forums thread links and videos much more desirable links to share, helping drive traffic to your site. (Thus allowing us to buy more and more XF licences )

If guests discover high quality content within that video, it may prove a highly effective driver to get them to become fully registered members in order to engage with the video thread and content creator, perhaps even converting to paying supporting members down the line.

I'm sure there are probably other ways of monetising this kind of content to help make it an affordable and competitive feature.

Video is becoming more and more important as a content type that community members want to share. We need to be able to monetise it where possible to pay for the large additional associated costs .

 

[ShinLim]

Or figure out a way to serve video ads before your videos, so you can at least monetise every play instance.

I mentioned that a bit in this transcoding suggestion here.



Video is becoming more and more important as a content type that community members want to share. We need to be able to monetise it where possible to pay for the large additional associated costs .

Put ads into video is a good idea, but i dont know how to put ads code into xenforo's video attachment. Do you know any tutorial for this stuff?

 

[Sam9]

I already showed you a couple of methods you can use to do this within your budget in your other thread.

1. If you want to continue using a S3 compatible service:
   1. Use a cheaper alternative such as Backblaze B2 or Wasabi instead of DigitalOcean
   2. Get a server from one of the companies I listed in the other thread
   3. Setup nginx on said server, to pull and locally cache from the S3 backend
   4. Update your Xenforo config to point the attachment hostname to the hostname of the above server
2. If you want to cut out the extra S3 cost:
   1. Get a server from one of the companies I listed in the other thread
   2. Install nginx and a FTPd
   3. Configure Xenforo to upload attachments to the server via FTP, using the built Flysystem, and update the hostname

#1 is a bit more expensive but has the added benefit of everything being backed up on S3.

Make sure you disable Cloudflare's caching for the attachment server hostname, otherwise Cloudflare will ban you again.

 

[ShinLim]

I already showed you a couple of methods you can use to do this within your budget in your other thread.

1. If you want to continue using a S3 compatible service:
   1. Use a cheaper alternative such as Backblaze B2 or Wasabi instead of DigitalOcean
   2. Get a server from one of the companies I listed in the other thread
   3. Setup nginx on said server, to pull and locally cache from the S3 backend
   4. Update your Xenforo config to point the attachment hostname to the hostname of the above server
2. If you want to cut out the extra S3 cost:
   1. Get a server from one of the companies I listed in the other thread
   2. Install nginx and a FTPd
   3. Configure Xenforo to upload attachments to the server via FTP, using the built Flysystem, and update the hostname

#1 is a bit more expensive but has the added benefit of everything being backed up on S3.

Make sure you disable Cloudflare's caching for the attachment server hostname, otherwise Cloudflare will ban you again.

First I want to say thank you, I'm really appreciate.
But there's some steps config I don't know how to do, like :

• Setup nginx on said server, to pull and locally cache from the S3 backend

How to setup nginx to pull and locally cache from S3 backend ?
And how to:

• Update your Xenforo config to point the attachment hostname to the hostname of the above server ?

And last question is what estimate cost for this bandwidth :

 

[Sam9]

How to setup nginx to pull and locally cache from S3 backend ?

You can do this with proxy_pass, here's a detailed answer on stackoverflow, although it uses AWS instead of Backblaze B2 or Wasabi, the premise is near identical - https://stackoverflow.com/questions/44639182/nginx-proxy-amazon-s3-resources/44749584#44749584

And last question is what estimate cost for this bandwidth :

So you're using about 100TB per month? That'll be doable on a single 12 EUR server from online.net. Bear in mind bandwidth quality isn't great, but beggars can't be choosers. You could also do it on a single Hetzner server, which has better quality bandwidth, but would be 2-3x more expensive, but would also come with multiple and larger SSDs.

Also note, these are single servers based in Europe, your content won't be cached around the world like with Cloudflare or an actual CDN, so speeds for EU users will be fine, but it won't be as fast for other locations, and potentially heavily impacted for remote locations such as Asia, Oceania, and west coast USA.