What would be a reasonable server disk size for a big forum with SSL on

[brunoa]

Hi, I switched my forum to HTTPS recently, so I had to turn on image proxy, which means disk space requirements went up considerably. It's a large forum with over 15 million posts (it's been online for a long time, but the average number of posts daily is around 4k-5k), images are linked (and therefore cached by Xenforo now) but no file attachments are allowed.

Since we activated image proxy we've been struggling with disk space. We've increased the disk size by 3x, from 50 GB to 150 GB, but it's still full and I don't know how to predict the correct demand. I've decreased the image cache life time for 3 days but still not sure if it's gonna be enough.

Does anyone know what would be a reasonable disk size for a forum with this number of posts? Thanks!

 

[S Thomas]

How many pictures do you have? You can estimate that with some commands:

$ cd /path/to/internal_data/
$ find ./image_cache -type f | wc -l ;  du -hs ./image_cache/

This will output you something like

5058
489M ./image_cache/

So we roughly have 100KB / img. Knowing that ~90% of our pictures are ~10KB, this makes a real average of around 1MB / img for us.
Next you want to fire up this SQL query:

SELECT SUM(ROUND(   
        (CHAR_LENGTH(message)
            - CHAR_LENGTH(REPLACE(lower(message), "[img]", ""))
        ) / CHAR_LENGTH("[img]")       
    )) as imgs   
FROM xf_post

This might take a while for 15M posts (5s for me @500k Posts). Now, this number is the total amount of pictures in posts - duplicates are count aswell. It's not the most realistic number, but it should be good enough for an upper limit. Multiply this with an good-enough average img size and you should know what disk capacity you might want to look for.

 

[Jake B.]

You can also probably decrease the TTL for the image proxy cache, just use your AdminCP search bar to find "imageCacheTTL"

 

[brunoa]

How many pictures do you have? You can estimate that with some commands:

$ cd /path/to/internal_data/
$ find ./image_cache -type f | wc -l ;  du -hs ./image_cache/

This will output you something like

So we roughly have 100KB / img. Knowing that ~90% of our pictures are ~10KB, this makes a real average of around 1MB / img for us.
Next you want to fire up this SQL query:

SELECT SUM(ROUND(  
        (CHAR_LENGTH(message)
            - CHAR_LENGTH(REPLACE(lower(message), "[img]", ""))
        ) / CHAR_LENGTH("[img]")      
    )) as imgs  
FROM xf_post

This might take a while for 15M posts (5s for me @500k Posts). Now, this number is the total amount of pictures in posts - duplicates are count aswell. It's not the most realistic number, but it should be good enough for an upper limit. Multiply this with an good-enough average img size and you should know what disk capacity you might want to look for.

I've tried the first command you suggested, but since our disk is full it will only show what we have inside that folder and not the actual demand for disk space if all the images could be cached. I'll try the sql query now. Thanks for helping!

 

[brunoa]

You can also probably decrease the TTL for the image proxy cache, just use your AdminCP search bar to find "imageCacheTTL"

This is the image cache life time, right? Decreased it from 7 to 3 already, but not sure if it's gonna be enough. Thanks!

 

[S Thomas]

I've tried the first command you suggested, but since our disk is full it will only show what we have inside that folder and not the actual demand for disk space if all the images could be cached

Of course, but you can build an average from those values (size divided by #files) and multiply that with the # from the query.

 

[eva2000]

Since we activated image proxy we've been struggling with disk space. We've increased the disk size by 3x, from 50 GB to 150 GB, but it's still full and I don't know how to predict the correct demand. I've decreased the image cache life time for 3 days but still not sure if it's gonna be enough.

Maybe set proxy cache TTL to 1 day so you can get a daily tally of cache size on disk more easily for working out your per day image cache size required

 

[rdn]

Install this: https://xenforo.com/community/resources/imageproxy-host.5030/
And Ignore HTTPS Images.

 

[brunoa]

Maybe set proxy cache TTL to 1 day so you can get a daily tally of cache size on disk more easily for working out your per day image cache size required

Thanks! Was about to test this indeed.

 

[brunoa]

Install this: https://xenforo.com/community/resources/imageproxy-host.5030/
And Ignore HTTPS Images.

This plugin is the best solution. This should be a standard option on XF. Installed and trying it now. Thanks a lot!

 

[eva2000]

curious how much disk space you saved from just ignoring https images ?

 

[brunoa]

curious how much disk space you saved from just ignoring https images ?

Will tell you tomorrow or once the plugin is effective.

 

[brunoa]

curious how much disk space you saved from just ignoring https images ?

After a few days, image cache folder went from 120 GB to 77 GB.

 

[mkucuksari]

Install this: https://xenforo.com/community/resources/imageproxy-host.5030/
And Ignore HTTPS Images.

Does this addon work with XF-2 ?

 

[S Thomas]

Does this addon work with XF-2 ?

https://xenforo.com/community/resources/imageproxy-host.6185/

 

[mkucuksari]

@yoloswaggerino thanks for the link.

 

[eva2000]

After a few days, image cache folder went from 120 GB to 77 GB.

That's some savings !