What would be a reasonable server disk size for a big forum with SSL on

brunoa

Member
Hi, I switched my forum to HTTPS recently, so I had to turn on image proxy, which means disk space requirements went up considerably. It's a large forum with over 15 million posts (it's been online for a long time, but the average number of posts daily is around 4k-5k), images are linked (and therefore cached by Xenforo now) but no file attachments are allowed.

Since we activated image proxy we've been struggling with disk space. We've increased the disk size by 3x, from 50 GB to 150 GB, but it's still full and I don't know how to predict the correct demand. I've decreased the image cache life time for 3 days but still not sure if it's gonna be enough.

Does anyone know what would be a reasonable disk size for a forum with this number of posts? Thanks!
 

S Thomas

Well-known member
How many pictures do you have? You can estimate that with some commands:
Code:
$ cd /path/to/internal_data/
$ find ./image_cache -type f | wc -l ;  du -hs ./image_cache/
This will output you something like
5058
489M ./image_cache/
So we roughly have 100KB / img. Knowing that ~90% of our pictures are ~10KB, this makes a real average of around 1MB / img for us.
Next you want to fire up this SQL query:
SQL:
SELECT SUM(ROUND(   
        (CHAR_LENGTH(message)
            - CHAR_LENGTH(REPLACE(lower(message), "[img]", ""))
        ) / CHAR_LENGTH("[img]")       
    )) as imgs   
FROM xf_post
This might take a while for 15M posts (5s for me @500k Posts). Now, this number is the total amount of pictures in posts - duplicates are count aswell. It's not the most realistic number, but it should be good enough for an upper limit. Multiply this with an good-enough average img size and you should know what disk capacity you might want to look for.
 

Jake B.

Well-known member
You can also probably decrease the TTL for the image proxy cache, just use your AdminCP search bar to find "imageCacheTTL"
 

brunoa

Member
How many pictures do you have? You can estimate that with some commands:
Code:
$ cd /path/to/internal_data/
$ find ./image_cache -type f | wc -l ;  du -hs ./image_cache/
This will output you something like

So we roughly have 100KB / img. Knowing that ~90% of our pictures are ~10KB, this makes a real average of around 1MB / img for us.
Next you want to fire up this SQL query:
SQL:
SELECT SUM(ROUND(  
        (CHAR_LENGTH(message)
            - CHAR_LENGTH(REPLACE(lower(message), "[img]", ""))
        ) / CHAR_LENGTH("[img]")      
    )) as imgs  
FROM xf_post
This might take a while for 15M posts (5s for me @500k Posts). Now, this number is the total amount of pictures in posts - duplicates are count aswell. It's not the most realistic number, but it should be good enough for an upper limit. Multiply this with an good-enough average img size and you should know what disk capacity you might want to look for.
I've tried the first command you suggested, but since our disk is full it will only show what we have inside that folder and not the actual demand for disk space if all the images could be cached. I'll try the sql query now. Thanks for helping!
 

brunoa

Member
You can also probably decrease the TTL for the image proxy cache, just use your AdminCP search bar to find "imageCacheTTL"
This is the image cache life time, right? Decreased it from 7 to 3 already, but not sure if it's gonna be enough. Thanks!
 

S Thomas

Well-known member
I've tried the first command you suggested, but since our disk is full it will only show what we have inside that folder and not the actual demand for disk space if all the images could be cached
Of course, but you can build an average from those values (size divided by #files) and multiply that with the # from the query.
 

eva2000

Well-known member
Since we activated image proxy we've been struggling with disk space. We've increased the disk size by 3x, from 50 GB to 150 GB, but it's still full and I don't know how to predict the correct demand. I've decreased the image cache life time for 3 days but still not sure if it's gonna be enough.
Maybe set proxy cache TTL to 1 day so you can get a daily tally of cache size on disk more easily for working out your per day image cache size required
 
Top