Preparing for a large site...

[ibinod]

Hello..
i've started a xf based site and with just 1k users i can see the database growing crazy.. it's currently at 200mb but i can see the database growing huge as the site grows..
how do i prepare for this..

currently it's so easy to backup the database and restore it if something goes wrong..

but it won't be the same once it starts increasing gigs..

also how do i prepare on the hosting end..

currently i'm in dedicated server with lemp environment.

 

[sbj]

Database growing crazy? You should worry about it if you ever reach huge numbers of users, say 100k. It would take you years to reach that, if at all.
Secondly, the databse would be how big for 100k users? Probably 3-5 GB.

What makes a problem are the attachments and video uploads. If your forum is heavily image/video oriented, then you can have a problem. Only images for 100k users would be something between 10-50 GB estimated.
Video uploads are a problem on their own.

For someone with a dedicated server, those numbers are ridiculously low.

What can be done, if you neeeded more? Well, get more storage of course. Your hosting company should be able to help you out for a fair price. But correct me if I am wrong. I believe you are in the price range of 25-50$ right now for a dedicated. You should have 50-100GB storage right now at least. So nothing to worry about.

If that not helps, see.

Using DigitalOcean Spaces or Amazon S3 for file storage in XF 2.1+

XenForo Version Compatibility The download is only compatible with XenForo 2.1 and above. Why this guide? Since XenForo 2.0.0 we have supported remote file storage using an abstracted file system named Flysystem. It's called an abstracted file...

xenforo.com

And/Or use one of the two image optimizer addons to reduce the load.

 

[ibinod]

Database growing crazy? You should worry about it if you ever reach huge numbers of users, say 100k. It would take you years to reach that, if at all.
Secondly, the databse would be how big for 100k users? Probably 3-5 GB.

I think you shouldn't jump to a conclusion without knowing a person's background.. i deal with sites that receives millions of visitors per day..
the forum I started 2 days ago already has gone to 3k visitors and 1k+ members..



and I'm someone who likes to plan for the future...

What makes a problem are the attachments and video uploads. If your forum is heavily image/video oriented, then you can have a problem. Only images for 100k users would be something between 10-50 GB estimated.
Video uploads are a problem on their own.

For someone with a dedicated server, those numbers are ridiculously low.

this is not a problem at all and the question was related to database... because i deal with big wordpress sites and with some buggy themes..
database is always a problem..

What can be done, if you neeeded more? Well, get more storage of course. Your hosting company should be able to help you out for a fair price. But correct me if I am wrong. I believe you are in the price range of 25-50$ right now for a dedicated. You should have 50-100GB storage right now at least. So nothing to worry about.

How old are you? how can you believe what I have and make estimation? did i specify those numbers..?

i'm someone who owns servers like these



if you know what that means...

I was asking a genuine question and a solution.. not someone's opinion with assumption with what i have and what i don't have.

 

[sbj]

i deal with sites that receives millions of visitors per day..
the forum I started 2 days ago already has gone to 3k visitors and 1k+ members..

So you deal with sites with millions of visitors per day, and yet here you come and ask the most basic intro question.
Sorry, but you were the one with "I have 1k members and the database is 2mb right now, what should I do".
I can't see your future or your situation to assume that you deal with millions of users per day. If that is true, then all scenarios should be trivially known for you...

How old are you? how can you believe what I have and make estimation? did i specify those numbers..?

Did I say you specificed those numbers? Do you have a problem with reading? I estimated those numbers. You know what estimation means? I know more or less the hosting server specs for dedicated solutions, so, I estimated. Big boy has a Dual Xeon... wow... impressed... How old are you again?

I was asking a genuine question and a solution.. not someone's opinion with assumption with what i have and what i don't have.

And I genuinely tried to answer your question. I can decide for myself how I will answer any question. Don't need your childish attitude. What a joke.

 

[ichpen]

Hello..
i've started a xf based site and with just 1k users i can see the database growing crazy.. it's currently at 2mb but i can see the database growing huge as the site grows..
how do i prepare for this..

currently it's so easy to backup the database and restore it if something goes wrong..

but it won't be the same once it starts increasing gigs..

also how do i prepare on the hosting end..

currently i'm in dedicated server with lemp environment.

What SLA are you aiming for? What are you actually wanting to prepare for exactly? What's your budget? What levels if any of redundancy do you need? Not sure I understand your ask here.

 

[WSWD]

currently it's so easy to backup the database and restore it if something goes wrong..

but it won't be the same once it starts increasing gigs..

Why is that? Restoring a large database is just as easy as restoring a small one, for the most part. It just takes more time.

The biggest limitation on hosting servers is almost always Disk I/O and the web server. You conquer that by using SSD drives, and Litespeed or Nginx or such for the web server. When you get to millions of users, it is quite possible to take more than one server, for load balancing.

 

[Bimble]

Our site backup is 67GB - hosted in the same data centre so it can be quickly and easily recovered to our NVMe equipped server; we have 2TB of backup history we can refer to in case of corrupted backups or config changes where we don't know exactly what date the impacting change/s were made.

From your second post though you appear to already have all the experience and resources you need to cover your XF installation.

 

[eva2000]

Our site backup is 67GB - hosted in the same data centre so it can be quickly and easily recovered to our NVMe equipped server; we have 2TB of backup history we can refer to in case of corrupted backups or config changes where we don't know exactly what date the impacting change/s were made.

From your second post though you appear to already have all the experience and resources you need to cover your XF installation.

If space is a concern, look at better compression algorithms to compress your backups. I benchmarked and compared some that I use and the difference in compressed backup sizes can be dramatic https://community.centminmod.com/th...d-vs-brotli-vs-pigz-vs-bzip2-vs-xz-etc.17259/

 

[Bimble]

Thanks @eva2000 - it's not an issue for us, we;ve got plenty of space - I was just quoting it as an example for the OP; but on reading their replies I can see they know what they're doing so will leave them to work it out for themselves.

 

[sbj]

Our site backup is 67GB -

How much of that database and how much of that attachments for how many posts and members, If I may ask. Want to see if my estimation was in the right direction or not.

 

[Bimble]

Couldn't tell you without looking in-depth but backup includes whole of home/ directory + SQL database files; I'd hazard a guess at around 3GB for mail, 16GB for database, and the bulk of the rest is likely to be attachments (since moving to https we've used an add-on to make all linked images attachments). Members 70k+; threads 300k+; posts several millions.

 

[Walter]

@ibinod One way to keep the database lean is to use Enhanced Search for Xenforo (official addon). Without it, the biggest table usually is not the post table but the search table.