Load testing of Xenforo, etc.

[craigiri]

Hello all,

My question has to do with how the developers or others test this and other forum software. It seems very difficult to predict actual forum activity or to use a basic linear scale to assume that "x" server setup can handle "x" users, each posting, looking, searching, etc.

This interests me both from a purely technical angle AND for actual consideration and purchase. As an example, my current forum software on a basic dedicated server hits a "wall" at about 20,000 visits (75K+ page views) in a day, whereas is handles 16,000 with ease. I should mention that the server is three years old (dual 2.5 2G Ram) and that the same site also serves some static and other content - but the forum is 90% of the load.

I know answers to this stuff are often dependent on many factors - but it would be interesting to have a suggested benchmark Big Board install which would at least be a rough reference.....i.e, "we, the developers, feel that a newer dedicated linux server with 2+G Ram and "y" processor could easily host a typical forum community which serves 200,000+ page views in a busy day".......or something of the sort!

Any hints, tips and links to other threads would be great! Often, we admins don't get to find out until it is too late!

 

[Xarcell]

maybe this helps a little: http://xenforo.com/community/threads/server-load-vb-xf-joomla-proof-is-everything.8073/

 

[Walter]

And maybe this thread gives you a bit confidence too:
http://xenforo.com/community/thread...-vbulletin-forum-with-2-8-million-posts.7003/

In general, XenForo seems to be lighter than e.g. vBulletin.

 

[craigiri]

Thanks! I had seen that thread and also had assumed it would be lower in overhead than VB...all good things! However, that data is closer to "here is one experience" than something from the developers. I guess I was hoping to hear about how they actually stress test to see that they have achieved their goals...as opposed to "joe says it is fast", etc.

There is a "wall" somewhere when it comes to this stuff, and it would be nice to know where it was located more accurately. I suppose as more members here adopt it, we might be able to compare more exact experiences. Still, it would be nice to hear from the developers that "we are confident that Xenforo can do this on this machine"......even a conservative measure. Of course, I'd only want to know that if it was more than a guess!

Thanks again!

 

[craigiri]

I'm going to bump this thread one more time in the hopes of hearing either from the developers or from someone else who is running XF with a heavy load and knows what it can handle.

I am convinced that speed and low overhead is going to be a BIG selling point with this and other modern forum software.....user behavior has changed vastly, and connections have gotten quicker.....also, users are expecting more and more. Here is one example from today....

I had a forum user complain that he hit the "throttle" limit I have set on my forums - which is currently set to allow loading of no more than 8 pages in 12 seconds. If you do so, you get a little message telling you you are moving too fast....and asking if you were actually reading the pages (just an attempt to make them slow down a bit)....and you have to wait 10 seconds. I thought that was a reasonable amount of potential bandwidth to give each user, but apparently not!

It turns out he...and probably others....use the tabs feature in their browser to open up 10 or more thread pages at a time, and then read them! As you can imagine, having hundreds of users...or even a couple dozen...doing that at the exact same time, could really call for a lot of server resources!

I know I can always throw hardware at a problem - load balance, move the db to another box, etc.....but I am getting tired of making up for outdated code that way - and doubly so when we can only imagine how fast users will suck down data in the coming years!

So, developers, friends, coders and countrymen.....someone devise a real world stress test or put XF on a site which really get's hammered, and let's find out where the limit is. Hey, we will find it sooner or later anyway!


I took a look at those facepunch forums and they definitely have a big load - but it seems to be running on multiple servers......and VB, so I can't gleam much from that.

Oh, edit - just found a thread where some basic advice and benchmarks (guesses) are being given for plain vanilla VB.....
http://www.webhostingtalk.com/showthread.php?t=909746

Seems that the admins there are saying that an older dedicated server can handle about 1,000 online users and a more expensive and faster server maybe 2,000 or more.

That's rough - but still a good example of data that has some use!

 

[Decent60]

Maybe Kier or Mike can give some examples (server specs/usage) from the load of this site. I know it must have a huge load on it and this is the same software that we get to download and use for our XenForo.

 

[craigiri]

Decent, according to the shown stats, this is a relatively tiny load - maybe 150 users max.
To really test it, we'd need either a real world situation with a forum with 700+ users online hitting pretty hard (the more, the merrier) - and also a large number of posts and members (making the dbs big), as well as some other stuff which Kier and Mike know much more about that I do. I suspect we could do some of this by using a script of sorts - like a denial of service attack, but friendly (that is, 100 or more of us download a little program which we run for an hour or so....the program grabs full page loads of various types randomly, etc.)

But real world does all that for us.......
What I see so far makes me very confident - but software is funny stuff. I can work perfectly until a certain point, then go downhill almost instantly after that.

 

[Dean]

It turns out he...and probably others....use the tabs feature in their browser to open up 10 or more thread pages at a time, and then read them! As you can imagine, having hundreds of users...or even a couple dozen...doing that at the exact same time, could really call for a lot of server resources!

I do that for 2 reasons.

• because the forum is slow, usually because of ads or other external data being served up, or
• because the forum is using a cookie timeout - if after a certain amount of time not clicking on the forum the forum is automatically marked as read, then if I need to go do something for a few minutes then I've lost track of all unread data.

You could use a variety of tools to measure site responsiveness & number of queries, here is one that tests from a variety of locations. Please note it is using IE7 as the test browser, so it is not really representative of actual speed/query count. If people have upgraded from IE6 then they will most probably be using IE8 or another browser completely. http://www.webpagetest.org/

And I will send you a PM with a test site that can load test for you, I certainly don't want to be the person posting it on this forum.

 

[craigiri]

Thanks - it's not I that needs the test - it's this site or someone who volunteers that has XF on a dedicated box.
Cheap enough, that's for sure!
I think those paid tests (mentioned in other threads) would be quite valid when compared to another forum on the same server as a reference. That is, if you installed XF on server A and ran the suite of tests with 1000 simulated users and then installed another piece of Forum SW on the same server and ran the test again.

I suspect we will eventually get such data from users and from the development team - obvious it is a big feather in their cap to blow the doors off of other forums software, so who wouldn't want to prove it?

 

[Dean]

You have the right idea. Dedicated box doing nothing else would be best. In this case an uber cheap low budget slow server might be better than other faster servers.

 

[Dean]

Actually I own vb3 vb4 xf and ipb... so possibly I could rent one of those cheap servers some day and compare them, but have no idea when that would happen. I could make an educated guess without doing that though. xf is extremely zippy.

 

[ckwalsh]

I actually have been doing forum software benchmarks for a while, and just redid my test suite/site at http://bb-bench.com/, including xenforo in the results. It's not perfect, with limited results, but for what it does measure, I believe the results are fair and important. I tried to be thorough in documenting the testing, but I'd love feedback.

 

[craigiri]

Both great info - Dean, if you end up needing a bit of financing to do some comparisons, I will add to the collection plate - as I'm sure others here would probably do also......

ck, I'll check it out. Thanks

 

[craigiri]

ck, great stuff.
As you mentioned, it doesn't show everything - my biggest beef is that it was run on a cloud, which is something that few of us do in the real world.

Were the requests sent in from the same cloud, or were they sent in from afar?

Thanks so much for doing this!

Other questions would be to somehow compare your request numbers to actual number of users - for instance, does this load simulate any scenario - for instance, perhaps 1000 "real world" users on a normal board? Or 2,000, etc.

Again, thanks for this - if nothing else it shows that many boards can do the job well, especially when tuned and not loaded up with plug-ins and add-ons. SMF 2 appears to do a heck of a job!

I suppose XF is not yet fully optimized in the code because they are working so hard on it. I'm not a programmer, but it does seem that code being modified often is often not just fully speed optimized....until the "gold" release.

Oh, another big deal is probably to test the "inserts" and "alters" on such forums.....IMHO, they provide a lot of load because pages have to be refreshed, etc.

Please let us know if you need some support - financial or others (cheering) to continue to do this type of testing.

 

[ckwalsh]

my biggest beef is that it was run on a cloud, which is something that few of us do in the real world.

I don't see how that is an issue. The application host may be virtual, but it was given a set amount of resources (aka, wasn't auto scaling as cloud computing advertises)

Were the requests sent in from the same cloud, or were they sent in from afar?

The requests were sent from the same machine they were being processed on (so essentially no network latency) and the database server was in the same availability zone as the application host (the least amount of latency you can get on AWS)

Other questions would be to somehow compare your request numbers to actual number of users - for instance, does this load simulate any scenario - for instance, perhaps 1000 "real world" users on a normal board? Or 2,000, etc.

That's really tricky, as it also depends on the behavior of your users. 2000 users that are reading long winded topics will act differently than 1000 users all participating in a flamewar, which will be different from a bunch of traffic from digg/reddit/etc.

I suppose XF is not yet fully optimized in the code because they are working so hard on it. I'm not a programmer, but it does seem that code being modified often is often not just fully speed optimized....until the "gold" release.

Personally, I think the usage of Zend Framework is the main difference. It's great for getting an application up and running, but is design to work with all applications, and is not tuned to the needs of a particular one.

Oh, another big deal is probably to test the "inserts" and "alters" on such forums.....IMHO, they provide a lot of load because pages have to be refreshed, etc.

Yes, I'm working on basher (I really need to think of a better name) to support posting a new topic, searching, logging in, etc, but it's not a simple task. There are a few tools that can do it, but they are primarily designed for automated testing of functionality, not benchmarking, and their performance is subpar for that purpose.

Please let us know if you need some support - financial or others (cheering) to continue to do this type of testing.

Cheering is nice, I need to put up some Google ads on there to make a little money. If anyone has any AWS credit vouchers they aren't planning on using, I would love to take them off your hands, but I won't be posting a paypal email.

To the Devs: I wouldn't be surprised if there are some settings that would increase the performance of XenForo, but not knowing the codebase I have no clue what they are. If you can point me to them, I would be more than willing to re-run the tests. I would ask in the customer forum, but this account isn't tied to the license I'm using (all of the software I'm using is properly licensed; I'm not a jerk)

 

[Paul B]

I would ask in the customer forum, but this account isn't tied to the license I'm using (all of the software I'm using is properly licensed; I'm not a jerk)

You can add up to 4 forum accounts to 1 license/customer account

 

[Dean]

Both great info - Dean, if you end up needing a bit of financing to do some comparisons, I will add to the collection plate - as I'm sure others here would probably do also......

Problem is, I am eminently unqualified to design a meaningful test, not the cost. I've played with different things, to me, these types of threads are much more meaningful
http://xenforo.com/community/threads/server-load-vb-xf-joomla-proof-is-everything.8073/

 

[ckwalsh]

You can add up to 4 forum accounts to 1 license/customer account

I will certainly prod the purchaser. Thanks!

Problem is, I am eminently unqualified to design a meaningful test, not the cost. I've played with different things, to me, these types of threads are much more meaningful
http://xenforo.com/community/threads/server-load-vb-xf-joomla-proof-is-everything.8073/

I wouldn't say "more meaningful", but different. All sites are different. That is a specific anecdote of how his site reacted, but that change could be due to fewer pageviews for some reason or another, fewer logins, or some other factor. The tests I've done are limited in that I don't logged in user activity (but I'm working on it) and am only interacting 3 ways with the board (which is not how normal users interact). However, they are much more precise and reproducible as all factors have been controlled, and can be performed on many more boards (imagine rotating installations every day, just to see how each performed; users will kill you!).