Web framework benchmarks

[ekine]

Anybody who is interested in web framework benchmark should take a look at this benchmark overview. There is also an active discussion going on about the article.

What do you think about it?

 

[SilverCircle]

No surprise here. JVM technologies came out on top, but this was to be expected (Java = slow is an urban myth and was partially true 15 years ago, but no longer).

It also shows that *any* framework adds a huge overhead, even relatively lightweight ones like wicket (and just compare raw php vs php+cake). Interesting to see that Java + a fat framework can still be faster than raw php.

One would argue that these tests have been made without PHP opcode caches (though the exact configuration is nowhere stated). Opcode caches don't improve execution speed (it's still interpreted bytecode), but they can cut off a large part of the compilation overhead, especially when lots of code is involved, like it is usually the case with frameworks.

Also, comparing frameworks to pure code without any framework at all isn't exactly fair. Frameworks tend to add overhead in the startup phase, so the impact on short running requests will be much higher than for the average web application where you usually have a mix of short and longer running requests.

All in all, performance is a good pro-Java argument and always has been, though not the only valid one

 

[Jason]

Meh, this isn't a useful benchmark.

For one, a lot of the code looks like "Java written in X", which just doesn't work.

The Python examples definitely suffer from the above. No one competent in the language would write code like that. They're allocating globs of memory for no reason, and then throwing it away without ever using it. You wouldn't use Django's JSON serializer -- it's known to be very slow. It only exists for people with older versions of Python. You'd use the JSON serializer found in the standard library, instead. There's a few other glaring mistakes, as well.

The Go examples also suffer from similar problems. In fact, it wouldn't surprise me if most of the other languages do. That said, almost all the bottlenecks were seen in the libraries, all of which are standard libraries written in C, so interpreter vs compiler vs JIT made very little difference.

The problem with these kind of benchmarks is they use "throw away" code, and only take part of the equation into account, arguably the wrong half. You have to know your application intimately and benchmark using data which is realistic for that application. This benchmark falls well short of the mark.