Well, I don't see any reasons to not update. I have a lot of add-ons installed, and so far I've not noticed bugs. In debug mode, there is a warning about preg_replace() though (being deprecated with /e modifier).
About speed, before, php-5.4 and APC were installed, now php-5.5 and Zend Optimizer+ ; from what I've seen my forum is really faster yes, more reactive for sure, at least if not much, it's still quite noticeable for me. For my case, I would say I gained around 15% in speed if not more -probably -.