On one of my website I'm using 39-40 addons without any problems, but it's a small board.
The only problem I've meet in using too much addons was the one cause but the Suhosin extentsion (
see here - Jake found the problem very quickly).
Now I think the coding part is important. For the templates listeners, the best thing would be to avoid them as much as possible thanks to Templates Modification System. All the modifications are done once in the admin side before to be cached, then it has no consequences on the users side.
I'm really not a specialist about optimization but what I know is when I started with XenForo and I wanted for example to call the XenForo Options object I did it once at the top of my callback and outside the conditional... which means the object was called for each hooks (or templates). So now, I prefer to call it inside the conditional for each hook. I'm not sure it has an influence but it seems better. An experimented coder might have the answer.