Disabling all addons results in a very small reduction in load time, but only around 10%.
I looked at a thread about load times (https://theadminzone.com/threads/your-xenforo-board-forum_list-index-loading-stats.116198/) and this seems to be above many of these, even on a powerful and empty server. IIS 8.5 isn't that bad! Google Page Insights even says "Reduce server response time" when I run a test.
Is there a way that I can break down that load time further? When I view the ?_debug=1 link it shows that the MySQL queries are running in milliseconds - so it seems to be PHP that is slow. I've tried PHP 5.4 and PHP 5.5, with and without WinCache and Zend OpCache enabled. The load times barely move. We're using MySQL 5.6.
Where else can I look? I'm sure there must be something I've overlooked to cause this.