I converted a vb 4.2.1 forum to xf 1.2 recently and with VB my Google webmaster tools crawl stats page showed between 170 to 190ms for time spent downloading a page and since the conversion to xenforo 1.2 it has been showing between 60ms to 80ms with no spikes above 80ms. This is mainly due to the size of the page but apart from the size I do see a decrease in page generation time. This is all on the same server with no changes.
Even with the ImageMagick-heavy custom application on our site, it's still significantly faster than phpBB ever was. And I've never ever heard of anyone complaining about how XenForo is slower than pretty much anything else before, so I'm finding it difficult to believe the software is at fault here.