It's not the load speed of the sites as XF sites are generally fast, but the lags that occur when you scroll the site. The scrolling doesn't look smooth like with other non-XF powered sites. The lags don't occur on xenforo.com, but for instance on http://8wayrun.com/, http://www.pfps.com.br/comunidade/forum/, http://www.8thos.com/, http://do4a.com/
I think the larger the amount of content, the worse those lags are, but I didn't experience those lags on other non-XF powered sites with a large amount of content. I don't think it's a browser issue as I experienced those lags both with Firefox and Chrome.