All on one server or multiple servers??
I am running 2 smaller linode vps servers (1 for the web side and 1 for the DB side), with apc install on the webserver and memcached installed on the DB server. My sire is fairly fast for 2 little 512MB slices and the amount of traffic I get. Once I got the memcached sorted out (documentation was wrong), there was a noticeable increase in side speed. Then I just have APC installed and let XF determine what goes into the APC cache.
Russ