I'm pretty much fed up with PHP-FPM.
I finally got my uploading bug fixed, only to find out that the fix breaks XF's multi-file uploader. That and the account/avatar bug that I can't figure out... just went back to nginx/Apache. The memory footprint for php-fpm is basically the same as my stripped apache. The page loads "feel" faster with PHP-FPM, but I think it's just because everything sort of organically appears, where apache waits a beat, then everything appears mostly all at once. Not sure that one is genuinely faster than the other, though.
I wouldn't remove Apache first (or at all) if you're installing nginx. If you have multiple external IP addresses, you can set apache to listen both on one external IP and on localhost:80, then set up nginx on another external IP with the reverse proxy stuff to apache (or php-fpm if you prefer). Once you're sure it's all working correctly, change your DNS record to point to the nginx IP address, wait a few days for everyone's records to update, then disable the external IP for apache (so it only listens on 127.0.0.1:80).
Doesn't the nginx repo have ubuntu packages? Could always give that a try rather than compiling your own. I switched to their repo (but for RHEL/CentOS) a few months ago.