Minify CSS is an option in XF (if you've implemented a caching mechanism) - go to your Admin Control Panel > Options > Performance - and tick Minify CSS and save.
Leveraging browser caching is also useful - overview here (scroll down for more detailed info for your respective web server).
So there's a few things you can do to get started and then fine-tune it further if you feel it is needed.