Thanks! I uninstalled some styles and now its really fast!It'll all be down to hardware performance, whether you have shared hosting or dedicated resources etc.
Also the number of styles you have installed has a massive impact -- moreso than the number of add-ons.
If you're uninstalling multiple add-ons to save yourself a bit of time, hit the back button in your browser just after you first see the rebuilding caches screen.
Then uninstall another and do the same.
On the last one, let the cache rebuild complete as normal.