I'm going to have to reject this suggestion because "make it faster" isn't really an actionable point. Nothing is intentionally slow and the work that's done is designed to ensure the system remains in a consistent and expected state. There isn't really a defined point that could ever make this suggestion implemented.
In general, with an optimized server setup and with a common production setup (minimal languages and styles), the rebuild process after installing an add-on shouldn't take more than 1-2 minutes. Less optimized systems or less common setups (many languages and/or styles) can take longer, though if it's taking hours, that's probably something to look at with your server (and the I/O of MySQL). In fact, most of the cache rebuild tasks are write heavy, so MySQL I/O speeds are very relevant.
I would note that if you know you're doing multiple add-on installs or the like, you can actually abort the rebuilds before they finish, though note that the system can be in a somewhat inconsistent state. It's not something I would officially recommend.
