1.3 added a number of indexes to help alleviate issues like this. If you're adding custom things to this, you should generally index the user_id field. (There are a couple changes that don't have indexes and the trade off wasn't considered worthwhile.)
This is something we may consider for the future, but generally indexing is going to give a significant amount of benefit.