This is largely by design. There are a bunch of assumptions we make that really rely on date ordering hence there's a bunch of compromises if we're not date ordered.
In this case, if we do try to mark things as read whilst sorting by vote score, there's a chance that we mark posts as read but they haven't actually been read yet.