The same potential issue happens with threads/posts with thread/forum watching. Unfortunately, we don't really have a great general solution for this. This is something we may need to look into for a future version.
A possible partial workaround involves limiting to only attempting to contact people that have been active on the forum in the last X days. (On the assumption that if someone hasn't visited in months, they're unlikely to come back but that may not be an accurate assumption in all cases.)