A record in xf_conversation_user should be sufficient to establish that the user can view the conversation -- they would only have that as a recipient that's active.
Presumably you have a situation where that doesn't hold? In which case, how that happened would be the important thing to establish.