This situation occurred because we do a recount of the badge count when the pop up opens. That recount is based on count($conversations['unread']). Obvious problem is, as we only retrieve a maximum of 10 conversations, if you have more than 10, then the count was rebuilt to 10.
Solution was just to query for the correct count. Costs an extra query but better than being wrong