Thanks!
I just tested once again and indeed it seems that the counts etc are updated via
Rebuild Board Totals Counter cron
The discrepancy also happens when a user is deleted.
Since this cron runs hourly, it means that during that period one can expect users to get a 404 for deleted user who still shows on the list. Can be a support headache