I understand, but they're technically going to have a banner if they're in any user group that has one. Groups without banners are thrown out of the equation entirely. You'd have to remove them from all groups with a banner, or come up with a workaround.
You could try setting a custom CSS class for the banner, and set it to display: none;
in your extra.css
or extra.less
template.