Bans are not determined by user groups in XenForo. To add a banned user to a banned user group you need to setup a user group promotion to be applied when the ban action is used. By default there is no banned user group so you must have already created that. Generally a banned user group would be created for styling purpose or a specific need.
Ok I have just done this.
User group promotions > promotion options > add to user group > banned.
Apply this promotion while.... > User state is: Rejected
This works and adds them to user group 'Banned'. Banned user group does not have permission to use the contact form, however they can still use the contact form.
It is only when I manually change their user state from 'Rejected' to 'Valid' that they can no longer use the contact form and its because 'When in a user state other than valid, users will receive permissions from the unregistered / unconfirmed group'.
So while rejected users are moved to the banned user group and all other restrictions are applied the contact form permissions are still the permissions from the unregistered / unconfirmed group for some reason (minus form verification).
The unregistered / unconfirmed group does have permission to use the contact form and I currently have it set that unregistered / unconfirmed group have to verify they are not a robot when using the contact form and legit registered users do not have to verify so I receive very little spam apart from spam from rejected users.
As far as I can see there is no way to automatically change their user state as well as adding them to the banned group so it does not solve my problem.
Looks like I will have to have verification on the contact form for registered members as well as unregistered in order to solve the problem.