Your probably better off setting up a custom page that loads a contact form that way XF permissions will not interfere for people trying to contact you, from my experience ive been trigger happy (like most webmasters) when it comes to banning members / ip's so usually i try to create my contact forms outside of the main platform permissions so that if a banned user has an excuase to contact me that they can indeed do so... XF conditionals should allow for to display certain pages to all usergroups even banned usergroups.
Good luck and hope that helps.
Darren