When I send out emails, it sends to me fine but not the admin email (I made an account using the Admin email), but as I said in the OP that it allows me to send emails from my personal email to the Admin email, and vice-versa.
If you can't send anything to your admin email, it sounds like the sender has been blocked or the email has been blacklisted at the server level. If you change the admin email to something else (your personal email), does it send the contact form?
It sounds like your administrator's email is being blocked either on your server somehow or at the provider for your administrative email. XenForo can't do this if you change emails and it sends. Contacting your host / email provider will be more beneficial.