Yeah the default email address would only affect outgoing emails. There are places where the default "from email" can be overridden, such as when you use Email Users in the Admin CP, you can specify any from email, in case you do actually want replies to that.
In terms of receiving emails from the contact form, that is the "Contact Email Address" and should be unaffected by changing the "Default Email Address".
So, just to check, does that cover what you are trying to do? If so I'll mark this as implemented