Don't know if this is the right place, but if I have a jpg in my media, which is a typical rectangle, and click media as an Avatar, it produces what looks to be proper resizing.
When I see the Avatar in the forum, it has improper perspective, like this
Unfortunately, I can't reproduce this, so it may be related to the specific version of the image processing library you are using. Do you know if you're using GD or ImageMagick? (Use the XF control panel search for "Default Image Processor".) If you have the ability, can you swap to the other one? Do you know the version of each that is being used (admin.php?tools/phpinfo - look for the sections labeled "gd" or "imagick", respectively).