Unfortunately, this definitely appears to be a Gravatar issue. We serve avatars directly from Gravatar, so we're at the mercy of their cache life time. Ok, that should be fine -- it's only 5 minutes. However, they're doing caching beyond that - at the Varnish level most likely, so it's not a client-side issue. If you set additional URL params, you can bust this caching on their end, but only for that URL.
This is something they'd have to resolve. Note that it's not specific to XF URLs. It may be related to the size options they provide and them not expiring the cache correctly. Unfortunately, they don't have any contact form on their site...