The lack of login is what concerns me. It sounds as though someone knowing the Gravatar address of User A (who has yet to register on my site) could conceivably use that info to create an account on my site and impersonate User A. Same name, same avatar; who'd know the difference?
I know, you don't need Gravatar to do this. What Gravatar would buy the impersonator is a dead simple to keep in sync with any avatar changes made by User A.
Maybe this could be avoided by giving Gravatar a separate email that isn't used for anything else. But that means that I'd be asking forum members to take two extra steps (create a new address & sign up with Gravatar) just so they can replace the default avatar with an image that's sitting on their desktop. I'm not sure I'd want to do that, especially if the users are paying subscription fees for premium membership.