Is it possible to add "custom" provider? Any tips or guides how to do that?Are you implying that your users would login/register via an existing OAuth2 provider? Or are you looking for the forum itself to provide an OAuth2 server?
If the former, we have an "external auth" framework which we use for the "Login with Facebook|Twitter|Google" feature that we have. This is fairly easy to extend to add new providers but that would require additional development.
I do have couple OAuth servers (providers) that I would like to try out with XenForo.