I went ahead and built a XF2 add-on the allows security keys to be used for two-step verification. My thought was to build that first (since that was more useful to me than the straight password-less login system) and then see about doing the password-less login system after the fact.
In building the two-step portion, I learned a whole lot about WebAuthn, and to be honest, I don't think it's a really good fit for password-less logins, and here's why...
Both the registration and authentication functions require that the end user pick a username
before doing the trickery with the security key. For example, in order for an existing user to use it to log in, it's not as simple as just using the security key... you also need to manually enter your username (at minimum). The technical reason for this is because when you request a user to authenticate with the key, you have to tell the API which credentials you already have that you will accept. The
allowCredentials option in the spec is technically optional, but for practical purposes with XenForo, it's not without major schema changes. I'm doing it by loading multiple security key credentials into a single xf_user_tfa.provider_data column. For two-step verification, I simply pull the record for that user, then look in the json array to see if the given credential matches. It's a lot more difficult going the other direction (where you might have a credentialId, and need to find which user it belongs to). Unless you created a whole new table to store security key credentials, it's not (realistically) a query you can do.
Registration would be a little kludgey, because you still need the user to pick a username (and probably an email address) when they register (but before they use their security key). You can't have a XF user without a username, so there's that (no way around them needing to pick a username). And at that point, you aren't saving a whole lot of time/energy for the registering user (they are needing to more or less do all the normal stuff at registration except pick a password, and instead of doing that, you are giving them something new to do [use the security key]).
You can kind of see what I mean with the demo here:
https://webauthn.io/ ...you can't Register
or Login without giving it your username first. It's not like you just come to the site and use your security key and are good to go. There's more to it.
Note about non-hardware security keys... you can use them, but it's really not that great for portability reasons because the security key is the physical device itself... like I can setup a security key based on Face ID of my phone, but that doesn't carry over to an iPad or macOS with the same Apple ID. This also means if my phone is replaced or lost, that security key is also lost. You can have multiple security keys, so if one wanted to go through the hassle of registering every possible device as a security key, I guess you could do that and unregister/reregister a new one if it's replaced. But that seems like a hassle to me... the YubiKey 5 NFC is the way to go for me... if I have a USB port or an NFC-enabled phone, it works with it all as a single security key that you can use with future devices.