Actually I have just tried it myself. Before this feature was core on Xenforo I used Andy's "New Conversation"-add-on. When the conversation was started, a email was sent for that conversation. Now I switched to Xenforo's option and tested it with a new user I registered. The conversation is there, the alert, too. But no email was sent. Can anyone of you reproduce it?