If you set up XF to process bounced emails, check the bounced email log in your ACP for error messages in the bounced emails. I had this issue recently and my email server wasn't fully configured (missing reverse DNS). Yahoo (which controls Yahoo, AOL, Verizon and other mail systems) wouldn't let any of my email through until we got it fixed.