1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

XF 1.4 Automated Bounce Email Handling

Discussion in 'Have You Seen...?' started by Mike, Jul 11, 2014.

  1. Mike

    Mike XenForo Developer Staff Member

    Most forums need to deal with members whose email address becomes invalid. Sometimes the user will go ahead and change it, but other times your forum will fire off emails to this now invalid address. When this happens, if you have a bounce email address setup in XenForo (and possibly with the -f option enabled in the mail configuration), you will receive an email telling you about the failure.

    Email delivery failures can happen for a wide variety of reasons. If you continually send to an address that has been disabled, for example, this can count against your domain/IP's email reputation. A lower email reputation can lead to reduced email deliverability as your domain may appear to be a spammer.

    Therefore, when you are notified that an email can't be delivered, you may want to prevent further emails from being sent to that user. XenForo 1.3 added a new user state that represents an invalid email, but you had to manually change users into this state. With a decent size forum, this could be tedious and time consuming at best and impossible to manage at worst.

    To handle this, XenForo 1.4 implements an automated bounce email processing system. This is a fairly detailed and technical system, but it allows an important maintenance action to be automated.

    Automated Bounce Parsing
    The core component of this system is a process that reads emails from a specific account and then parses those emails to determine if they are bounce messages.

    As bounce notifications will be sent to the value you enter for the "Bounced Email Address", you would enter the POP3 or IMAP details for that account here. Once you do that, the system will automatically start checking this account for bounce messages. Note that the emails in this account will be removed after reading so this account needs to be dedicated to bounces from your forum only.

    Every email that is read from this account will be analyzed to determine if it fits a known type and the appropriate action will be taken (changing the user state to indicate that the email is invalid, if appropriate). This is all displayed in the email bounce log in the control panel:


    The action taken will depend on the content of the email. The sections below discuss why an email could be untrusted and the distinction between soft and hard bounces.

    Bounce email notification parsing is notoriously difficult due to a wide range of servers that don't actually give good bounce responses. We have covered a number of common cases but this will certainly need to be expanded over time, so if you use this feature, keep an eye on the bounce log and let us know if you see emails that aren't being handled correctly. The "view original" link will take you to a raw output of the original email (headers and all encoded data). This data can be used to help us determine any additional cases to handle.

    Note that you may get non-bounce emails to this address, such as delay notifications, challenge verifications and auto-replies. These will be logged, but they don't represent any errors so no action will be taken.

    Bounce email logs will be maintained for 30 days.

    VERP and Validation
    While you would think that a bounce notification would always tell you who the email was sent to, you'd be wrong. Sometimes, the notification will only show you the address of the final recipient, and maybe they forwarded their email to another address. In rare cases, it might not list any address.

    A common way around this is with a technique called Variable Envelope Return Path (VERP). The concept behind VERP is to encode the recipient email into the Return-Path header (which is where the bounce notification is sent).

    Another important component is validating that the original email was actually sent by your forum. Without this, someone could manually create a bounce notification, send it to your bounce address, and trigger the bounce behavior for any email. By adding a validation hash into the return path, we can simply ignore any email that doesn't validate. This is what the "untrusted" component is in the email bounce log screenshot.

    Let's look at a quick VERP example:
    • Bounce handler address: bounce@yourdomain.com
    • Email recipient: user@example.com
    The VERP may look like: bounce+1234abcd+user=example.com@yourdomain.com. The "1234abcd" part is the validation key that is generated using a secret key.

    Keep in mind that the Return-Path is never shown to the end user (unless they look at the raw email contents). The Return-Path and the From/Sender values are always unrelated.


    We would recommend enabling VERP support if you enable the automated bounce handling option. However, it does require that your SMTP server ignores anything after the + sign. This is quite common these days; Gmail does it, for example.

    If you don't (or can't) enable the VERP option, we will embed this data in the original email via another header. Some bounce notifications include the original message (or the headers from it) and we can parse this data out of that. However, it may not be as reliable as the VERP option.

    Soft and Hard Bounces
    Bounce notifications report a wide range of issues. Some of these are unlikely to ever be resolved and others may be temporary or specific to a single message. This is where a soft and hard bounce distinction becomes necessary.

    When a hard bounce happens, action will be taken immediately (setting the user state to indicate that their email is invalid). These issues are unlikely to ever be resolved, except maybe through dumb luck. Most commonly, this will be an invalid mailbox (the part to the left of the @ in the address).

    Soft bounces include:
    • The recipient's mailbox being full
    • DNS/connectivity issues
    • The message being too long (as other messages may go through)
    Soft bounces don't trigger the user state change until multiple messages bounce. The exact behavior is defined by a set of criteria:
    The criteria is designed to give the recipient (or their server admin) a chance to resolve the issue. If the limit was based solely on the number of emails, a flurry of messages in a short time span could trigger the soft bounce threshold.

    Using the values in the screenshot, the email would only be considered invalid if:
    • We received 3 or more bounce notifications
    • The bounce notifications happened on 3 or more distinct days
    • The amount of time between the first and last bounce is at least 5 days
    These values can be tuned to fit your needs and how strict you want to be.

    Phew, that was a long one. Let us know if you have any questions or thoughts and keep an eye out for more 1.4 features being revealed soon.

    Just a reminder: Please do not post suggestions in this thread (even if you feel they are related). Use the dedicated suggestion forum so they can be tracked; suggestions made in this thread are unlikely to be implemented.
    rafass, SBell, Puntocom and 102 others like this.
  2. imthebest

    imthebest Formerly Super120

    Shut up and take my money!!!

    Mike you're a genius, really man you're very smart and I admire you.
    Last edited: Jul 11, 2014
    rafass, vbuser, dwdmadmac and 8 others like this.
  3. RoldanLT

    RoldanLT Well-Known Member

    Minus BD Mail addon ! (y)
    AzzidReign, MattW and Shyuan like this.
  4. Neal

    Neal Well-Known Member

    I agree. What a great new feature!
  5. Slavik

    Slavik XenForo Moderator Staff Member

    Given how notoriously difficult bounce handling can be, I have to tip my hat to Mike at working out some of the difficult points.
    Claudio likes this.
  6. digitalpoint

    digitalpoint Well-Known Member

    Whilly t, Markos and erich37 like this.
  7. Pereira

    Pereira Well-Known Member


    Two 1.4 announcements in two days.
  8. Walter

    Walter Well-Known Member

    Two questions:

    I assume VERP works with newsletters sent from ACP too?
    Is there a way to fetch the bounce mails not only from an email account but also from a non Xenforo database table (we already have a central bounce system in place and I want to keep it this way as we have the same users using several of ours forums)?
  9. Jeremy

    Jeremy Well-Known Member

    I would assume that any integrations would require an add-on.
  10. nrep

    nrep Well-Known Member

    Brilliant! Something I've been wanting for ages :D.

    What happens if two XF sites use the same send e-mail address for notifications? Would that cause problems?
  11. Jeremy

    Jeremy Well-Known Member

    Yes, you should use a dedicated email address per forum to ensure the bounce emails are read by the proper forum.
  12. Liam W

    Liam W Well-Known Member

    Wow, perfect!

    This is excellent.

    cPanel servers don't support plus sign things, but I guess I can create a bounce subdomain and a catch all email address for it.
  13. DJ XtAzY

    DJ XtAzY Active Member


    JK!..but yes this will be an amazing feature, cant imaging how hard for this to be coded. I actually like 2 HYS per week.
  14. kontrabass

    kontrabass Well-Known Member

    Two words: Woo Hoo.

  15. erich37

    erich37 Well-Known Member

  16. AndyB

    AndyB Well-Known Member


    Thank you, Mike and the XenForo team for making this feature. I sure look forward to XenForo v1.4.
  17. CyclingTribe

    CyclingTribe Well-Known Member

  18. pegasus

    pegasus Well-Known Member

    What if the bounce notification is due to the recipient server having ours blacklisted? Would this be considered a hard or a soft bounce? Perhaps a different status altogether would be appropriate?

    I don't think all recipients @aol.com should be changed if aol.com has our server blacklisted, even after multiple bounces. The recipients could get eventually start getting future emails once we request a de-list from the blacklist.
  19. Slavik

    Slavik XenForo Moderator Staff Member

    Id imagine that would come back as a soft bounce.
  20. Jeremy

    Jeremy Well-Known Member

    Yes, you would need a dedicated email.

Share This Page