Implemented Bounced Email Management

[Alpha1]

Please do not forget bounced email processing. The absence of bounced email processing in vbulletin has led to a large number of boards being blacklisted as a spammer. As a result their members do not receive emails from the board.
vbulletin is one of the largest spammers on the internet, breaking a large number of laws.

Its important that XenForo does not make this mistake.

 

[Gene Steinberg]

At this stage, I think the chances are slim to none that XF will have this as a core feature anytime soon. I assume Kier and Mike are busy wrapping up 1.1, and will be considering options for the future, not the present.

But this would be an ace opportunity for an add-on developer to deliver a solution. I presume that the add-on would somehow have to monitor your chosen "bounce" mailbox for headers that contain info that the message was sent via XF.

 

[Alpha1]

So is this something an add-on user can do, or must it be done in the software itself? Please clarify.

Its not enough to just blacklist accounts that get more than X bounces, because a wide array of bounces can be harmless. It needs specific definitions how each type of bounce is handled.

While its possible to create an addon to do this, there is a serious maintenance problem with this. Because bounced email changes quickly, definitions about what is bounced email and how the system should respond to specific bounces changes rapidly. Such definitions need to be kept up to date.

For example: many people use bluebottle now. It blocks valid users from receiving email until the admin clicks the bluebottle link. This is one of many new phenomenons that greatly change bounced mail. Disposable email domains and their ever changing aliases is another.

I doubt that an addon developer will release new definitions several times a year.

 

[Gene Steinberg]

I think an add-on of this sort would have to concentrate on the low hanging fruit, the XF email that is simply returned to the sender for any reason. Period. The member is automatically placed in an Inactive or similar category. At the administrator's leisure, they can look at exceptions, such as services that require email senders to click a link to get permission to contact the subscriber to that service. This is nothing new. The board administrator should, as a matter of course, inform members that they should whitelist forum mail to avoid such problems, period.

That shouldn't require special coding or definition changing, beyond looking for common messages about why a message is rejected. Our mailing list software works this way (we use Dada Mail). Bounced messages are scored at 4 points each. At 10, the email address is deleted from the list. This allows for inadvertent delivery issues that may cause a couple of messages to be returned.

 

[Alpha1]

Nowadays, 'mailbox full' or 'delayed X hours' bounces can be received a lot of times. There is no need to deactivate those accounts. While 'email does not exist' may be sent only once, but should definitely be blocked right away.
I doubt that the points system works well if your system sends out thousands of emails per month or more.

 

[Gene Steinberg]

Those return messages would be temporary. Unless you are sending messages to the same people every day or two, it shouldn't be an issue. But if the message bounced a week later, or a week after that, it's fair to consider the possibility the messages won't get through. I would expect board staffers to manually examine the returns also. The add-on shouldn't be a panacea.

 

[digitalpoint]

Bounced mail processing is a little tricker than one would think. You have a few issues to deal with...

- Some mail servers accept mail even if the address is invalid, so the initial mail log shows it as accepted for delivery.

- If you want to check a specific email account for bounces and give it a proper bounce address in the header, it doesn't mean the receiving mail server adheres to it. You also run the risk of the user responding to an automated message and them accidentally sending it to the bounce address.

- Some bounces are temporary (over mail quota for example) or their destination mail server is temporarily down.

For my site, I've taken the "safest" approach... if a mail accepts something for delivery initially, it's on them. I'm not going to bother with "after the fact" bounces or temporary mail server problems/bounces.

What I do is just kick off a daily crontab job that does a zgrep (same as grep but works on zipped files like logs) on our 4 servers that can send outgoing email and check for bounces or emails that have been deferred for more than 4 days (down mail server). Then I simply move all users with those emails to the non-email confirmed usergroup. They can't do anything like post or anything else until they confirm their email as valid again or change their email to a valid one.

But my little process is designed around my servers... multiple servers, assumes you are using postfix for the mail log format, etc... Once you get into trying to automatically manage bounces going to an actual email address it gets a whole lot more difficult/tricky.

 

[Gene Steinberg]

I opt for simplicity. If someone's email is bounced several times, for whatever reason, they're off my mailing list. That's how our mailing list software works. It's not my concern what's wrong with their email; it's their responsibility to fix the problem, assuming the address is still valid.

As to a bounced email add-on, again the same consideration. If the email bounces several times, the member is placed in a different group, and, perhaps, you open a conversation explaining the problem and telling them what they might do to fix it. It's up to the individual board owner to decide if that different group also reduces the member's privileges. With me it does. End of story.

 

[Brent W]

I use http://postmarkapp.com/ to send out emails on my newest site. They offer a location to insert a url for a bounce hook (More here: http://support.postmarkapp.com/customer/portal/articles/64744-what-are-bounce-hooks-)

For those of us that out source email sending, it would be incredibly amazing to have xenForo support bounced processing in such a way

 

[Hoffi]

this is really important.

You need to build a bounce handler that scores the possible hard- and soft-bounces.

At a appreciated score, the user should't get any more mails, to prevent SPAM Listing.
For a special function, the user also should be moved into a special usergroup.

Nearly every professional newsletter-tool can handle this. This is a must for every tool, that itself sends mails.

 

[digitalpoint]

this is really important.

You need to build a bounce handler that scores the possible hard- and soft-bounces.

At a appreciated score, the user should't get any more mails, to prevent SPAM Listing.
For a special function, the user also should be moved into a special usergroup.

Not even sure you need to differentiate between soft/hard bounces. A bounce is a bounce really... And as far as moving to a user group, I'd just say if it bounces (any sort), the user simply goes to the "Unconfirmed Email" group. Then they don't get any more email and once they reconfirm their email (or change their email to one that can be confirmed), they are back to normal.

 

[Hoffi]

Oh yes, it is a big difference.

A soft Bounce telly you e.g. this Mailbox is full. So the score is low and a new attempt will be in one or two days.

A hard Bounce can be "Server not avaibale" or "Mailbox did not existit". In first Message the next try may be also the next day. But if the Mailbox did not exists, the next attempt would be in one Week, or never.

 

[digitalpoint]

Well personally, I'd stop sending email permanently even on a soft bounce until the user rectifies their problem (even if it's "just" a mailbox full).

 

[snoopy5]

Well personally, I'd stop sending email permanently even on a soft bounce until the user rectifies their problem (even if it's "just" a mailbox full).

+1

So please can someone make an addon for XF for this?

 

[snoopy5]

How are bounced currently handled with xf? I am importing tonight many vb forums. I just used the plugin ezbounce for this to clean everything before naking the final import. Not really convenient, but better than nothing.

So after my import into xf, how can I handle in the future new bounces with the normal features of xf?

 

[snoopy5]

... A bounce is a bounce really... And as far as moving to a user group, I'd just say if it bounces (any sort), the user simply goes to the "Unconfirmed Email" group. Then they don't get any more email and once they reconfirm their email (or change their email to one that can be confirmed), they are back to normal.

This is also the way I want to have it. Simple and efficient. Everything else is up to the xenforo team as soon as they plan to release a feature like this in the next core.

I need this really and I would be happy to pay someone to write an addon for this.

Can someone who is interested give me a quote for this?

If someone else wants to join to finance this, you are welcome. But I do not want to have a supercomplicated addon. Just plain simple as I described it above with the quote of digitalpoint.

 

[snoopy5]

Hi

I just want to push this thread up again, since development for the next XF version started. I do think that a bounced emails processing within XF core is one of the very few things XF really needs asap. Nobody wnants to get blacklisted because of the lack of this feature.

Please also check these threads for this:

http://xenforo.com/community/threads/bounced-emails-the-nightmare.41239/

http://xenforo.com/community/threads/update-user-account-from-bounced-emails.44174/

 

[thumped]

All* of my users are on a Mailchimp mailing list. I trust MC to handle bounces / address updates, so would it be possible to use their API to periodically check for dead addresses & update forum accounts accordingly?

* in reality about half of them are, because after 12 years the other half are defunct addresses....

 

[snoopy5]

I do think that we need to have the bounced email feater as simple as possible to be able to make it already in the next version. There is no problem to enhance in future XF versions later on. A kind of 80/20 solution now and if neded refinements later.
Everything else makes it too complicated to code and wth this the likelyhood that this feature makes it already in the next release will decrease.

Therefor I would vote for a very basic feature for handling bounced emails at the beginning. It has to handle them without third party add-ons or other mailing lkist software. Most likely with a simple link in the bounced email to manually click on and changing with that in the ACP the settings automatically. So that plain vanilla XF owners can handle bounced emails rudimentary as a first step.

Maybe add a script to read out bounced emails that came back already in the past and convert it into a list that generate these links. With this we could immediately benefit also for our already baiunced emails significamtly. I still have over 1000 bouced emails in my outlook folder that needs to be cleaned/chnages in ACP. I do not want to send them again an email just to be able to use a bounced email feature

I guess many here are in the same boat. So with these very basic 2 solutions, you would solve already 80% of our problems and it is fatser to code than complicated solutions. Everything else then (if really needed in the core) at a later time...

 

[Alpha1]

I trust MC to handle bounces / address updates

I don't. MailChimp sucks at defining bounced email. The process it uses is really crappy as it does not distinguish between bounce types. So a full_inbox bounce is treated the same as email_does_not_exist.

 

[thumped]

I don't. MailChimp sucks at defining bounced email. The process it uses is really crappy as it does not distinguish between bounce types. So a full_inbox bounce is treated the same as email_does_not_exist.

sorry, i meant their list cleaning processes

Removing Soft Bounces
MailChimp gives "soft bounces" (server overloaded or down at the time you sent to your recipient, etc) 8 chances before cleaning them from your list.

Removing Hard Bounces
When MailChimp receives a "hard bounce," (such as an inactive account, or undeliverable address) we remove the email address from your list. We store these at Lists > View > Subscribers > choose "cleaned" from the drop down box (see screenshot above).
We also factor engagement activity into our bounce cleaning decisions (read about how MailChimp measures engagement). For example, if we send an email and a receiving server tells us that a recipient “does not exist,” but we have open and click activity in the last 45 days to prove otherwise, we’re not going to blindly clean that recipient from the list. We know they exist, and we know their account works, so we’re going to give them a few more chances than we normally do. If, however, we see that there’s very little (or no) activity by that recipient, we clean them under the same rules we’ve used in the past.
We can't get into specifics about how many stars justifies a “clean vs. a keep,” or exactly how many chances we give hard and soft bounces. The algorithm will surely be adjusted and tweaked over time.

if they're cleaned from mailchimp i'd be quite happy to have them automatically moved to an 'inactive' xf group with no posting rights or notifications until they update their mail address.