- Affected version
- XF 2.2
Last week I asked if anybody has an working example of an bounce email that the automatic bounce handler recognizes.
i got no answers as I was afraid since almost all discussion about this handler show almost nobody has it working and many people giving up on it.
I don't know if even xenforo has it working on their own boards that they operate.
My system is different from most others of you since I do not get an email but rather an webhook from email provider if email bounces and therefore I have the freedom to craft any kind of email I want based on the webhook I get then to be sent to the xenforo automatic email bounce handler.
The example bounce emails I get from sptm.gmail.com and smtp.zoho.com neither works to get recognized in the automatic bounce handler so something is wrong.
Well, I had some spare time and after sending about 100 different test bounce emails I finally got a working system for me where I can send an email and it gets recognized. During this process I encountered some bugs that I needed to work around to get the emails recognized.
The good news is that both of these problems sound like very easy to fix and there seems to be a bunch of working code in the backend of the system if just emails would get past the recognizing phase of the automatic email handler.
The main problem seems to be related to line breaks of the email.
How the email system works is that while sending emails there are extra line breaks added to the email content body which the receiving end is supposed to strip out before handing the email contents to the recipient. Xenforo has forgot to put this stripping of extra line breaks part of the process of receiving emails into their system and so these extra line breaks are left into the content body and they work as corruption preventing the automatic handler to correctly read the interesting parts of the email. You can see these extra line breaks if you go look at the raw contents of received emails.
The extra line breaks are added to about every 70th character in the email body and since there seems to be 3 important sections that the automatic handler looks for and each of these important sections are about 20-40 characters long the probability that none of the important sections are corrupted is about 10%. 90% of the bounce emails fail at this first step.
Except if one ticks the box in option "Enable variable email address values for automated email handling" which makes the email address that the automatic handler is supposed to read from the content much longer meaning that the probability that this longer email address survives the corruption from extra line breaks is almost zero.
The process of adding line breaks is deterministic so some email service provider could send bounce emails that happen to have all important sections of the bounce email uncorrupted and this might be the reason this feature was working when somebody wrote the code to xenforo without noticing that it is missing important part of the system which makes so that almost none of other email service provider bounce emails are similarly lucky. Some of them could be lucky though so somebody could have this feature working right now. However that percentage is likely to be too small to have anybody of those lucky individuals to show up on this xenforo support forum.
There is still another additional problem though. It seems like the automatic bounce handler is looking for some english language abracadabra that triggers the whole detection process. This seems also very fragile system as every email service provider can send their bounce emails with completely different sections of english language explanations. The bounce emails from gmail and zoho look completely different and one from gmail does not seem to contain the right abracadabra and so the automatic handler seems to discard all bounces from smtp.gmail.com right away.
I think the abracadabra check needs to be made more universal so that many different kinds of bounce emails from multiple email service providers have a hope of getting recognized. One way to do this is to make the abracadabra to be user configurable by instructing the forum administrator to take look what kind of bounce emails his email service provider sends and forum administrator picks some abracadabra section that he then configures into xenforo to look for.
Another possible approach is for xenforo to choose some number of biggest email service providers and add support for each of their abracadabra to maintain support for these X number of supported email service providers which xenforo then makes public to recommend those providers.
Without doing something to this abracadabra problem I don't think the probability of getting bounce emails recognized can get over 50%.
There could be a problem also in the handling of the error code to detect either hard or soft bounce. I just got the feeling that it might be fragile during testing but since I got the system working for me I did not prod this issue further.
These are my observations. There seems to be a lot of good code in the backend if only this bounce email recognization problems are fixed and both of these problems sound very easy to fix.
If some xenforo developer wants to get my email format that I achieved to get working past the automatic detection you can email me and I will send it. It is unlikely to help anybody else right now since most of you are at the mercy of whatever your email service provider happens to send to you and there is nothing you can do to help this.
i got no answers as I was afraid since almost all discussion about this handler show almost nobody has it working and many people giving up on it.
I don't know if even xenforo has it working on their own boards that they operate.
My system is different from most others of you since I do not get an email but rather an webhook from email provider if email bounces and therefore I have the freedom to craft any kind of email I want based on the webhook I get then to be sent to the xenforo automatic email bounce handler.
The example bounce emails I get from sptm.gmail.com and smtp.zoho.com neither works to get recognized in the automatic bounce handler so something is wrong.
Well, I had some spare time and after sending about 100 different test bounce emails I finally got a working system for me where I can send an email and it gets recognized. During this process I encountered some bugs that I needed to work around to get the emails recognized.
The good news is that both of these problems sound like very easy to fix and there seems to be a bunch of working code in the backend of the system if just emails would get past the recognizing phase of the automatic email handler.
The main problem seems to be related to line breaks of the email.
How the email system works is that while sending emails there are extra line breaks added to the email content body which the receiving end is supposed to strip out before handing the email contents to the recipient. Xenforo has forgot to put this stripping of extra line breaks part of the process of receiving emails into their system and so these extra line breaks are left into the content body and they work as corruption preventing the automatic handler to correctly read the interesting parts of the email. You can see these extra line breaks if you go look at the raw contents of received emails.
The extra line breaks are added to about every 70th character in the email body and since there seems to be 3 important sections that the automatic handler looks for and each of these important sections are about 20-40 characters long the probability that none of the important sections are corrupted is about 10%. 90% of the bounce emails fail at this first step.
Except if one ticks the box in option "Enable variable email address values for automated email handling" which makes the email address that the automatic handler is supposed to read from the content much longer meaning that the probability that this longer email address survives the corruption from extra line breaks is almost zero.
The process of adding line breaks is deterministic so some email service provider could send bounce emails that happen to have all important sections of the bounce email uncorrupted and this might be the reason this feature was working when somebody wrote the code to xenforo without noticing that it is missing important part of the system which makes so that almost none of other email service provider bounce emails are similarly lucky. Some of them could be lucky though so somebody could have this feature working right now. However that percentage is likely to be too small to have anybody of those lucky individuals to show up on this xenforo support forum.
There is still another additional problem though. It seems like the automatic bounce handler is looking for some english language abracadabra that triggers the whole detection process. This seems also very fragile system as every email service provider can send their bounce emails with completely different sections of english language explanations. The bounce emails from gmail and zoho look completely different and one from gmail does not seem to contain the right abracadabra and so the automatic handler seems to discard all bounces from smtp.gmail.com right away.
I think the abracadabra check needs to be made more universal so that many different kinds of bounce emails from multiple email service providers have a hope of getting recognized. One way to do this is to make the abracadabra to be user configurable by instructing the forum administrator to take look what kind of bounce emails his email service provider sends and forum administrator picks some abracadabra section that he then configures into xenforo to look for.
Another possible approach is for xenforo to choose some number of biggest email service providers and add support for each of their abracadabra to maintain support for these X number of supported email service providers which xenforo then makes public to recommend those providers.
Without doing something to this abracadabra problem I don't think the probability of getting bounce emails recognized can get over 50%.
There could be a problem also in the handling of the error code to detect either hard or soft bounce. I just got the feeling that it might be fragile during testing but since I got the system working for me I did not prod this issue further.
These are my observations. There seems to be a lot of good code in the backend if only this bounce email recognization problems are fixed and both of these problems sound very easy to fix.
If some xenforo developer wants to get my email format that I achieved to get working past the automatic detection you can email me and I will send it. It is unlikely to help anybody else right now since most of you are at the mercy of whatever your email service provider happens to send to you and there is nothing you can do to help this.
