XF 1.5 SMTP problems

[imthebest]

Hello,

I have two servers, each server running a different 1.5 forum. Both of these forums are configured to use a SMTP server for email sending. This SMTP server is a VPS that I also own. On both forums at Email Options I have:

SMTP: smtp.mysite.com:25
Authentication: none
Encryption: none

The SMTP server is running Postfix and is protected against unauthorized use by iptables and the mynetworks directive in the Postfix config file.

Well the problem is that one of my forums is randomly and many times a day logging the following errors:

Email to xxxxxxxx failed: Connection refused
Email to xxxxxxxx failed: Could not open socket
Email to xxxxxxxx failed: No connection has been established to smtp.mydomain.com

The other forum is working fine. The key here appears to be the activity of both forums: the forum not experiencing issues isn't that active as the forum experiencing issues. On the forum experiencing issues, it looks like XenForo sends many emails per second (or minute, I don't know) and this causes the SMTP server to refuse connections. The strange thing is that my Postfix don't appear to have any throttling nor limit regarding connections so maybe the problem is the way in how XenForo handles the mail?

I know that in 1.5 the email is now being sent in a deferred process (which implies a batch of emails to be sent at once unless I'm wrong). I have to say however that this problem has been present even in previous versions like 1.4 and even 1.3 IIRC.

Thanks,
Super120

 

[Chris D]

I know that in 1.5 the email is now being sent in a deferred process (which implies a batch of emails to be sent at once unless I'm wrong).

This only applies to the Email Users function in the Admin CP.

Generally those errors can only ever be connectivity issues. If the errors are happening as a result of some sort of limit or throttling somewhere, those limits would need to be increased once you ascertain what they are.

I would also hope that there would be some logs available on the mail server side that may suggest what is happening on the server when these errors occur.

 

[imthebest]

I know that in 1.5 the email is now being sent in a deferred process (which implies a batch of emails to be sent at once unless I'm wrong).

This only applies to the Email Users function in the Admin CP.

That doesn't seem quite right because the "Request State" for those errors is almost all the time this:

array(3) {
["url"] => string(37) "http://www.mysite.com/deferred.php"
["_GET"] => array(0) {
}
["_POST"] => array(4) {
["_xfRequestUri"] => string(18) "/conversations/hello.1/ OR /forums/node.1/create-thread"
["_xfNoRedirect"] => string(1) "1"
["_xfToken"] => string(8) "********"
["_xfResponseType"] => string(4) "json"
}
}

Which makes me think that actually those emails are being sent in a deferred process.

Unfortunately no errors are logged in the maillog file of my Postfix SMTP server. It looks like XenForo randomly isn't even able to reach the SMTP server (thus nothing get logged in the maillog) which makes me think about a possible iptables issue however I have double checked everything and I can't figure out why it works fine for one of my forums and why it fails for the other.

Please see attached screenshot:



Look at the time when they happen, as you can see they seem to happen randomly and some times these errors happen at the same time.

Thanks,
Super120

 

[Chris D]

I should have made myself more clear.

There was a misconception that XenForo 1.5 included functionality similar to this:
Email Queuing Enhancements

But in this particular area, nothing has changed in XF 1.5. There is a mail queue system, but it's not new. It does process the queue in a deferred job, as it has done since 1.2.

It may well be that the number of emails queued up by XF is too much for your server to handle, but there's not really any configurable options here. You can disable the mail queue:

$config['enableMailQueue'] = false;

However, if this is a case of your server rejecting the connection because of some sort of throttling or other limits, those limits will potentially still be hit, perhaps even more so. Imagine a thread/forum that a lot of users are watching, this could generate a lot of emails to be sent all in one go, whereas in the case of the queue system, it will send for a maximum of X seconds at a time.

The X seconds, incidentally, comes from:

$config['rebuildMaxExecution'] = 8;

8 is the default, and this is the length of time in seconds that deferred jobs that use the standard target run time should run for before being re-queued. Changing this would likely impact on other things, too, such as template rebuilds, etc. It isn't usually required to change it. Though if you did change it to a lower number, then the mail queue should execute for less time before re-queueing the job.

Ultimately, though, the issues you're experiencing aren't something we would expect to see, and rarely do with any level of regularity, so it still firmly rests in the direction of being a configuration / server / network issue. The solution shouldn't really be to change any of the things talked about in this post, the solution should be working out why the connection to the mail server is dropping.

 

[Mike]

This isn't directly answering the question, but since you're talking about Postfix and running your own mail server, why have XF use an SMTP at all, instead just talking direct to Postfix locally? There is also the option of having Postfix be your relay itself, if you really do want to use an external SMTP server: Configure Postfix to Send Mail Using an External SMTP Server This Postfix would handle all the queuing and retrying then if there were network issues; that's a fundamental part of its job.

 

[imthebest]

Hi Mike,

I'm protecting the IP of my servers behind CloudFlare. If at Email Options I choose "Default" instead of "SMTP" then my server IP is going to be exposed in the headers. What I'm doing is using a cheap small VPS as a shared SMTP server for my forums so this way the original IP of my servers doesn't gets exposed because I have the following on the header_checks file of my Postfix (on my SMTP server):

/^Received:/ IGNORE
/^X-Originating-IP:/ IGNORE

Do you know if it is possible to send XF transactional emails using my own server without exposing its IP? If that is possible then I would be really interested in sending the email through my own server and removing the dependency of an external SMTP server.

Thanks,
Super120

 

[Mike]

The IP logging is actually part of the spec if I recall, but it's handled by the receiving server's end. This is where you'd use Postfix to relay to another SMTP server (your other postfix server), which would scrub what you want and then actually send the mail.

 

[imthebest]

So basically you are saying that my implementation could be improved by following these steps:

1. Configure Postfix on each of my servers to not send emails directly but instead use an external SMTP server.
2. At Email Options change from "SMTP" to "Default".

And by doing those steps I would get rid of the issues because my local Postfix is going to detect connection issues to the external Postfix and then queue for retry any failed email delivery attempts?

 

[Mike]

Roughly speaking, correct. Clearly, it would need testing to confirm it works as expected (especially with the header scrubbing you want to do).

 

[Nuno]

You need your own relay to hide your IP.
You can hide "Received: by" that is written by your local postfix but mandril/sengrid/mailgun will write "Received: from ...".

I read somewhere that AMZ SES hides your IP.