Strange Email Address Being Used?

[amirm]

Affected version

2.2.7

Hello everyone. Appreciate help with this odd problem.

Once in a while, I get large number of Abuse complaints from Comcast regarding thread notifications. This happened last night with many thread notifications (to different users I assume). The notification from Comcast shows this offending email address: f55c1b21f62ba5f57743eac74ff63e6a@comcast.net reported in the "inline.txt" abuse message. Naturally there is no such email address in the system. It appears that a GUID is substituted for email address???

Meanwhile Amazon SES has paused my email sending account. I tried blocking the address on their system but they won't let me do that while the account is paused! I am escalating this to them.

I imagine this is either a bug in Xenforo or Comcast is reporting this incorrectly.

Appreciate any help on how to investigate this further.

 

[Paul B]

Emails sent by XF use the address entered in the ACP.

Any changes being made to the address will be due to your email provider.

 

[amirm]

Thanks for the reply. What is strange here is that email notifications to many members were changed to that one single email address, causing Comcast to send the Abuse message. I find it hard to believe Amazon SES service has such a bug.

Once this happened, large number of thread notifications back to back were sent to this one address. This tends to indicate it is Xenforo that is doing this.

This has happened once before.

 

[amirm]

I tracked this down to user alias being corrupted even in the email it was trying to deliver!



Strangely, a different set of hex values were used for the actual email to comcast. The email was being sent to the person quoted above.

This has happened once or twice before but many months apart.

For a span of 3 minutes before Amazon SES paused my account, dozens of such emails were sent.

I am wondering if the hex values represent the alias and somehow they were used rather than the ASCII text.

 

[Paul B]

The post/quote content has no bearing on the email addresses involved.

Emails are sent from the email address entered in the ACP and to the email address registered against the member account.

 

[amirm]

I understand that. But if that text field is corrupted in the same manner as the email alias, then it points to the problem being a bug in Xenforo and not elsewhere.

Let me be clear: hundreds of thousands of emails are sent correctly for months. But suddenly, this bug occurs causing cascade of events with our email provider pausing my account. I am hoping someone can look at the code with the clues I am providing to figure out how it could possibly occur. Without a fix, I am facing this happening over and over again and possibly getting banned altogether from Amazon SES.

Since there is no Abuse handler either in Xenforo or Amazon SES, I am stuck in the middle on this thing with no solution.

 

[amirm]

Turned emails back on and in just a few minutes, the issue came right back with notification to that specific user above! I have shut down his notifications for now and will attempt to retest. Since this problem is now reproduceable, what is the hope that someone from Xenforo can look at this? I have an open ticket but it has gone cold with no response.

Our site is rather large with 2M+ visits a month. Not being able to send email is impacting our operation significantly. As an ex software developer and manager I appreciate that obscure bugs like this are hard to work on. Or get anyone motivated to look at it. But I really need a solution here.

 

[whynot]

Does it happen when your addons are disabled?

 

[amirm]

I haven't tried. I can't figure out why they would only impact one account this way.

 

[Paul B]

For the same reason why you think the core code could.

There have been no other reports of this and it's not a constant error so ruling out third party code is the first step.

The next is to check the server mail log to confirm the email address the emails are being sent to when XF hands them off to the server.

 

[amirm]

OK, you got me on that.

I am going to look into whether there is a log I can extract from Amazon SES. For now, I think the answer to this is no, but is there a log kept by Xenforo on emails it has sent?

 

[Paul B]

No because the mail log is typically stored on the server.

You can though add this to the config.php file which will write all emails to an .eml file:

$c->extend('mailer.transport', function()
{
    return \XF\Mail\Mailer::getTransportFromOption('file', [
        'path' => \XF::getRootDirectory() . '/_temp'
    ]);
});

 

[amirm]

Most excellent. Thank you! Just put that in.

 

[amirm]

Just an update on why I am seeing this problem and others not.

1. We have some very long sticky threads. When someone posts on them, naturally it creates a lot of thread watch notifications.

2. Majority of our members are from US and Comcast is the major (#1?) consumer ISP provider. So naturally a lot of the hits from #1 above go to Comcast.

3. Comcast must have some kind of spam detection based on frequency of emails alone. In the span of 3 minutes until Amazon paused our account, it generated some 50+ Abuse messages. I don't get Abuse messages from hardly any other ISP.

4. Amazon SES guards its IP pool used for SES email service like it is made out of gold. It constantly monitors such events and with slightest hint of an issue, it pauses the account. In my case, I think it reported that Abuse had reached 1.5% of our traffic and that was the reason it paused our account.

5. On the strange IP address in the Abuse message, my contact at Amazon thinks that may be an obfuscation attempt by Comcast to keep the spammer from confirming the email address. If so, then it means I have no chance of finding these accounts to manage them!

What this means is that unless you are situated similarly to me in all of the above, you are not going to see the problem. But the problem is there in our case with no solution at hand.

None of this explains why the message itself has the hex alias instead of actual member alias. But ignoring that, we still have an operational issue that is impacting our core operation.

Some suggestion then:
1. It would be great to have some kind of logging of outbound email addresses and statistics on how many are sent. Outside of this issue, I frequently get complains about not getting an email from the system. Would be great to show that we have indeed sent the message.

2. In the Abuse message above, Comcast sends back the message itself. But I have no way of being able to find out who that message was supposed to go to. Can the message start with: "Dear Member Alias" where that would be the member's actual alias? That would make the message more friendly anyway. And give us a way to track who the bounced/abuse message was supposed to go to.

3. Some kind of rate control on too many Abuse messages. Obviously Amazon SES is doing this for me but they give no warning and just pause or possibly ban the account. Would have been nice in my case to have that set to say, 5 messages a day and have it pause email notifications or something like that. In my situation above, I was sleep while the Abuse messages flowed and Amazon put my account in Pause. It took more than 24 hours for Amazon to lift that Pause by the way.

Anyway, I hope you see where I am going with this. We have an open-loop system with Abuse messages and we somehow need to close that.

 

[FTL]

@amirm you're clearly the victim of automated email abuse detection systems by third parties which are obviously out of your control and I can feel your pain. We can blame the spammers for ruining it for others.

I know that this is a real longshot, but is there any way that you could perhaps contact Comcast and have them whitelist your email address? That would go a long way to solving this problem for you. Perhaps another approach would be to ask AWS to disregard abuse notifications from Comcast. Even if that worked, it's still not as good, since Comcast could end up completely blocking emails from your XF account if they perceive it to be a constant spam source.

As far as that username hex alias, I'm as mystified as you.

 

[jeb35]

Disable all your addons and see if it continues. A crazy addon somewhere seems more likely the cause, member names just don't get changed like that.

 

[amirm]

I know that this is a real longshot, but is there any way that you could perhaps contact Comcast and have them whitelist your email address?

Thank you for very understanding message. Someone working for me in my last corporate life is an executive at Comcast now. He is a different division but I will see if I can get him to get me the correct contact.

 

[amirm]

Disable all your addons and see if it continues. A crazy addon somewhere seems more likely the cause, member names just don't get changed like that.

It may take months for the problem to come back due to conditions I mentioned above. I can't operate the forum without those handful of extensions for that long. Further, when the problem comes back, I will have no more diagnostic information than I do now.

I know it is not something that will be done but a one line check on email address being that long sequence of hex values and throwing an error would go a long way to catching the conditions under which this happens and have some hope of finding/ resolving the bug.

 

[FTL]

Thank you for very understanding message. Someone working for me in my last corporate life is an executive at Comcast now. He is a different division but I will see if I can get him to get me the correct contact.

You're welcome.

Oh wow, that's quite an inside contact you've got there, much better than speaking to their useless "tech support". Good luck with it and please let us know how you get on.

 

[Black Tiger]

Strangely, a different set of hex values were used for the actual email to comcast.

If I'm correct that is not the case. The actual mail to comcast is correct, but the spam report -you- are getting will have these kind of random values and you can only see @provider.com in real names, so in your case @comcast.net.

The cause here is that it seems they most likely are not allowed to send the full mail address to you probably due to privacy reasons.

I had a bunch of these too as hosting provider where the address was for example 2s0f891h3osfA3@ziggo.nl and I couldn't find who was sending the spam reports or where it came from.
It's very hard to track down due to this lack of information. For hotmail/outlook it's a bit easier, as hoster you can register your ip there and have spam reports being send to you as attachment and then the original names are not anonimized anymore.
Problem is that registration at certain sites is only available for hosters, some even only for big hosters/isp's.

So that anonimisation part, is imho done by Comcast and probaly not by any forum software or addon. Just FYI. But one never knows, never say never.
Sometimes privacy is a bless, some other times it's hell.

The spamming could well be caused by some addon or something, I don't know, but I never seen it changing names yet (which does not means its impossible).