XF 2.2 Xenforo sent 400.000 emails

[Salamanca]

Hi there,

We have been running Xenforo v2.2.3 Patch 1 for more than a year now.
Our normal email daily sends are 1000 to 2000 emails. All of them are notifications for subscriptions or for new conversations started.

Yesterday, all of sudden, we noticed that Xenforo had sent 400.000 emails via our email smtp provider (Sendgrid) in just a few hours.
These were all duplicates of the same message. For example, a notification email "You have a new private message" was sent to some users 20.000 times (each).

Horror story, really.

We cleared the SMTP details in Xenforo admin, and also deleted the sendgrid API key, just in case.

After a bit of investigation, we confirmed that all those emails were indeed sent by Xenforo. They were typical Xenforo messages, but sent thousand of times.

We haven't however found the route cause of this.

I cannot think of anything else other than a nasty bug OR a xenforo exploit (hack).

Does this ring a bell to anyone?

Is this a bug fixed on later versions that could explain this?

Thanks!
Nick

 

[Chris D]

Yesterday, all of sudden, we noticed that Xenforo had sent 400.000 emails via our email smtp provider (Sendgrid) in just a few hours.
These were all duplicates of the same message. For example, a notification email "You have a new private message" was sent to some users 20.000 times (each).

Can you be more specific about the content of the email?

"Private message" isn't terminology that we use in XenForo. We do send emails when a conversation is started with a user but, naturally, it is only triggered when the conversation is created so I can't imagine under what circumstances that would be sent repeatedly.

 

[Salamanca]

Hi,

Some of them were email notifications of conversation started, other were thread or forum subscriptions.

Thanks
N.

 

[Chris D]

Ok but what do they pertain to exactly? Do they pertain to real conversations that were started? Were the conversations actually started at that time or some time before?

Like I said, the code doesn't just decide emails for no reason. They are sent the very moment that the new conversation is started. There is a queuing system but clearly emails haven't been stuck in a queue as you've seen 1000-2000 emails sent per day.

This really needs a lot more investigation as to what triggered these which, if possible, will involve you looking specifically at what the emails were, who they were sent to, when they were sent, and what content specifically they related to.

 

[JulianD]

One day I wanted to see how many emails were being sent from my forum so I created this addon. It's a very simple and incomplete addon that inserts a database record every time an email is sent and then displays it in the statistics menu. If you want to compare the number of actual emails being sent by your forum installation, you might want to check it out.

Also, you can review if the Swift Mailer id at the bottom of the source of each mail is the same for the duplicate emails or if it's different. If it's different, then these emails probably came from your forum. If they are the same, they got duplicated somewhere else. The Swift Mailer id looks like this: --=swift_1636723780_e851e6ed72d4fdf0c6d5662abbb6f370=--

Hope it helps.

 

[ivanr]

I am investigating this really nasty incident and it was bad. We have an addon that logs sent emails (time, subject, address) and you can see number of emails sent to a user. Its all "separate" emails but with same origin, top 3 are for example, notification about new conversation. Basically, looks like there was a loop that sent one notification email 170k times to one user's email. Weird thing is that I wasnt able to find conversation by subject. Other remarks are that there are other types of notifications, watched thread etc. Addon we are using is simple and it properly extends \XF\Mail\Mailer class and just logs needed data after parent sent the email.
Also, the incident happend during only one day, before that things were normal for months and normal now. Im trying to log additional data from swift transport in case it happens again so i could find anything that would tell me why or where it happend. My guess is either nasty bug or some kind of exploit/attack.
Any additional ideas on how to figure this out is greatly appreciated. Thanks!

 

[Salamanca]

Hi,

Is there any bug fixed lately that could explain this?

Thanks.

 

[Onlyme]

Not an addon issue?

Something similar happened to me in the past with an addon. Andys email log, I didn't read correctly that smtp isn't supported.

 

[Chris D]

Hi,

Is there any bug fixed lately that could explain this?

Thanks.

No. I hesitate to repeat myself but essentially with the way the code works, there's not really an opportunity for XF to just resend emails repeatedly.

Not an addon issue?

Something similar happened to me in the past with an addon. Andys email log, I didn't read correctly that smtp isn't supported.

This?

Email log

Premium upgrade: This XF2 add-on along with the entire collection can be purchased for $35.00 USD. Your Premium upgrade will allow you to download as many XF2 add-ons as you like for one year. Please see the entire collection located in the...

xenforo.com

What was the issue exactly? I don't see any mention of incompatibility with SMTP nor do I understand why it would be a problem.

 

[Onlyme]

No. I hesitate to repeat myself but essentially with the way the code works, there's not really an opportunity for XF to just resend emails repeatedly.


This?

Email log

Premium upgrade: This XF2 add-on along with the entire collection can be purchased for $35.00 USD. Your Premium upgrade will allow you to download as many XF2 add-ons as you like for one year. Please see the entire collection located in the...

xenforo.com

What was the issue exactly? I don't see any mention of incompatibility with SMTP nor do I understand why it would be a problem.

It was on the earlier version: email log 1.6. Maybe it's fix now I haven't used it since.

It sent thousands of duplicate emails. Never went into depth why, just stopped using it.

​

 

[Chris D]

So when you were running XF1?

 

[Onlyme]

So when you were running XF1?

Yup. Didn't realise it was that long ago. Probably shouldn't have mentioned it

 

[Chris D]

Well, it might be relevant if that add-on is in use and the bug still exists in a new version. So definitely worth mentioning.