What's New Digest 5.0.6

[Sim]

to determine whether it's responsible for the high complaint rate that we've been seeing

FWIW, it is quite likely that WhatsNewDigest is at least indirectly responsible - it is going to be the number one source of bulk email sends from your site and generates a significant volume of email traffic.

If your mailing list isn't already clean and you aren't already restricting who receives the digest emails - you are naturally going to get a lot of complaints or bounces from each send.

Here is a chart of my daily sending reputation on SparkPost - it drops most Sundays when my digest emails go out and I get an increase in bounces/complaints:



I don't think the emails being sent are themselves invalid - they should all have the one-click unsubscribe header added (make sure you are running XF 2.2.15 and WND 5.0.6) - it will be the volume of emails and who they are being sent to that will be the issue.

 

[Stuart Wright]

FWIW, it is quite likely that WhatsNewDigest is at least indirectly responsible - it is going to be the number one source of bulk email sends from your site and generates a significant volume of email traffic.

If your mailing list isn't already clean and you aren't already restricting who receives the digest emails - you are naturally going to get a lot of complaints or bounces from each send.

Here is a chart of my daily sending reputation on SparkPost - it drops most Sundays when my digest emails go out and I get an increase in bounces/complaints:

View attachment 299409

I don't think the emails being sent are themselves invalid - they should all have the one-click unsubscribe header added (make sure you are running XF 2.2.15 and WND 5.0.6) - it will be the volume of emails and who they are being sent to that will be the issue.

Thanks for the reply. Do bounces & spam reports from the digest emails get handled properly?

 

[Sim]

Thanks for the reply. Do bounces & spam reports from the digest emails get handled properly?

It simply uses the standard XF mail sending infrastructure - so any bounces & spam reports that XF can handle are handled for digest emails as well.

 

[Alpha1]

I think it may be interesting to investigate if xenforo core bounce processing handles SES bounces and SES complaints well. Previously it did not. To accurately classify these SES bounces is quite complex due to the diversity . @DragonByte Tech developed accurate processing of SES bounces for me for xf1 to resolve this issue. I do not know how well xf 2.2 processes SES complaints if at all. I know that until recently xenforo didn't handle SES bounces well. I don't know if recent changes have changed this.

I would expect that the high number of bounces as a natural result of a what's new digest coupled with potential inaccurate handling of bounces may impact SES reputation score.

 

[Mouth]

Finally - I don't know if AWS has any kind of suppression list functionality?

Managing lists and subscriptions in Amazon Simple Email Service - Amazon Simple Email Service

Provides information about managing email lists and subscriptions with Amazon SES.

docs.aws.amazon.com

Do bounces & spam reports from the digest emails get handled properly?

I know that until recently xenforo didn't handle SES bounces well. I don't know if recent changes have changed this.

Regardless of this, SES default is to add bouncing and complaint recipients into your account level suppression list, so that any emails sent from XF (or other sources) though your SES account will not be sent/delivered to email addresses that have bounced or complained. Access/review your account (and configuration-sets lists if you want to get sophisticated) suppression list via SES console and Configuration | Supression List



I too would be interested in stats of XF bounce handling when using SES, but feel quite confident it's working well and appropriately. My email bounce log appears to align well with SES suppression list, and activity dates on XF bounce log align well with WND sending dates.

 

[Mouth]

2 feature suggestions....

1. On the /admin.php?tools/whatsnewdigest-test page it would be great if the Test Type: List Users output gave the following:

a. Block header includes count of users, eg. "327 users who will receive the weekly email"

b. Underneath the block container listing users, show the count of excluded users based on options/groups/whatsnewdigest/ settings. Eg.

100 users excluded due to setting Only send to users opting to receive news and update emails
200 users excluded due to setting Recently Active Users Only
10 user excluded due to setting Exclude Discouraged Users

2. Add a /admin.php?logs/whatsnewdigest/ page that gives entries for the cron sending actions. Similar in display to the /admin.php?logs/email-bounces. Eg.

Date	Type	Recipients
30 mins ago	Weekly	327

The recipients hyperlink then gives a list, showing username and email address, of the recipients of that send

Question ...

My /admin.php?options/groups/whatsnewdigest/ default frequency is monthly. Going to /admin.php?tools/whatsnewdigest-test for List Users | Monthly, the output users listed is only 200 shown which is a lot less than actual. Is the output list being constrained (intentionally or bug?) to only 200 users? Does that mean the email would only be sent to those 200 users, or the much larger quantity that applies?

 

[Sim]

My /admin.php?options/groups/whatsnewdigest/ default frequency is monthly. Going to /admin.php?tools/whatsnewdigest-test for List Users | Monthly, the output users listed is only 200 shown which is a lot less than actual. Is the output list being constrained (intentionally or bug?) to only 200 users? Does that mean the email would only be sent to those 200 users, or the much larger quantity that applies?

Yes, it is hard coded to only show the first 200 - because it was intended to be a simple test rather than a comprehensive list and I didn't want to go to the trouble of implementing paging to show more.

I'm not sure what the use-case is for showing more than 200 users here?

More user count stats would definitely be useful.

 

[Mouth]

I'm not sure what the use-case is for showing more than 200 users here?

None really, other than confusion and unsurity. I'd suggest showing none, just the count(s), to be accurate and factual.

 

[Sim]

None really, other than confusion and unsurity. I'd suggest showing none, just the count(s), to be accurate and factual.

It's more intended as a testing tool to ensure that the settings chosen are including the users we expect. I could probably just document it better.

 

[Gsk8]

Hey @Sim I'm still having problems with the digest picking up threads from forum IDs I "excluded" (V. 5.0.6)?

​

 

[Sim]

Hey @Sim I'm still having problems with the digest picking up threads from forum IDs I "excluded" (V. 5.0.6)?

​

Is this in the admin interface or the actual emails going out?

 

[Gsk8]

Is this in the admin interface or the actual emails going out?

This is in both the "test run" and the "test" emails

 

[Gsk8]

@Sim just checking in to see if you had a chance to figure out the above?

 

[Stuart Wright]

@Sim (yes, I'm systematically going through my 110 addons asking the authors when they are releasing a 2.3 update!)
So...
Do you have an ETA please?
Thanks.

 

[Sim]

No ETA - but I will be checking and will post an update about 2.3 compatibility once I've had time to test it

 

[Robert9]

I haven't looked at the code yet, but I assume the emails are sent out once a week?

If that's the case, I have an idea for people using the wrong hosting service. You could divide the digits 0-9 across the weekdays. Or, if you'd prefer to leave people alone on the weekends, you could use five days and ten digits.

For example:

• Users with user_id ending in 0 or 1 would receive emails on Monday.
• Users with user_id ending in 2 or 3 would receive emails on Tuesday.
• And so on.

This approach would reduce the email volume to one-fifth of its current level. If you further split the daily emails into two batches, it could look like this:

• Monday at 2 AM: All user_ids ending in 0.
• Monday at 5 AM: All user_ids ending in 1.

This would further spread out the email load.

If that's still not enough, you could do:

• 0, 1, 2 on Monday at 1 AM, and 3, 4, 5 later on.

You could even distribute them based on high-traffic and low-traffic days.

I find this approach quite appealing!

 

[Mouth]

but I assume the emails are sent out once a week?

The overview and screenshots shows that it can be daily, weekly, or monthly sends.

I find this approach quite appealing!

It's a good approach for rate limiting sends.

 

[Robert9]

It looks like I'll have the same great fun every Monday from now on. I receive an email from a hosting provider telling me that my mailbox has been attacked. Then follows a great explanation, something like this:

Sent 3000
Rejected as spam: 2!
Classified as spam by the provider: 0!
Non-existent: 100!
Spam complaints: 0!

And then: Oh, well, maybe it’s not spam after all, since it was sent through my mailbox, but for security reasons, it’s blocked anyway.

Then I change the password of the mailbox and in XenForo and look forward to next week.

Alternatively, I now have to look at your script and try to implement one of my suggestions, like sending daily on different days of the week:

Monday for user_ids ending in 0, Tuesday 1, Wednesday 2, Thursday 3+4, Friday 5, Saturday 6-7, Sunday 8+9.

 

[Sim]

It looks like I'll have the same great fun every Monday from now on. I receive an email from a hosting provider telling me that my mailbox has been attacked. Then follows a great explanation, something like this:

Sent 3000
Rejected as spam: 2!
Classified as spam by the provider: 0!
Non-existent: 100!
Spam complaints: 0!

And then: Oh, well, maybe it’s not spam after all, since it was sent through my mailbox, but for security reasons, it’s blocked anyway.

Then I change the password of the mailbox and in XenForo and look forward to next week.

Alternatively, I now have to look at your script and try to implement one of my suggestions, like sending daily on different days of the week:

Monday for user_ids ending in 0, Tuesday 1, Wednesday 2, Thursday 3+4, Friday 5, Saturday 6-7, Sunday 8+9.

I thought you were using SparkPost?

 

[Robert9]

For the largest customer’s forum, SparkPost is used, and it takes at least a month or two to clean up the mailing list. But no one really wants to pay 30 euros per month for email sending when they’re running their own servers. However, these servers are somewhat limited by the provider when it comes to sending emails.

For another customer’s forum, I’m dealing with the issues mentioned above. I understand that mail providers don’t want too many emails sent in a short period. Sending 3,000 emails in one night should be spread out over 6 or more hours.

Back in the 1970s—just kidding—there was a program called Gammadyne that allowed you to throttle email delivery, both overall and by provider.

Why don’t we have a function that handles this?