Fixed Email bouncing error

@Chris D I've attached a SQL file for inserting the broken entry into the bounce log. I don't think there's any PII / non-ephemeral keys in the message, and the email appears to be attached to a spam bot so I don't think I'm leaking anyone's valid emails. If I'm wrong, feel free to edit the message after you have the info you need :)

Error Message:
Zend\Mail\Exception\RuntimeException: Bounce message processing failed: Malformed header detected
src/vendor/zendframework/zend-mail/src/Headers.php:88
Generated by: Unknown account - Aug 23, 2018 at 8:23 PM

Stack Trace:
#0 src/vendor/zendframework/zend-mime/src/Decode.php(141): Zend\Mail\Headers::fromString('Return-Path: <>...', '\n')
#1 src/vendor/zendframework/zend-mail/src/Storage/Part.php(99): Zend\Mime\Decode::splitMessage('Return-Path: <>...', 'Return-Path: <>...', '------=_Part_38...', '\n', false)
#2 src/vendor/zendframework/zend-mail/src/Storage/Message.php(54): Zend\Mail\Storage\Part->__construct(Array)
#3 src/XF/EmailBounce/Parser.php(169): Zend\Mail\Storage\Message->__construct(Array)
#4 src/XF/EmailBounce/Processor.php(69): XF\EmailBounce\Parser->parseMessage('Return-Path: <>...')
#5 src/XF/EmailBounce/Processor.php(53): XF\EmailBounce\Processor->processMessage('Return-Path: <>...')
#6 src/XF/Job/EmailBounce.php(27): XF\EmailBounce\Processor->processFromStorage(Object(Zend\Mail\Storage\Imap), 8)
#7 src/XF/Job/Manager.php(241): XF\Job\EmailBounce->run(8)
#8 src/XF/Job/Manager.php(187): XF\Job\Manager->runJobInternal(Array, 8)
#9 src/XF/Job/Manager.php(76): XF\Job\Manager->runJobEntry(Array, 8)
#10 job.php(15): XF\Job\Manager->runQueue(false, 8)
#11 {main}

raw_message
Code:
Return-Path: <>
Delivered-To: bounce@dragonbytetech.com
Received: from web02.dragonbyte-tech.com
    by web02.dragonbyte-tech.com with LMTP id QOusMegHf1veawAAmma+EA
    for <bounce@dragonbytetech.com>; Thu, 23 Aug 2018 20:15:52 +0100
Return-path: <>
Envelope-to: bounce@dragonbytetech.com
Delivery-date: Thu, 23 Aug 2018 20:15:52 +0100
Received: from a7-14.smtp-out.eu-west-1.amazonses.com ([54.240.7.14]:37364)
    by web02.dragonbyte-tech.com with esmtps (TLSv1.2:ECDHE-RSA-AES128-SHA256:128)
    (Exim 4.91)
    id 1fsv5E-000794-0G
    for bounce@dragonbytetech.com; Thu, 23 Aug 2018 20:15:52 +0100
From: MAILER-DAEMON@eu-west-1.amazonses.com
To: bounce@dragonbytetech.com
Message-ID: <0102016568364366-a3538a5d-dbce-449b-9af3-efc8a9606a34-000000@eu-west-1.amazonses.com>
Subject: Delivery Status Notification (Failure)
MIME-Version: 1.0
Content-Type: multipart/report; 
    boundary="----=_Part_382740_1809377826.1535051711352"; 
    report-type=delivery-status
Date: Thu, 23 Aug 2018 19:15:11 +0000
X-SES-Outgoing: 2018.08.23-54.240.7.14
X-Spam-Status: No, score=0.0
X-Spam-Score: 0
X-Spam-Bar: /
X-Ham-Report: Spam detection software, running on the system "web02.dragonbyte-tech.com",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 root\@localhost for details.
 
 Content preview:  An error occurred while trying to deliver the mail to the
   following recipients: dguig.abdelaziz@yahoo.com lery, in order to complete
    your registration or reactivate your account at DragonByte Tech | X
 
 Content analysis details:   (0.0 points, 5.0 required)
 
  pts rule name              description
 ---- ---------------------- --------------------------------------------------
 -0.0 RCVD_IN_DNSWL_NONE     RBL: Sender listed at http://www.dnswl.org/, no
                             trust
                             [54.240.7.14 listed in list.dnswl.org]
  0.0 HTML_MESSAGE           BODY: HTML included in message
 -0.0 BAYES_20               BODY: Bayes spam probability is 5 to 20%
                             [score: 0.1726]
X-Spam-Flag: NO

------=_Part_382740_1809377826.1535051711352
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Description: Notification

An error occurred while trying to deliver the mail to the following recipients:
dguig.abdelaziz@yahoo.com
------=_Part_382740_1809377826.1535051711352
Content-Type: message/delivery-status
Content-Transfer-Encoding: 7bit
Content-Description: Delivery Status Notification

Reporting-MTA: dsn; a4-1.smtp-out.eu-west-1.amazonses.com

Action: failed
Final-Recipient: rfc822; dguig.abdelaziz@yahoo.com
Diagnostic-Code: smtp; 554 delivery error: dd This user doesn't have a yahoo.com account (dguig.abdelaziz@yahoo.com) [0] - mta4348.mail.ne1.yahoo.com
Status: 5.3.0


------=_Part_382740_1809377826.1535051711352
Content-Type: message/rfc822
Content-Description: Undelivered Message

Message-ID: <0102016568363c3e-3d406e9f-3e9c-436f-b0dd-3217ee47f9cb-000000@eu-west-1.amazonses.com>
Date: Thu, 23 Aug 2018 19:15:09 +0000
Subject: DragonByte Tech | XenForo and vBulletin Mods & Addons - Account
 confirmation required
From: DragonByte Tech | XenForo and vBulletin Mods & Addons
 <dbtech@dragonbyte-tech.com>
To: lery <dguig.abdelaziz@yahoo.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="_=_swift_v4_1535051709_d92649e2095eedb18835fa1aec9983a5_=_"
X-To-Validate: f06cdef0+dguig.abdelaziz@yahoo.com
X-SES-Outgoing: 2018.08.23-54.240.4.3
Feedback-ID: 1.eu-west-1.rqFLe/K6Rujqlv0M0C8a4TCJipFLr43+F05d3mJRahs=:AmazonSES
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple;
    s=uku4taia5b5tsbglxyj6zym32efj7xqv; d=amazonses.com; t=1535051709;
    h=Message-ID:Date:Subject:From:To:MIME-Version:Content-Type:Feedback-ID;
    bh=mCG0qMdQ/AUskt/Z/ABLfJl5lil6VxPg4UB+ky7cqMY=;
    b=QjLq5b76bEbznuPhO9fupk97DdUtOEvBA8oDnbKX9nQKxddC+/7/h8T9RVugLvsI
    BDMcUOgem4VWZ8KgNxihM8fvn4RBQDYezEFn+UzRU6ZFHmBEaE6kOdKpH58yphBXCWh
    mutZyghTbBUVP9BuNAIKR4FLFxhK8DjI8j6oX6Gg=


--_=_swift_v4_1535051709_d92649e2095eedb18835fa1aec9983a5_=_
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

lery, in order to complete your registration or reactivate your account at =
DragonByte Tech | XenForo and vBulletin Mods & Addons (https://www.dragonby=
te-tech.com/), you need to confirm your email address by clicking the butto=
n below.

Confirm your email (https://www.dragonbyte-tech.com/account-c=
onfirmation/lery.21958/email?c=3DfhxRsJXN6rsldFz8)

-------------------=
----------

Visit DragonByte Tech | XenForo and vBulletin Mods & Addons=
: https://www.dragonbyte-tech.com/

--_=_swift_v4_1535051709_d92649e2095eedb18835fa1aec9983a5_=_
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE html>
<html lang=3D"en-US" dir=3D"LTR">
<head>
=09<meta htt=
p-equiv=3D"Content-Type" content=3D"text/html; charset=3Dutf-8">
=09<base=
 href=3D"https://www.dragonbyte-tech.com/">
=09<meta name=3D"viewport" co=
ntent=3D"width=3Ddevice-width, initial-scale=3D1">
=09<meta http-equiv=3D=
"X-UA-Compatible" content=3D"IE=3Dedge">
=09<meta name=3D"format-detectio=
n" content=3D"telephone=3Dno">
=09<title>DragonByte Tech | XenForo and vB=
ulletin Mods &amp; Addons - Account confirmation required</title>
</head>=

<body dir=3D"LTR" leftmargin=3D"0" topmargin=3D"0" marginwidth=3D"0" mar=
ginheight=3D"0" style=3D"margin: 0; padding: 0; word-wrap: break-word; -ms-=
text-size-adjust: 100%; -webkit-text-size-adjust: 100%; background-color: #=
f0f1f3; font-size: 15px; font-family: 'Segoe UI','Helvetica Neue',Helvetica=
,Roboto,Oxygen,Ubuntu,Cantarell,'Fira Sans','Droid Sans',sans-serif; line-h=
eight: 1.4; color: #141414;">

<table id=3D"bodyTable" border=3D"0" wid=
th=3D"100%" height=3D"100%" cellpadding=3D"0" cellspacing=3D"0" style=3D"bo=
rder-spacing: 0; mso-table-lspace: 0pt; mso-table-rspace: 0pt; border-colla=
pse: collapse; height: 100% !important; width: 100% !important; margin: 0; =
padding: 0; background-color: #f0f1f3;">
<tr>
=09<td align=3D"center" v=
align=3D"top" id=3D"bodyTableContainer" style=3D"border-collapse: collapse;=
 background-color: #f0f1f3;">
=09=09<table border=3D"0" width=3D"600" cel=
lpadding=3D"0" cellspacing=3D"0" class=3D"container" dir=3D"LTR" style=3D"b=
order-spacing: 0; mso-table-lspace: 0pt; mso-table-rspace: 0pt; border-coll=
apse: collapse; width: 100%; max-width: 600px;">
=09=09<tr>
=09=09=09<t=
d class=3D"header" align=3D"center" valign=3D"top" style=3D"border-collapse=
: collapse; color: #444e50; padding: 6px 10px; border-top-left-radius: 4px;=
 border-top-right-radius: 4px; font-family: 'Segoe UI','Helvetica Neue',Hel=
vetica,Roboto,Oxygen,Ubuntu,Cantarell,'Fira Sans','Droid Sans',sans-serif; =
font-size: 24px; line-height: 1.4;">
=09=09=09=09<a href=3D"https://www.d=
ragonbyte-tech.com/" style=3D"color: #444e50; text-decoration: none;">Drago=
nByte Tech | XenForo and vBulletin Mods &amp; Addons</a>
=09=09=09</td>=

=09=09</tr>
=09=09<tr>
=09=09=09<td class=3D"content" align=3D"left"=
 valign=3D"top" style=3D"border-collapse: collapse; background-color: #fefe=
fe; border-radius: 2px; color: #141414; padding: 10px; font-size: 15px; fon=
t-family: 'Segoe UI','Helvetica Neue',Helvetica,Roboto,Oxygen,Ubuntu,Cantar=
ell,'Fira Sans','Droid Sans',sans-serif; line-height: 1.4;">

<p style=
=3D"margin-top: 0;">lery, in order to complete your registration or reactiv=
ate your account at <a href=3D"https://www.dragonbyte-tech.com/" style=3D"c=
olor: #2577b1; text-decoration: none;">DragonByte Tech | XenForo and vBulle=
tin Mods &amp; Addons</a>, you need to confirm your email address by clicki=
ng the button below.</p>

<p style=3D"margin-bottom: 0;"><a href=3D"htt=
ps://www.dragonbyte-tech.com/account-confirmation/lery.21958/email?c=3DfhxR=
sJXN6rsldFz8" class=3D"button" style=3D"color: #f0f1f3; text-decoration: no=
ne; display: inline-block; padding: 5px 10px; background-color: #2b3335; bo=
rder: none; border-radius: 4px; font-size: 13px;">Confirm your email</a></p=
>

=09=09=09</td>
=09=09</tr>
=09=09<tr>
=09=09=09<td class=3D"fo=
oter" align=3D"center" valign=3D"top" style=3D"border-collapse: collapse; p=
adding: 6px 10px; text-align: center; color: #8c8c8c; font-size: 13px; font=
-family: 'Segoe UI','Helvetica Neue',Helvetica,Roboto,Oxygen,Ubuntu,Cantare=
ll,'Fira Sans','Droid Sans',sans-serif; line-height: 1.4;">
=09=09=09=09<=
div><a href=3D"https://www.dragonbyte-tech.com/" style=3D"color: #8c8c8c; t=
ext-decoration: underline;">Visit DragonByte Tech | XenForo and vBulletin M=
ods &amp; Addons</a></div>

=09=09=09=09
=09=09=09</td>
=09=09</tr>=

=09=09</table>
=09</td>
</tr>
</table>

</body>
</html>

--_=_swift_v4_1535051709_d92649e2095eedb18835fa1aec9983a5_=_--


------=_Part_382740_1809377826.1535051711352--
 

Attachments

Also, for anyone interested, this is the custom action I wrote in order to fix the bounced email:

PHP:
<?php

namespace DBTech\Custom\Admin\Controller;

use XF\Admin\Controller\AbstractController;
use XF\Mvc\FormAction;
use XF\Mvc\ParameterBag;

/**
 * Class FixEmailBounce
 * @package DBTech\Custom\Admin\Controller
 */
class FixEmailBounce extends AbstractController
{
    /**
     * @param $action
     * @param ParameterBag $params
     * @throws \XF\Mvc\Reply\Exception
     */
    protected function preDispatchController($action, ParameterBag $params)
    {
        $this->assertAdminPermission('user');
    }
    
    /**
     * @return \XF\Mvc\Reply\Message
     * @throws \XF\PrintableException
     */
    public function actionIndex()
    {
        $bounceContainer = \XF::app()->bounce();
        
        $processor = $bounceContainer->processor();
        
        /** @var \XF\Repository\EmailBounce $bounceRepo */
        $bounceRepo = $this->repository('XF:EmailBounce');
        $finder = $bounceRepo->findEmailBounceLogsForList()
            ->where('user_id', NULL)
            ->order('bounce_id', 'ASC');
        
        $found = $finder->total();
        
        /** @var \XF\Entity\EmailBounceLog $bounce */
        foreach ($finder->fetch() as $bounce)
        {
            $parsed = \imap_rfc822_parse_headers($bounce->raw_message);
            
            if (count($parsed->to) == 1
                && ($parsed->to[0]->host == 'dragonbyte-tech.com' || $parsed->to[0]->host == 'dragonbytetech.com')
            )
            {
                continue;
            }
            else
            {
                if ($parsed->to[0]->host == 'dragonbyte-tech.com' || $parsed->to[0]->host == 'dragonbytetech.com')
                {
                    $toEmail = $parsed->to[1]->mailbox . '@' . $parsed->to[1]->host;
                }
                else
                {
                    $toEmail = $parsed->to[0]->mailbox . '@' . $parsed->to[0]->host;
                }
            }
            
            if ($toEmail)
            {
                /** @var \XF\Entity\User $user */
                $user = $this->app->em()
                    ->findOne('XF:User', ['email' => $toEmail])
                ;
            }
            else
            {
                $user = null;
                
                echo "Missing email:<br />";
                \XF::dump($parsed);
                die();
            }
            
            $action = null;
            
            if ($user)
            {
                $action = $processor->takeBounceAction($user, 'hard', $bounce->log_date);
                
                $bounce->bulkSet([
                    'message_type' => 'bounce',
                    'action_taken' => $action ?: '',
                    'user_id'      => $user ? $user->user_id : null,
                    'recipient'    => $user->email,
                ]);
                $bounce->save();
            }
            else
            {
                echo "Missing user:<br />";
                \XF::dump($parsed);
                die();
            }
        }
        
        /** @var \XF\Repository\EmailBounce $bounceRepo */
        $bounceRepo = $this->repository('XF:EmailBounce');
        $finder = $bounceRepo->findEmailBounceLogsForList()
            ->where('user_id', NULL);
        
        return $this->message('Parsed ' . $found . ' emails. There are ' . $finder->total() . ' outstanding bounced emails.');
    }
}


Fillip
 
Actually, it may not be Spamassassin but it looks to be caused by whatever you have which adds the "X-Ham-Report" header.
 
@DragonByte Tech and @Tom are you both running "Spam Assassin" on your server/mail server?
I believe so. It's a WHM / cPanel server, using these settings:

BuGIRLp.png


Global settings:

8ou4K7l.png


I will try disabling SA for that particular account, as the domain we use for bounce handling is not used for anything other than that.

If so, can you provide details on which version etc. if available?
I've no idea how to find the version in WHM 🤔

We're running the latest WHM (v74.0.5 on CentOS 7.5).

---
I'll try registering with a fake email and see whether it processes this bounce successfully.


Fillip
 
Actually, it may not be Spamassassin but it looks to be caused by whatever you have which adds the "X-Ham-Report" header.
Confirmed that SpamAssassin is a potential culprit for this:

qgDsBqv.png


It should still be fixed just for maximum compatibilityness sake, but at least I can bin my custom code now 😄


Fillip
 
Well, strictly speaking, Spamassassin isn't actually doing anything wrong. It mostly seems to be an issue with Zend Mail and their over-zealousness when parsing the mail headers.

It's quite possibly a major PITA to fix.
 
Well, it became less of a PITA because we've forked Zend Mail and we're now using that in XF 2.0. We've also submitted an issue for consideration for future versions of Zend Mail which we should be able to use in XF 2.1.

Either way, we'll call this fixed for the next release.
 
I'm having a similar issue as well, and we are also running SA on our server.
Code:
Zend\Mail\Exception\RuntimeException: Bounce message processing failed: Line "Hello," does not match header format! src\vendor-patch\Zend\Mail\Headers.php:110

Code:
Stack trace
#0 src\vendor\zendframework\zend-mime\src\Decode.php(149): Zend\Mail\Headers::fromString('Content-Type: t...', '
')
#1 src\vendor\zendframework\zend-mime\src\Decode.php(80): Zend\Mime\Decode::splitMessage('Content-Type: t...', 'Content-Type: t...', 'Unfortunately y...', '
')
#2 src\vendor\zendframework\zend-mail\src\Storage\Part.php(193): Zend\Mime\Decode::splitMessageStruct('--=============...', '===============...')
#3 src\vendor\zendframework\zend-mail\src\Storage\Part.php(255): Zend\Mail\Storage\Part->cacheContent()
#4 src\vendor\zendframework\zend-mail\src\Storage\Part.php(470): Zend\Mail\Storage\Part->countParts()
#5 src\XF\EmailBounce\Parser.php(243): Zend\Mail\Storage\Part->rewind()
#6 src\XF\EmailBounce\Parser.php(203): XF\EmailBounce\Parser->extractContent(Object(Zend\Mail\Storage\Message), Object(XF\EmailBounce\ParseResult))
#7 src\XF\EmailBounce\Processor.php(92): XF\EmailBounce\Parser->parseMessage(Object(Zend\Mail\Storage\Message))
#8 src\XF\EmailBounce\Processor.php(68): XF\EmailBounce\Processor->processMessage('Return-Path: <>...')
#9 src\XF\Job\EmailBounce.php(27): XF\EmailBounce\Processor->processFromStorage(Object(Zend\Mail\Storage\Pop3), G)
#10 src\XF\Job\Manager.php(253): XF\Job\EmailBounce->run(G)
#11 src\XF\Job\Manager.php(195): XF\Job\Manager->runJobInternal(Array, G)
#12 src\XF\Job\Manager.php(79): XF\Job\Manager->runJobEntry(Array, G)
#13 job.php(42): XF\Job\Manager->runQueue(false, 8)
#14 {main}

Code:
Request state
array(4) {
  ["url"] => string(8) "/job.php"
  ["referrer"] => string(79) "https://forums.fordthunderbirdforum.com/threads/1994-supercharger-rebuild.8606/"
  ["_GET"] => array(0) {
  }
  ["_POST"] => array(0) {
  }
}



Also got this:

Code:
ErrorException: [E_NOTICE] Undefined variable: content src\XF\EmailBounce\Processor.php:64

Code:
Stack trace
#0 src\XF\EmailBounce\Processor.php(64): XF::handlePhpError(8, '[E_NOTICE] Unde...', 'D:\\cuswebs\\www1...', 64, Array)
#1 src\XF\Job\EmailBounce.php(27): XF\EmailBounce\Processor->processFromStorage(Object(Zend\Mail\Storage\Pop3), G)
#2 src\XF\Job\Manager.php(253): XF\Job\EmailBounce->run(G)
#3 src\XF\Job\Manager.php(195): XF\Job\Manager->runJobInternal(Array, G)
#4 src\XF\Job\Manager.php(79): XF\Job\Manager->runJobEntry(Array, G)
#5 job.php(42): XF\Job\Manager->runQueue(false, 8)
#6 {main}

Code:
Request state
array(4) {
  ["url"] => string(8) "/job.php"
  ["referrer"] => string(73) "https://forums.fordthunderbirdforum.com/thunderbird-photos/no-title.3774/"
  ["_GET"] => array(0) {
  }
  ["_POST"] => array(0) {
  }
}
 
Anybody know what the current status of this is? I'm starting to get ACP errors as well from the "X-HAM-REPORT" header in bounced emails. :(
 
Top Bottom