DKIM signing causes serious performance issue when sending emails

[Kirby]

Affected version

2.2.9

In order to test the performance impact of DKIM signing I wrote a little test script:

$dir = __DIR__;
require($dir . '/src/XF.php');

XF::start($dir);

$app = \XF::setupApp('XF\Pub\App');

$options = $app->options();
$mailer = $app->mailer();
$transport = $mailer->getDefaultTransport();

$start = microtime(true);

for ($i = 0; $i < 200; $i++)
{
    $mail = $mailer->newMail();
    $mail->setTo('webmaster@local.host');
    $mail->setContent(
        \XF::phrase('outbound_email_test_subject', ['board' => $options->boardTitle])->render('raw'),
        \XF::phrase('outbound_email_test_body', ['username' => \XF::visitor()->username, 'board' => $options->boardTitle])
    );
    $mail->send($transport, false);
}

$end = microtime(true);

echo("Sending mails took " . ($end-$start) . "\n");

I ran this script in a local VM (located in Europe) via CLI with three different setups:

1. DKIM signing disabled
2. DKIM signing enabled, all storage local
3. DKIM signing enabled, internal-data configured to use an Amazon S3 Bucket in US-West

In all three setups actual email sending was disabled via

$config['enableMail'] = false;

With setup 1) it took around 800ms to generate the emails.
With setup 2) it took around 1s to generate the emails which seems plausible and quite acceptable (for a pretty limited VM).
With setup 3) however the process took well over 190 seconds to complete.

The root cause for this massive slowdown seems to be

$dkimOptions = \XF::options()->emailDkim;
if ($dkimOptions['enabled']
    && extension_loaded('openssl')
    && $dkimOptions['domain'] == substr(strrchr($this->getFromAddress(), '@'), 1)
)
{
    $keyFile = null;

    try
    {
        $path = 'internal-data://keys/' . $dkimOptions['privateKey'];
        $keyFile = \XF::fs()->read($path);
    }
    catch (\Exception $e)
    {
        \XF::logError("Email DKIM: Key not found at $path");
    }

    if ($keyFile)
    {
        $message->attachSigner(new \Swift_Signers_DKIMSigner($keyFile, $dkimOptions['domain'], 'xenforo'));
    }
}

in XF\Mail\Mail::send.

This effecively reads the key for every single mail being send, in case of internal-datsa using a remote storage (like S3) this is getting really slow as the file has to be dowoleded over and over again.

Moving signer initialization to the container and changing above code to smth. like

$dkimOptions = \XF::options()->emailDkim;

if (
        $dkimOptions['enabled']
        && $dkimOptions['domain'] == substr(strrchr($this->getFromAddress(), '@'), 1)
)
{
    $dkimSigner = \XF::app()->get('mailer.dkimsigner');

    if ($dkimSigner)
    {
            $message->attachSigner($dkimSigner);
    }
}

reduced the runtime for setup 3) to an acceptable value of about 2.7 seconds.

 

[digitalpoint]

Honestly, even a local filesystem probably isn't the best place for the DKIM keys. It more or less never changes, isn't very big and needs to be read a LOT (every time you send an email). Even if it exists in the local filesystem, if I send 50,000 emails, do I really want to be doing 50,000 filesystem stat/reads to get the same thing each time? Not really. It's exponentially worse when you are using S3 or some other network-based file system (as you noticed).

My original suggestion for the DKIM signer had the key stored in normal XF options (small, rarely changes, accessed a lot): https://xenforo.com/community/threads/signing-of-outgoing-emails-with-dkim.203563/

That being said, an argument could also be made that if a site is "advanced enough" to be using S3 (or something similar), they are hopefully also advanced enough to be doing DKIM signing at the MTA level rather than the application level. Although there are going to be cases where someone simply can't do it at the MTA level because they aren't the administrator of their server.

But yes... reading the key from disk (or network) every time you send an email probably isn't the best idea.

 

[duderuud]

There are plenty of hosters where you can create a DKIM key (or create a cname to the keys from the hoster) you can import into your dns settings so that shouldn't be a big problem.

 

[Black Tiger]

Correct. Normally the mailserver of the host takes care of DKIM signing, loads of hosters provide SPF, DKIM and DMARC for their customers.