Fixed Non-ASCII characters not supported in local-part

[Walter]

Affected version

2.28 Patch 1

A user changed his email address to something stupid like ihatetheadmin@blöd.de ("blöd" is German for stupid), an obviously not existing email address. Now I get several errors from Swift telling me that umlauts are not allowed in an email address. It would be better to prevent umlauts in the user interface when entering an email.

Swift_AddressEncoderException: Email to xxxxx@blöd.xxx failed: Non-ASCII characters not supported in local-part src/vendor/swiftmailer/swiftmailer/lib/classes/Swift/AddressEncoder/IdnAddressEncoder.php:40

src/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Mime/Headers/PathHeader.php(132): Swift_AddressEncoder_IdnAddressEncoder->encodeString('bounce+621802e1...')
#1 src/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Mime/SimpleHeaderSet.php(309): Swift_Mime_Headers_PathHeader->getFieldBody()
#2 src/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Mime/SimpleMimeEntity.php(529): Swift_Mime_SimpleHeaderSet->toString()
#3 src/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Mime/SimpleMessage.php(604): Swift_Mime_SimpleMimeEntity->toByteStream(Object(Swift_Transport_StreamBuffer))
#4 src/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Message.php(162): Swift_Mime_SimpleMessage->toByteStream(Object(Swift_Transport_StreamBuffer))
#5 src/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Transport/SendmailTransport.php(129): Swift_Message->toByteStream(Object(Swift_Transport_StreamBuffer))
#6 src/XF/Mail/Mailer.php(306): Swift_Transport_SendmailTransport->send(Object(Swift_Message), Array)
#7 src/XF/Mail/Queue.php(148): XF\Mail\Mailer->send(Object(Swift_Message), Object(Swift_SendmailTransport), Array)
#8 src/XF/Job/MailQueue.php(12): XF\Mail\Queue->run(7.90058)
#9 src/XF/Job/Manager.php(260): XF\Job\MailQueue->run(7.90058)
#10 src/XF/Job/Manager.php(202): XF\Job\Manager->runJobInternal(Array, 7.90058)
#11 src/XF/Job/Manager.php(86): XF\Job\Manager->runJobEntry(Array, 7.90058)
#12 job.php(43): XF\Job\Manager->runQueue(false, 8)
#13 {main}

 

[Kirby]

This should be adressed ASAP

$email = 'someone@♥local.host';

if (filter_var($email, FILTER_VALIDATE_EMAIL))
{
    echo('PHP: E-Mail is valid');
}
else
{
    echo('PHP: E-Mail is NOT valid!');
}

echo("\n");

$startTime = microtime(true);
$phpVersion = phpversion();
if (version_compare($phpVersion, '7.0.0', '<'))
{
        die("PHP 7.0.0 or newer is required. $phpVersion does not meet this requirement. Please ask your host to upgrade PHP.");
}

$dir = __DIR__;
require($dir . '/src/XF.php');

XF::start($dir);

$validator = \XF::app()->validator('Email');

if ($validator->isValid($email))
{
    echo('XF: E-Mail is valid');
}
else
{
    echo('XF: E-Mail is not valid!');
}

echo("\n");

Output

PHP: E-Mail is NOT valid!
XF: E-Mail is valid

 

[MySiteGuy]

Technically, Xenforo allowing it is correct. And PHP and Swift are wrong.

Email characters are not limited to 7 bit ASCII characters, haven't been for a long time, and most Unicode characters are allowed per RFC 6532

 

[Kirby]

I tend to disagree

While you are correct that it would be valid per RFC 6532 3.2, I'd expect that most servers out there only support RFC 5322 (or maybe even just RFC 822).

So even if XenForo "is correct" and it would pass SwiftMailer (more likely Symfony Mailer for XF 2.3+, which also wouldn't accept an address like in my example) emails wouldn't be accepted by many (most?) mailservers.

Just an example:

> telnet gmail-smtp-in.l.google.com 25
Trying 74.125.133.26...
Connected to gmail-smtp-in.l.google.com.
Escape character is '^]'.
220 mx.google.com ESMTP b15-20020a05600018af00b00226df33238asi825462wri.858 - gsmtp
HELO mail.example.com
250 mx.google.com at your service
MAIL FROM: <info@example.com>
250 2.1.0 OK b15-20020a05600018af00b00226df33238asi825462wri.858 - gsmtp
RCPT TO: <someone@♥local.host>
555 5.5.2 Syntax error. b15-20020a05600018af00b00226df33238asi825462wri.858 - gsmtp

So what's the point of allowing such addresses in first place if it's not realistically possible to send email there?
This just creates support problems.

 

[Kirby]

Related issue: https://xenforo.com/community/threads/idn-for-emails-is-not-handled-consistently.208749/

 

[MySiteGuy]

Gmail does support non-Latin characters and has since 2014 ( https://blog.google/products/gmail/a-first-step-toward-more-global-email/ ).

Your example is not sending the RCPT TO address correctly for non-Latin character characters. For example someone@♥local.host is sent perfectly to gmail from Thunderbird. It has to be surrounded by encoding information which states the character set. I don't have the format right off hand, but I do recall seeing an example on stackexchange or one of the similar tech discussion sites.

I think Xenforo should not strip out support for more than half the world's languages which are widely used outside of English first countries. Rather, the solution should be to properly implement the sending standard, which actually is supported by most widely used email services and server software these days.

 

[Kirby]

Gmail does support non-Latin characters and has since 2014 ( https://blog.google/products/gmail/a-first-step-toward-more-global-email/ ).

I wasn't talking about non-ascii charaters in email adresses but about a specific character in the domain.

As per IDNA 2003/2008 the character ♥ (Unicode U+2665) is disallowed:

https://www.icann.org/en/system/files/files/idna-protocol-2003.txt

2665 : DISALLOWED : BLACK HEART SUIT

So while an address someone@♥local.host is valid from SMTP protocol view (for a mailerserver supporting SMTPUTF8 extension, eg RFC eg. 6531) such a domain name remains "invalid" for general purpose use (eg. on the Internet) as it wouldn't be possible to register a domain i♥unicode.com

I think Xenforo should not strip out support for more than half the world's languages which are widely used outside of English first countries. Rather, the solution should be to properly implement the sending standard, which actually is supported by most widely used email services and server software these days.

I already does that for the local part /mailbox (as the unterlaying transports do not generally support that):
XF\Validator\Email::isValid

public function isValid($value, &$errorKey = null)
{
    if ($value === '')
    {
        if ($this->options['allow_empty'])
        {
            return true;
        }

        $errorKey = 'empty';
        return false;
    }

    $validator = new EmailValidator();
    if (!$validator->isValid($value, new RFCValidation()))
    {
        $errorKey = 'invalid';
        return false;
    }

    list ($local, $domain) = explode('@', $value);

    if (preg_match('/[^\x00-\x7F]/', $local))
    {
        // Swiftmailer will error if it runs into this
        $errorKey = 'invalid';
        return false;
    }

This specific issue arises if VERP is enabled and the recpient domain contains such non-ascii characters as the domain becomes part of localpart / mailbox.

The fix here would be to properly IDN-encode the email address first before applying VERP.

But even if this is fixed IMHO XenForo should handle IDN properly and not accept email addresses like someone@i♥unicode.com

 

[PaulB]

It has to be surrounded by encoding information which states the character set. I don't have the format right off hand, but I do recall seeing an example on stackexchange or one of the similar tech discussion sites.

I believe you're confusing the local portion and the hostname portion. The hostname needs to be punycode if it's an IDN, but that particular symbol is invalid in public IDNs

Swift_AddressEncoderException: Email to xxxxx@blöd.xxx failed: Non-ASCII characters not supported in local-part

That sounds like it's complaining about the portion of the email address before the @ symbol, not after, which contradicts your example. Can you double-check? They're handled differently and require different encoding techniques.

 

[Kirby]

That sounds like it's complaining about the portion of the email address before the @ symbol, not after, which contradicts your example. Can you double-check? They're handled differently and require different encoding techniques.

The domain is causing the problem as it becomes part of the local part, see https://xenforo.com/community/threa...t-supported-in-local-part.204337/post-1590114

 

[PaulB]

Ah, that’s annoying.

 

[Jeremy P]

I think this should be resolved in XF 2.2.13+ as we now convert UTF-8 domains in email addresses into their ASCII form. Any existing UTF-8 email addresses should be converted during the upgrade.