Non-ASCII characters not supported in local-part

Walter

Well-known member
Affected version
2.28 Patch 1
A user changed his email address to something stupid like ihatetheadmin@blöd.de ("blöd" is German for stupid), an obviously not existing email address. Now I get several errors from Swift telling me that umlauts are not allowed in an email address. It would be better to prevent umlauts in the user interface when entering an email.

Swift_AddressEncoderException: Email to xxxxx@blöd.xxx failed: Non-ASCII characters not supported in local-part src/vendor/swiftmailer/swiftmailer/lib/classes/Swift/AddressEncoder/IdnAddressEncoder.php:40

src/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Mime/Headers/PathHeader.php(132): Swift_AddressEncoder_IdnAddressEncoder->encodeString('bounce+621802e1...')
#1 src/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Mime/SimpleHeaderSet.php(309): Swift_Mime_Headers_PathHeader->getFieldBody()
#2 src/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Mime/SimpleMimeEntity.php(529): Swift_Mime_SimpleHeaderSet->toString()
#3 src/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Mime/SimpleMessage.php(604): Swift_Mime_SimpleMimeEntity->toByteStream(Object(Swift_Transport_StreamBuffer))
#4 src/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Message.php(162): Swift_Mime_SimpleMessage->toByteStream(Object(Swift_Transport_StreamBuffer))
#5 src/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Transport/SendmailTransport.php(129): Swift_Message->toByteStream(Object(Swift_Transport_StreamBuffer))
#6 src/XF/Mail/Mailer.php(306): Swift_Transport_SendmailTransport->send(Object(Swift_Message), Array)
#7 src/XF/Mail/Queue.php(148): XF\Mail\Mailer->send(Object(Swift_Message), Object(Swift_SendmailTransport), Array)
#8 src/XF/Job/MailQueue.php(12): XF\Mail\Queue->run(7.90058)
#9 src/XF/Job/Manager.php(260): XF\Job\MailQueue->run(7.90058)
#10 src/XF/Job/Manager.php(202): XF\Job\Manager->runJobInternal(Array, 7.90058)
#11 src/XF/Job/Manager.php(86): XF\Job\Manager->runJobEntry(Array, 7.90058)
#12 job.php(43): XF\Job\Manager->runQueue(false, 8)
#13 {main}
 

Kirby

Well-known member
This should be adressed ASAP

PHP:
$email = 'someone@♥local.host';

if (filter_var($email, FILTER_VALIDATE_EMAIL))
{
    echo('PHP: E-Mail is valid');
}
else
{
    echo('PHP: E-Mail is NOT valid!');
}

echo("\n");

$startTime = microtime(true);
$phpVersion = phpversion();
if (version_compare($phpVersion, '7.0.0', '<'))
{
        die("PHP 7.0.0 or newer is required. $phpVersion does not meet this requirement. Please ask your host to upgrade PHP.");
}

$dir = __DIR__;
require($dir . '/src/XF.php');

XF::start($dir);

$validator = \XF::app()->validator('Email');

if ($validator->isValid($email))
{
    echo('XF: E-Mail is valid');
}
else
{
    echo('XF: E-Mail is not valid!');
}

echo("\n");


Output
Code:
PHP: E-Mail is NOT valid!
XF: E-Mail is valid
 

MySiteGuy

Well-known member
Technically, Xenforo allowing it is correct. And PHP and Swift are wrong.

Email characters are not limited to 7 bit ASCII characters, haven't been for a long time, and most Unicode characters are allowed per RFC 6532
 

Kirby

Well-known member
I tend to disagree :)

While you are correct that it would be valid per RFC 6532 3.2, I'd expect that most servers out there only support RFC 5322 (or maybe even just RFC 822).

So even if XenForo "is correct" and it would pass SwiftMailer (more likely Symfony Mailer for XF 2.3+, which also wouldn't accept an address like in my example) emails wouldn't be accepted by many (most?) mailservers.

Just an example:
Code:
> telnet gmail-smtp-in.l.google.com 25
Trying 74.125.133.26...
Connected to gmail-smtp-in.l.google.com.
Escape character is '^]'.
220 mx.google.com ESMTP b15-20020a05600018af00b00226df33238asi825462wri.858 - gsmtp
HELO mail.example.com
250 mx.google.com at your service
MAIL FROM: <info@example.com>
250 2.1.0 OK b15-20020a05600018af00b00226df33238asi825462wri.858 - gsmtp
RCPT TO: <someone@♥local.host>
555 5.5.2 Syntax error. b15-20020a05600018af00b00226df33238asi825462wri.858 - gsmtp

So what's the point of allowing such addresses in first place if it's not realistically possible to send email there?
This just creates support problems.
 
Last edited:

MySiteGuy

Well-known member
Gmail does support non-Latin characters and has since 2014 ( https://blog.google/products/gmail/a-first-step-toward-more-global-email/ ).

Your example is not sending the RCPT TO address correctly for non-Latin character characters. For example someone@♥local.host is sent perfectly to gmail from Thunderbird. It has to be surrounded by encoding information which states the character set. I don't have the format right off hand, but I do recall seeing an example on stackexchange or one of the similar tech discussion sites.

I think Xenforo should not strip out support for more than half the world's languages which are widely used outside of English first countries. Rather, the solution should be to properly implement the sending standard, which actually is supported by most widely used email services and server software these days.
 

Kirby

Well-known member
Gmail does support non-Latin characters and has since 2014 ( https://blog.google/products/gmail/a-first-step-toward-more-global-email/ ).
I wasn't talking about non-ascii charaters in email adresses but about a specific character in the domain.

As per IDNA 2003/2008 the character (Unicode U+2665) is disallowed:
Code:
2665 : DISALLOWED : BLACK HEART SUIT
So while an address someone@♥local.host is valid from SMTP protocol view (for a mailerserver supporting SMTPUTF8 extension, eg RFC eg. 6531) such a domain name remains "invalid" for general purpose use (eg. on the Internet) as it wouldn't be possible to register a domain i♥unicode.com

I think Xenforo should not strip out support for more than half the world's languages which are widely used outside of English first countries. Rather, the solution should be to properly implement the sending standard, which actually is supported by most widely used email services and server software these days.
I already does that for the local part /mailbox (as the unterlaying transports do not generally support that):
XF\Validator\Email::isValid
PHP:
public function isValid($value, &$errorKey = null)
{
    if ($value === '')
    {
        if ($this->options['allow_empty'])
        {
            return true;
        }

        $errorKey = 'empty';
        return false;
    }

    $validator = new EmailValidator();
    if (!$validator->isValid($value, new RFCValidation()))
    {
        $errorKey = 'invalid';
        return false;
    }

    list ($local, $domain) = explode('@', $value);

    if (preg_match('/[^\x00-\x7F]/', $local))
    {
        // Swiftmailer will error if it runs into this
        $errorKey = 'invalid';
        return false;
    }

This specific issue arises if VERP is enabled and the recpient domain contains such non-ascii characters as the domain becomes part of localpart / mailbox.

The fix here would be to properly IDN-encode the email address first before applying VERP.

But even if this is fixed IMHO XenForo should handle IDN properly and not accept email addresses like someone@i♥unicode.com
 
Top