XF 2.2 When upgrading Unicode characters in titels are converted to question marks. :(

[Masetrix]

When upgrading from XF 2.1.x to XF 2.2.x, all Unicode characters in (widget) titels are converted to question marks.

Example.
The widget title: New posts... becomes: ? New posts... and so on.

It would be nice if we didn't have to manually adjust all Unicode characters in titles during the final upgrade.

 

[djbaxter]

Is this a style issue perhaps? Does it happen with the default style?

 

[Masetrix]

Is this a style issue perhaps? Does it happen with the default style?

Yes, I have a backup of XF 2.1. (Default-Style and some changes in "extra.less") restored and then upgraded to XF2.2 Beta4 from there.

The result stays the same. All Unicodes in titles are shown as question marks.

 

[Chromaniac]

ooh. i read it as thread title. and i added an emoji to a test thread before upgrade. that emoji survived the update from 2.1.10 to 2.2b4. too late to test with widget titles.

 

[Kirby]

Hmm ... does this really only affect titles?

Did you check posts containing utf8 characters with more than 2 bytes (eg. emojis), are they okay?

If not: https://xenforo.com/community/threads/public-service-announcement-check-your-mysql-backup.152740/

 

[Masetrix]

Hmm ... does this really only affect titles?

Did you check posts containing utf8 characters with more than 2 bytes (eg. emojis), are they okay?

If not: https://xenforo.com/community/threads/public-service-announcement-check-your-mysql-backup.152740/

Have a look...



vs.
Only Unicodes in Widgettitles are shown as question marks.



Also i have run the conversionscript for testing but: "No convertable tables found. No action required."

All tables in the XF database are utf8mb4.
When creating a backup and then restoring, all unicodes are as they should be.
Question marks only appear when upgrading to 2.2.BetaX

I do not know the upgrade process.
But it seems as if a "default-character-set = utf8mb4" is missing in the upgrade.

My backup also explicitly sets the "default character set" according to the database it is backing up

I cannot do this as the "default" in Mysql because I have to use different character sets from database to database.

 

[Chris D]

There would be no changes in the 2.2 upgrade that would cause this so I would still strongly suspect that it is the backup/restore process of the database that is the issue.

You seem to imply that you have checked the character set of the database in the backup but this very much sounds like exactly the same issue we warned about here:

Public service announcement: Check your MySQL backup!

Having moved servers today, we learned that our MySQL backups were subtly corrupted by an insidious issue that can be difficult to spot until it's too late. All emoji and high-ascii characters in our database dumps were invalid. This is due to a behaviour in mysqldump, which defaults the...

xenforo.com

Are you able to verify that when you restore the 2.1 database, before the upgrade to 2.2, that emoji characters are working? Based on everything, I'm sure the issue should exist before the 2.2 upgrade too.

 

[Kirby]

All tables in the XF database are utf8mb4.
When creating a backup and then restoring, all unicodes are as they should be.
Question marks only appear when upgrading to 2.2.BetaX

All tables being utf8mb4 does not necessarily mean that the characters inside them are actually valid utf8mb4

Did you verify that the restored backup does work correctly with XF 2.1 (eg. does show the correct unicode characters) before running the 2.2 upgrade?

 

[Masetrix]

Did you verify that the restored backup does work correctly with XF 2.1 (eg does show the correct unicode characters) before running the 2.2 upgrade?

Yes.