XF 1.5 Post migration issues

[Jake B.]

A site that was imported from vBulletin 3.X is having some odd issues, some (not all) apostrophes, for example, are being replaced with ’ so that's is displaying as That’s, etc. Curious if anyone has run into this, and if there is an tool somewhere to fix these already

Also, they had an [ame] bbcode for embedding media on vBulletin, is there anything that exists already for converting these to XF media tags?

 

[Chris D]

It's likely that an incorrect character set was chosen on the import config page. Usually windows-1252 would be the correct value to use.

Also, they had an [ame] bbcode for embedding media on vBulletin, is there anything that exists already for converting these to XF media tags?

Just the post content find/replace tool in the RM.

 

[Jake B.]

It's likely that an incorrect character set was chosen on the import config page. Usually windows-1252 would be the correct value to use.

Yeah, that's what I used (and what it appears to default to). Haven't had an issue with it before, will see what we can do to fix it now

 

[Mike]

Just to clarify, the force character set option should probably be "latin1". (It refers to a MySQL character set and has nothing to do with what may be configured within the app; it's more related to table and character set collations.)

 

[Jake B.]

It looks like it attempts to auto detect the charset on import if it's left blank

$defaultLanguageId = $db->fetchOne('
   SELECT value
   FROM ' . $config['db']['prefix'] . 'setting
   WHERE varname = \'languageid\'
');
$defaultCharset = $db->fetchOne('
   SELECT charset
   FROM ' . $config['db']['prefix'] . 'language
   WHERE languageid = ?
', $defaultLanguageId);
if (!$defaultCharset || str_replace('-', '', strtolower($defaultCharset)) == 'iso88591')
{
   $config['charset'] = 'windows-1252';
}
else
{
   $config['charset'] = strtolower($defaultCharset);
}

Interestingly enough, I don't see this occurring on any of the test migrations we did, only on the actual migration, which is unfortunate lol. We'll probably just use the post find and replace to fix these as we come across them for now

 

[Mike]

The forcing option isn't related to that -- it's down to how the data is retrieved from MySQL. Different MySQL versions have different default values, so it's possible that doing imports on different servers/DBs can cause different behaviors. For example, if you have latin1 (which is basically Windows-1252) but MySQL is using utf8 for results (which is the default in recent versions), the content will be converted to UTF-8 directly from MySQL. We then see that the data should be in Windows-1252 (or whatever is stored in the source DB option) and encoded that to UTF-8 again. Hence, the double encoding that you're seeing.

Conversely, if you force the results to be in latin1 (probably because that's what was being used elsewhere), it will only do the single encoding.