Won't fix vBulletin 3.8 import errors

[Jake B.]

Affected version

Latest

So far it's only happened once, and running the command has continued successfully, but I'm running the importer with 12 processes and ended up getting the following error:

XF\Db\DuplicateKeyException: MySQL query error [1062]: Duplicate entry '???529' for key 'username' src/XF/Db/AbstractStatement.php:228

INSERT  INTO `xf_user` (`user_id`, `username`, `email`, `style_id`, `language_id`, `timezone`, `visible`, `activity_visible`, `user_group_id`, `secondary_group_ids`, `display_style_group_id`, `permission_combination_id`, `message_count`, `alerts_unread`, `conversations_unread`, `register_date`, `last_activity`, `trophy_points`, `avatar_date`, `avatar_width`, `avatar_height`, `avatar_highdpi`, `gravatar`, `user_state`, `is_moderator`, `is_admin`, `is_staff`, `is_banned`, `reaction_score`, `custom_title`, `warning_points`, `secret_key`, `privacy_policy_accepted`, `terms_accepted`) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
------------

#0 src/XF/Db/Mysqli/Statement.php(196): XF\Db\AbstractStatement->getException()
#1 src/XF/Db/Mysqli/Statement.php(77): XF\Db\Mysqli\Statement->getException()
#2 src/XF/Db/AbstractAdapter.php(94): XF\Db\Mysqli\Statement->execute()
#3 src/XF/Db/AbstractAdapter.php(218): XF\Db\AbstractAdapter->query()
#4 src/XF/Import/Data/EntityEmulator.php(326): XF\Db\AbstractAdapter->insert()
#5 src/XF/Import/Data/User.php(201): XF\Import\Data\EntityEmulator->insert()
#6 src/XF/Import/Data/AbstractData.php(127): XF\Import\Data\User->write()
#7 src/XF/Import/Helper.php(107): XF\Import\Data\AbstractData->save()
#8 src/XF/Import/Importer/AbstractCoreImporter.php(107): XF\Import\Helper->importUser()
#9 src/addons/XFI/Import/Importer/vBulletin.php(1001): XF\Import\Importer\AbstractCoreImporter->importUser()
#10 src/XF/Import/Runner.php(231): XFI\Import\Importer\vBulletin->stepUsers()
#11 src/XF/Import/ParallelRunner.php(212): XF\Import\Runner->runStep()
#12 src/XF/Cli/Command/ImportChildProcess.php(78): XF\Import\ParallelRunner->runChildProcess()
#13 src/vendor/symfony/console/Command/Command.php(255): XF\Cli\Command\ImportChildProcess->execute()
#14 src/vendor/symfony/console/Application.php(982): Symfony\Component\Console\Command\Command->run()
#15 src/vendor/symfony/console/Application.php(255): Symfony\Component\Console\Application->doRunCommand()
#16 src/vendor/symfony/console/Application.php(148): Symfony\Component\Console\Application->doRun()
#17 src/XF/Cli/Runner.php(63): Symfony\Component\Console\Application->run()
#18 cmd.php(15): XF\Cli\Runner->run()
#19 {main}

array(1) {
["cli"] => string(95) "PATH/cmd.php xf:import-child-process users 4598576 4601576"
}

 

[Jake B.]

Just after posting this it happened again and stopped the importer

 

[Jake B.]

This ended up crashing a few more times so switched back to the normal single process import

 

[Mike]

I don't think we're going to make any changes here at this time. While there is potentially an edge case issue here, I think the problem you're running into is potentially exaggerated by what looks like a character set issue.

Roughly, it looks like you're running into a race condition where we detect a duplicate username and try to determine what to rename that user to. We appear to be trying to rename two users to "???529" at the same time. The race condition happens between that determination and the insert. Normally, this would be incredibly rare as it would require a username (or usernames that are equivalent in MySQL) to be used at least 3 times in the DB and for 2 distinct processes to end up processing the users simultaneously. There's a good chance that a restart would move past it too, due to the nature of a race condition.

It looks to me like this issue is coming up very frequently in this import due to a character set issue. Based on the name "???529", I assume this is the 530th instance of the name "???" that we found (with 529 of them being renamed). It seems very unlikely that there are that many users named "???" in the DB unless something went terribly wrong with the source data. "?" is also used when a character can't be represented correctly in a character set, so it looks likely that the username was distinct but had characters converted to "?" at some point during the process. This might be related to a dump/restore with the wrong character set (eg, utf8 vs utf8mb4) or incorrect settings for forcing a character set during the import (including not doing it when needed). I suspect if the character set issue that caused this conversion was resolved, the issue would probably go away.

Any steps we would try to take to resolve an issue like this would likely lead to a significant slow down to the multi-process importer, which roughly defeats the point. So therefore, we're not going to make any changes to prevent/suppress this error.

 

[Jake B.]

Just checked some of the users that were converted to ???, and it looks like many of them were using Japanese characters. Checked with them and they forced their vBulletin database to be utf8mb4, which wasn't autodetected for some reason. Thanks!