Duplicate unable to import from vb 3.8. Unicode Bug

[Earl]

I've got two users with usernames like 'محمد العلاف' (looks like Arabic characters) and also have plenty of other users with other language characters, in my db:


This is where the script stopped at the beginning:

user with user ID 161751 has its username this 'محمد العلاف'

and the other user have this 'محمد العتوم' as their usernames.

These usernames are different, But script detects it as duplicate users. seems like it only compares the first few characters and ignores the last parts, then it throws this DuplicateKeyException exception.


XF\Db\DuplicateKeyException: MySQL query error [1062]: Duplicate entry 'محمد الع' for key 'username' in src/XF/Db/AbstractStatement.php at line 212

1. XF\Db\AbstractStatement->getException() insrc/XF/Db/Mysqli/Statement.php at line 196
2. XF\Db\Mysqli\Statement->getException() insrc/XF/Db/Mysqli/Statement.php at line 77
3. XF\Db\Mysqli\Statement->execute() in src/XF/Db/AbstractAdapter.phpat line 79
4. XF\Db\AbstractAdapter->query() in src/XF/Db/AbstractAdapter.php at line 160
5. XF\Db\AbstractAdapter->insert() insrc/XF/Import/Data/EntityEmulator.php at line 320
6. XF\Import\Data\EntityEmulator->insert() insrc/XF/Import/Data/User.php at line 190
7. XF\Import\Data\User->write() in src/XF/Import/Data/AbstractData.phpat line 126
8. XF\Import\Data\AbstractData->save() in src/XF/Import/Helper.php at line 107
9. XF\Import\Helper->importUser() insrc/XF/Import/Importer/AbstractCoreImporter.php at line 107
10. XF\Import\Importer\AbstractCoreImporter->importUser() insrc/addons/XFI/Import/Importer/vBulletin.php at line 1013
11. XFI\Import\Importer\vBulletin->stepUsers() insrc/XF/Import/Runner.php at line 160
12. XF\Import\Runner->runStep() in src/XF/Import/Runner.php at line 74
13. XF\Import\Runner->run() in src/XF/Admin/Controller/Import.php at line232
14. XF\Admin\Controller\Import->actionRun() insrc/XF/Mvc/Dispatcher.php at line 249
15. XF\Mvc\Dispatcher->dispatchClass() in src/XF/Mvc/Dispatcher.php at line88
16. XF\Mvc\Dispatcher->dispatchLoop() in src/XF/Mvc/Dispatcher.php at line41
17. XF\Mvc\Dispatcher->run() in src/XF/App.php at line 1931
18. XF\App->run() in src/XF.php at line 328
19. XF::runApp() in admin.php at line 13

Luckily these IDs were some random Arabic spammers, so I could remove these two users, But after I re-execute the import command, The script started detecting more users with Unicode characters in their usernames as duplicates. Now I'm totally stuck at this point, there are thousands of users that this script detects as duplicates, and I cant remove them all, They are legit users. (but I tried deleting those users and it allowed to continue the script running.)

Please help

 

[Slavik]

See dragonbyte techs diff files here: https://xenforo.com/community/threads/vbulletin-importers-unicode-entities-are-not-decoded.144529/

 

[Earl]

See dragonbyte techs diff files here: https://xenforo.com/community/threads/vbulletin-importers-unicode-entities-are-not-decoded.144529/

See dragonbyte techs diff files here: https://xenforo.com/community/threads/vbulletin-importers-unicode-entities-are-not-decoded.144529/

Okay, I applied the patch and the situation got worse and turned into a more dangerous one! lol

I've got these two users:

Before import

UserID: 38647
Username: "Leni"

UserID: 260394
Username: "Leni"




The second user is fake and trying to appear as the original user.

This is what happened after applying the patch and import

After Import:
UserID: 38647
Username: "Leni"

UserID: 260394
Username: "Leni"



As you can see, It trimmed out the &#XXX characters and made both usernames exact same.
Is that even possible to have same name users in DB?
That would become a major security bug, wouldn't it?

 

[Earl]

hmm seems like both usernames are not equal, even if we can see it as equal.

<?php


$value = 'Leni&#8302;&#8302;&#8302;&#8302;&#8302;&#8302;&#8302;&#8302;&#8302;&#8302;&#8302;&#8302;&#8302;';
$value = html_entity_decode(strval($value), ENT_QUOTES, 'UTF-8');

echo $value=='Leni'? "$value is Equal to Leni" : "$value is not equal to Leni"; // Leni is not equal to Leni

?>

https://ideone.com/VqQSzH

I'm not sure what to do, Is it safe to keep it in DB like this?