Duplicate unable to import from vb 3.8. Unicode Bug

Earl

Active member
I've got two users with usernames like 'محمد العلاف' (looks like Arabic characters) and also have plenty of other users with other language characters, in my db:
1530088987454.png

This is where the script stopped at the beginning:

user with user ID 161751 has its username this 'محمد العلاف'

and the other user have this 'محمد العتوم' as their usernames.

These usernames are different, But script detects it as duplicate users. seems like it only compares the first few characters and ignores the last parts, then it throws this DuplicateKeyException exception.


XF\Db\DuplicateKeyException: MySQL query error [1062]: Duplicate entry 'محمد الع' for key 'username' in src/XF/Db/AbstractStatement.php at line 212
  1. XF\Db\AbstractStatement->getException() insrc/XF/Db/Mysqli/Statement.php at line 196
  2. XF\Db\Mysqli\Statement->getException() insrc/XF/Db/Mysqli/Statement.php at line 77
  3. XF\Db\Mysqli\Statement->execute() in src/XF/Db/AbstractAdapter.phpat line 79
  4. XF\Db\AbstractAdapter->query() in src/XF/Db/AbstractAdapter.php at line 160
  5. XF\Db\AbstractAdapter->insert() insrc/XF/Import/Data/EntityEmulator.php at line 320
  6. XF\Import\Data\EntityEmulator->insert() insrc/XF/Import/Data/User.php at line 190
  7. XF\Import\Data\User->write() in src/XF/Import/Data/AbstractData.phpat line 126
  8. XF\Import\Data\AbstractData->save() in src/XF/Import/Helper.php at line 107
  9. XF\Import\Helper->importUser() insrc/XF/Import/Importer/AbstractCoreImporter.php at line 107
  10. XF\Import\Importer\AbstractCoreImporter->importUser() insrc/addons/XFI/Import/Importer/vBulletin.php at line 1013
  11. XFI\Import\Importer\vBulletin->stepUsers() insrc/XF/Import/Runner.php at line 160
  12. XF\Import\Runner->runStep() in src/XF/Import/Runner.php at line 74
  13. XF\Import\Runner->run() in src/XF/Admin/Controller/Import.php at line232
  14. XF\Admin\Controller\Import->actionRun() insrc/XF/Mvc/Dispatcher.php at line 249
  15. XF\Mvc\Dispatcher->dispatchClass() in src/XF/Mvc/Dispatcher.php at line88
  16. XF\Mvc\Dispatcher->dispatchLoop() in src/XF/Mvc/Dispatcher.php at line41
  17. XF\Mvc\Dispatcher->run() in src/XF/App.php at line 1931
  18. XF\App->run() in src/XF.php at line 328
  19. XF::runApp() in admin.php at line 13

Luckily these IDs were some random Arabic spammers, so I could remove these two users, But after I re-execute the import command, The script started detecting more users with Unicode characters in their usernames as duplicates. Now I'm totally stuck at this point, there are thousands of users that this script detects as duplicates, and I cant remove them all, They are legit users. (but I tried deleting those users and it allowed to continue the script running.)

Please help
 
Last edited:

Earl

Active member
Okay, I applied the patch and the situation got worse and turned into a more dangerous one! :LOL: lol

I've got these two users:

Before import

UserID: 38647
Username: "Leni"


UserID: 260394
Username: "Leni"


1530170675405.png


The second user is fake and trying to appear as the original user.

This is what happened after applying the patch and import

After Import:
UserID: 38647
Username: "Leni"


UserID: 260394
Username: "Leni"

1530170533768.png


As you can see, It trimmed out the &#XXX characters and made both usernames exact same. o_O
Is that even possible to have same name users in DB?
That would become a major security bug, wouldn't it?
 

Earl

Active member
hmm seems like both usernames are not equal, even if we can see it as equal.
PHP:
<?php


$value = 'Leni&#8302;&#8302;&#8302;&#8302;&#8302;&#8302;&#8302;&#8302;&#8302;&#8302;&#8302;&#8302;&#8302;';
$value = html_entity_decode(strval($value), ENT_QUOTES, 'UTF-8');

echo $value=='Leni'? "$value is Equal to Leni" : "$value is not equal to Leni"; // Leni is not equal to Leni

?>

https://ideone.com/VqQSzH

I'm not sure what to do, Is it safe to keep it in DB like this?
 
Last edited:
Top