vBulletin Big Board Importer [vBulletin 3 + vBulletin 4] [Paid] 1.5.0

[Ridemonkey]

I ran into some weirdness with authentication last night. It's fixed but I'd like to know what happened.

I'm running the Export.php on the web server, which is remote to the DB server. I filled out the credentials and information for the vBulletin database user.

However, I kept getting authentication failures for the XenForo database user:
<!-- Mysqli prepare error: SELECT command denied to user 'xenforouser'@'prod-web01' for table 'user' -->

The 'user' table is definitely a VB table, and there is no reference whatsoever to the xenforouser in the Export.php script. I moved past this by granting the xenforo user all rights to the VB database, but why is that happening?

 

[Jake Bunce]

I ran into some weirdness with authentication last night. It's fixed but I'd like to know what happened.

I'm running the Export.php on the web server, which is remote to the DB server. I filled out the credentials and information for the vBulletin database user.

However, I kept getting authentication failures for the XenForo database user:
<!-- Mysqli prepare error: SELECT command denied to user 'xenforouser'@'prod-web01' for table 'user' -->

The 'user' table is definitely a VB table, and there is no reference whatsoever to the xenforouser in the Export.php script. I moved past this by granting the xenforo user all rights to the VB database, but why is that happening?

There is actually some overlap in the db connections in the Export script. The Export connects to both vB and XF at certain points using the same db connection (and therefore db user). So both databases need to be accessible with the same MySQL user.

 

[Ridemonkey]

Gotcha. Thanks.

Got some user permission problems when rebuilding the cache, looks like a few users came over with no usergroup and it's causing errors in XF. Fixed that in the DB and I'm rebuilding the user cache now.

 

[Ridemonkey]

So... thread prefixes. Is there a way to import those? We have a couple forums that use them and I'd like to maintain them.

 

[Jake Bunce]

So... thread prefixes. Is there a way to import those? We have a couple forums that use them and I'd like to maintain them.

This importer does not include thread prefixes. Custom code is required.

Or it should be easy to bulk insert and update the prefixes after the import with a couple of queries.

 

[kontrabass]

Hi guys - as you know I've had tremendous success with this importer, however in the last few runs I've noticed a big issue with conversations.

After importing and after rebuilding all caches, if I go and create a conversation to any one person, after I create the conversation I'll see that it has two extra participants, seemingly two random participants.

I looked into this further, and it turns out that the conversation ID number I just started (ie, "4779375") is already been used in the vbulletin data (I searched the export .txt files and found the conversation ID in there, and in vbulletin the conversation was, of course, between the two users that are added as extra recipients on my xf import board.

So, how do I tell xenforo to start making conversation ID's at a higher number.... so that they won't go "on top" of existing vbulletin PM id's? Does this make any sense?

 

[Jake Bunce]

Hi guys - as you know I've had tremendous success with this importer, however in the last few runs I've noticed a big issue with conversations.

After importing and after rebuilding all caches, if I go and create a conversation to any one person, after I create the conversation I'll see that it has two extra participants, seemingly two random participants.

I looked into this further, and it turns out that the conversation ID number I just started (ie, "4779375") is already been used in the vbulletin data (I searched the export .txt files and found the conversation ID in there, and in vbulletin the conversation was, of course, between the two users that are added as extra recipients on my xf import board.

So, how do I tell xenforo to start making conversation ID's at a higher number.... so that they won't go "on top" of existing vbulletin PM id's? Does this make any sense?

Setting the autoincrement id after the import would be a bandaid. We should really figure out why it's happening in the first place, as there shouldn't be collisions.

I would like to examine your data if you can give me access to phpmyadmin.

 

[kontrabass]

Setting the autoincrement id after the import would be a bandaid. We should really figure out why it's happening in the first place, as there shouldn't be collisions.

I would like to examine your data if you can give me access to phpmyadmin.

Hi Jake,

Thanks for your reply. Your thoughts got me on the right track - it shouldn't have happened, and the reason why I'm fairly confident, is I am using a live slave database for the export. So conversation_master finishes first, and by the time conversation_recipient is done, 20 or so more records have been added to conversation_recipient.

Duh. I should have realized. Of course during the real migration the board will be totally shut down, so this is a non-issue. Thanks for helping me think this through

 

[Jake Bunce]

Hi Jake,

Thanks for your reply. Your thoughts got me on the right track - it shouldn't have happened, and the reason why I'm fairly confident, is I am using a live slave database for the export. So conversation_master finishes first, and by the time conversation_recipient is done, 20 or so more records have been added to conversation_recipient.

Duh. I should have realized. Of course during the real migration the board will be totally shut down, so this is a non-issue. Thanks for helping me think this through

Ah yes, that explains it. Importing against live data on extremely busy (ie large) forums will often result in small "misalignments" in the resulting data.

 

[Ridemonkey]

It looks like visitor messages (i.e. profile comments in XF) aren't imported with this tool. From what I can see on screen shots, the official VB importer does it - only I can't get there from here.

Am I doing something wrong? There are references in Export.php to the visitor message table and import references to profile_post, but I can't see any profile posts on anyone who has them in VB. I checked the DB and the xf_profile_post table is empty. The importData directory has a profile_post.txt file which is not empty.

 

[Jake Bunce]

It looks like visitor messages (i.e. profile comments in XF) aren't imported with this tool. From what I can see on screen shots, the official VB importer does it - only I can't get there from here.

Am I doing something wrong? There are references in Export.php to the visitor message table and import references to profile_post, but I can't see any profile posts on anyone who has them in VB. I checked the DB and the xf_profile_post table is empty. The importData directory has a profile_post.txt file which is not empty.

Profile posts *are* imported by this importer.

The importData directory should have a non-empty xf_profile_post.txt file after running the Export. Then the Import script imports that file assuming you haven't commented or removed this line:

importTable('xf_profile_post');

 

[Ridemonkey]

Profile posts *are* imported by this importer.

The importData directory should have a non-empty xf_profile_post.txt file after running the Export. Then the Import script imports that file assuming you haven't commented or removed this line:

importTable('xf_profile_post');

Sorry, I meant, "aren't being imported with this tool." Like I said, I see the import references, and I have an xf_profile_post.txt file which contains data, but my XF install has an empty xf_profile_post table.

I think I see the problem - on the Import.php file, it looks like this:

// FOR INCREMENTAL IMPORT OF POSTS USE THE 'false' LINE SO IT DOESN'T TRUNCATE
importTable('xf_post');
//importTable('xf_post', false);importTable('xf_profile_post');
importTable('xf_report');
importTable('xf_report_comment');

Not sure how the xf_profile_post table line got up next to the commented out xf_post line, but I'll make sure it's not up there next time I run the importer. Thanks for pointing me in the right direction!

 

[Donny]

What sort of QTY of threads and posts do you consider this starts to become more viable than the stock importer for VB4 to XF?

I have a few boards with around 800k posts, 80k threads, 50k members. Would I need this?

 

[Jake Bunce]

What sort of QTY of threads and posts do you consider this starts to become more viable than the stock importer for VB4 to XF?

I have a few boards with around 800k posts, 80k threads, 50k members. Would I need this?

You don't need this importer.

I would say more than 2 million posts is when you should consider this importer.

 

[mastahg]

Hello, working on doing a 'large' import, around 2m posts 200k users.

Few things:

1)After changing charset from utf-8 to binary in Application.php should I revert it after the export and before the import, after the import or just leave it binary?

2)User group mapping, there is no importation of the vb settings or usergroups correct? I'll have to manually add the groups to xenforo before the export?

3)Attachment errors:

Sometimes during the attachment phase

exec('echo $\'' . addslashes($attachment['filename']) . '\' ' . self::$extraOutCommand , $attachFileNameTmp);

will error about sh:unescaped ")" at line 1

Is this fine, or should I disable the extracommandsection for the attachements.

 

[Jake Bunce]

@mastahg

1) It can be reverted when you are done with your final import.

2) Correct.

3) Possibly something to do with the file name. Are you able to capture the whole command that is generating the error?

 

[mastahg]

Thanks for 1 and 2
Ended up getting #3 to work, addslashes is not sufficent as it does not escape ( or ) so I used addcslashes

                    exec('echo $"' . addcslashes($attachment['filename'],"\'()") . '"' . self::$extraOutCommand , $attachFileNameTmp);

 

[mastahg]

During the import phase, lots of different warnings being thrown around...

xf_conversation_recipient951
Warning 1062    Duplicate entry '161-7547' for key 'PRIMARY'
Warning 1062    Duplicate entry '237-7647' for key 'PRIMARY'
Warning 1062    Duplicate entry '5778-12063' for key 'PRIMARY'
.
.
.
.
.

xf_poll12
Warning 1062    Duplicate entry '956' for key 'PRIMARY'
Warning 1062    Duplicate entry '957' for key 'PRIMARY'
.
.
.

xf_post.4
Warning 1265    Data truncated for column 'username' at row 542814
Warning 1265    Data truncated for column 'username' at row 622044
Warning 1265    Data truncated for column 'username' at row 842195
Warning 1265    Data truncated for column 'username' at row 868292

Etc.. any particular reason why tables arent properly sized and duplicate primaries are being created?

Edit:
Looks like it doesnt import some of the more unique name sub forums while the build in importer does..

http://i.imgur.com/MagUxbn.png

Same with posts
http://i.imgur.com/2jiMYxv.png

 

[Jake Bunce]

@mastahg

"Duplicate entry" errors shouldn't be happening. It could be a problem with the source data.

"Data truncated" errors are normal given the nature of this importer. But if there is any question then you need to examine the intermediate file (xf_post) to confirm the data.

 

[jaymsgq]

Hi Hornstar,
Did you find a fix for this error Mysqli statement execute error : Incorrect integer value: '' for column 'user_group_id' at row 1 ?
Thanks