vBulletin Big Board Importer [vBulletin 3 + vBulletin 4] [Paid] 1.5.0

[Ludachris]

Attaching the errors via a PDF file...

 

[jaymsgq]

I don't know if I can fit them all here... wonder if I should post them into a txt file and try to upload?

@Ludachris
You mentioned AdminCP is broken ? how did it break? can you login to the AdminCP ? if not try using the VB username and password in XF. After you login do you see anything the XF server log?

 

[Ludachris]

It looked as if it did a partial import and all the navigation links in admincp were gone except Tools and there was only one option left in Tools (the test facebook integration link). And it showed 1 member in the stats, and when I clicked the home link it said I didn't have permission. I've since restored the db to where it was before and can navigate around just fine again.

 

[Ludachris]

Can anyone confirm that I do need to replicate the user fields I have in vb, then map them all in the Import.php file, then try the import again? Not sure if that is what caused the errors that I posted above (in the attached pdf file) or not.

 

[Ludachris]

Well, I guess I'm going to try another import. I'm not going to set up the custom fields in XF admincp since nobody has chimed in saying that needs to be done. I did notice that there are avatars in the avatars directory. I'm guessing I'll need to delete those in order to prevent duplicate data.

 

[Jake Bunce]

Well, I guess I'm going to try another import. I'm not going to set up the custom fields in XF admincp since nobody has chimed in saying that needs to be done. I did notice that there are avatars in the avatars directory. I'm guessing I'll need to delete those in order to prevent duplicate data.

It's not necessary to delete the already-imported avatars for subsequent import attempts. This big board importer can be run over and over again against the same installation of XF.

Can anyone confirm that I do need to replicate the user fields I have in vb, then map them all in the Import.php file, then try the import again? Not sure if that is what caused the errors that I posted above (in the attached pdf file) or not.

For user fields you must create the user fields in XF's Admin CP first, then edit this block of the Export script to do the mapping:

		$start = microtime(true);
		echo "          skipping user fields";
		//Map any custom user feilds you have here. Disabled by default.
		// exec('mysql -h' . $this->slaveDbHost . ' ' . $this->sourceDb . ' ' . $this->sourceDBuser . ' ' . $this->sourceDBpassword . ' -N -q -e "SET NAMES binary;SELECT SQL_NO_CACHE userid AS user_id, \'interests\' AS field_id, field3 AS field_value FROM userfield WHERE field3 != \'\' ORDER BY userid" ' . self::$extraOutCommand . ' > ' . self::$dataDir . 'xf_user_field_value.txt');
		// exec('mysql -h' . $this->slaveDbHost . ' ' . $this->sourceDb . ' ' . $this->sourceDBuser . ' ' . $this->sourceDBpassword . ' -N -q -e "SET NAMES binary;SELECT SQL_NO_CACHE userid AS user_id, \'twitter\' AS field_id, field14 AS field_value FROM userfield WHERE field14 != \'\' ORDER BY userid" ' . self::$extraOutCommand . ' >> ' . self::$dataDir . 'xf_user_field_value.txt');
		// exec('mysql -h' . $this->slaveDbHost . ' ' . $this->sourceDb . ' ' . $this->sourceDBuser . ' ' . $this->sourceDBpassword . ' -N -q -e "SET NAMES binary;SELECT SQL_NO_CACHE userid AS user_id, \'last_bump_date\' AS field_id, field17 AS field_value FROM userfield WHERE field17 > 0 ORDER BY userid" ' . self::$extraOutCommand . ' >> ' . self::$dataDir . 'xf_user_field_value.txt');
		// exec('mysql -h' . $this->slaveDbHost . ' ' . $this->sourceDb . ' ' . $this->sourceDBuser . ' ' . $this->sourceDBpassword . ' -N -q -e "SET NAMES binary;SELECT SQL_NO_CACHE userid AS user_id, \'icq\' AS field_id, icq AS field_value FROM user WHERE icq != \'\' ORDER BY userid" ' . self::$extraOutCommand . ' >> ' . self::$dataDir . 'xf_user_field_value.txt');
		// exec('mysql -h' . $this->slaveDbHost . ' ' . $this->sourceDb . ' ' . $this->sourceDBuser . ' ' . $this->sourceDBpassword . ' -N -q -e "SET NAMES binary;SELECT SQL_NO_CACHE userid AS user_id, \'aim\' AS field_id, aim AS field_value FROM user WHERE aim != \'\' ORDER BY userid" ' . self::$extraOutCommand . ' >> ' . self::$dataDir . 'xf_user_field_value.txt');
		// exec('mysql -h' . $this->slaveDbHost . ' ' . $this->sourceDb . ' ' . $this->sourceDBuser . ' ' . $this->sourceDBpassword . ' -N -q -e "SET NAMES binary;SELECT SQL_NO_CACHE userid AS user_id, \'yahoo\' AS field_id, yahoo AS field_value FROM user WHERE yahoo != \'\' ORDER BY userid" ' . self::$extraOutCommand . ' >> ' . self::$dataDir . 'xf_user_field_value.txt');
		// exec('mysql -h' . $this->slaveDbHost . ' ' . $this->sourceDb . ' ' . $this->sourceDBuser . ' ' . $this->sourceDBpassword . ' -N -q -e "SET NAMES binary;SELECT SQL_NO_CACHE userid AS user_id, \'skype\' AS field_id, skype AS field_value FROM user WHERE skype != \'\' ORDER BY userid" ' . self::$extraOutCommand . ' >> ' . self::$dataDir . 'xf_user_field_value.txt');
		// exec('mysql -h' . $this->slaveDbHost . ' ' . $this->sourceDb . ' ' . $this->sourceDBuser . ' ' . $this->sourceDBpassword . ' -N -q -e "SET NAMES binary;SELECT SQL_NO_CACHE userid AS user_id, \'msn\' AS field_id, msn AS field_value FROM user WHERE msn != \'\' ORDER BY userid" ' . self::$extraOutCommand . ' >> ' . self::$dataDir . 'xf_user_field_value.txt');

It looked as if it did a partial import and all the navigation links in admincp were gone except Tools and there was only one option left in Tools (the test facebook integration link). And it showed 1 member in the stats, and when I clicked the home link it said I didn't have permission. I've since restored the db to where it was before and can navigate around just fine again.

You may have lacked admin permissions after the import. Admin permissions will need to be setup for any imported admins. Or just edit XF's library/config.php file and specify your imported admin's userid as a super admin.

Attaching the errors via a PDF file...

The nature of this importer is such that some mysql "strict" errors are expected. Most are harmless and can be ignored. From your log, the only thing that concerns me are errors like this:

xf_user_profile3306 Warning 1366
Incorrect integer value: 'mysql Ver 14.12 Distrib 5.0.96, for unknown-linux-gnu (x86_64) using readline 5.1' for column 'user_id' at row 1

Errors like this occur throughout your log. It's weird because it appears to be part of a console message and not part of the actual data in question. You may want to look into this further. But like I said, some "strict" errors are expected and can usually be ignored.

 

[Ludachris]

Actually, when I first ran the Import file I got a bunch of console messages (I think I shared them in previous posts). Turns out the db user and pass weren't working and I think that's why the console messages were coming back like that. Could it be that those console messages were stored or imported? Or, maybe there's a problem with what was exported. Can I clear all the data in the ImportData folder and try exporting again?

I'll go through and map the user profile fields and address the super admin issue. Thanks for the reply.

 

[Jake Bunce]

Could it be that those console messages were stored or imported?

Yeah, possibly.

Can I clear all the data in the ImportData folder and try exporting again?

No need to clear the folder. Re-running the Export will overwrite what's there.

 

[Ludachris]

Can anyone tell me roughly how long the Export and Import files should take to run on a board with 3.8 million posts and 140k users? All file uploads are stored in the file system. I just want to know if it should be a minute, a few minutes, a few hours. I have no idea what to expect in terms of what's normal. Thanks.

 

[Jake Bunce]

Can anyone tell me roughly how long the Export and Import files should take to run on a board with 3.8 million posts and 140k users? All file uploads are stored in the file system. I just want to know if it should be a minute, a few minutes, a few hours. I have no idea what to expect in terms of what's normal. Thanks.

I have done two imports of roughly 17 million posts. Both took about 2 hours total for Export and Import combined, excluding attachments.

 

[Ludachris]

Thanks Jake, does one script typically finish quicker than the other?

 

[Slavik]

Thanks Jake, is one script typically much shorter than the other?

It depends,

The attachments and avatars if you have a lot can take a long time if not pre-imported beforehand. Otherwise its mainly the import that takes the main bulk of the time.

 

[Ludachris]

How can I pre-import the avatars and attachments? Did I miss that in this thread somewhere?

 

[Slavik]

Just run those parts of the export file first.

 

[Ludachris]

I know you can choose to export those only, but how do you import them only? Since I had already exported data and did a failed import, will I have to clear the data import folder so that the import php file doesn't try to import everything in there? this way I can do a new export on only the avatars and attachments?

 

[Slavik]

You dont need to import those parts, the export does the whole process for those 2 things.

 

[Ludachris]

So if I ran the export file, then that means all the attachments and avatars should already be imported? I see the avatars but not the attachments.

 

[Ludachris]

Ha, found the problem with the attachments - my fault in the export file. Running the export file again now... got a db error that stopped it, but I just started running it again and it looks like it picks up right where it left off from the error. Will keep working at it.

 

[Ludachris]

@Jake Bunce

I have just about mapped all of my additional vb user profile fields in the Export.php file as suggested after replicating them in XF. I asked a related question in a different thread but realize now that I should have asked it here:

I have a profile field I'll be mapping from vb 3.8 that I have just set up in XF. It is a select menu field with 70 or so options. Is there a faster way to get those select options into XF than pasting each in one by one? Would there be a way to put them in via phpMyAdmin or some other way? @Mike mentioned the standard vB importer imports profile fields. Is there a way to use the standard importer to bring in the profile fields only? Just looking for the most efficient way to do this.

 

[Jake Bunce]

Is there a faster way to get those select options into XF than pasting each in one by one? Would there be a way to put them in via phpMyAdmin or some other way?

Custom programming is required to automate the process.

Is there a way to use the standard importer to bring in the profile fields only?

Actually that might work. You can try running the built-in vB importer up through that step. You will need to test this, but I think that a partial import for this purpose may work well.