vBulletin Big Board Importer [vBulletin 3 + vBulletin 4] [Paid] 1.5.0

[Jake Bunce]

2) OK, strangely I can see the forums now. Maybe it was a caching issue.

3) I don't quite understand this part. Are you saying that fixing the xf_user_field stuff in Export.php may cause the administrators and moderators and their moderated forums to be imported correctly. Here is my user group mapping:

// Created in the destination Xenforo
// 5-Supermods
// 6-Banned
// 7-Active
// 8-PMs
// 9-Summarizers
    // MAP VB GROUP IDS TO XF GROUP IDS
    public static $groupMap = array(
                1 => 1,
                2 => 2,
                3 => 1,
                4 => 1,
                5 => 5,
                6 => 3,
                7 => 4,
                8 => 6,
                18 => 7,
                21 => 8,
                23 => 9,
            );

The custom fields are unrelated.

The previous error of "Incorrect integer value: '' for column 'user_group_id'" is what concerned me. If the data and ordering is OK and the $groupMap is complete then I would not expect any invalid user_group_id's. Check that column in xf_user and look at the values. You should see nothing but numbers. If there are lots of invalid values then that is a problem which might be from bad ordering in the Export.

 

[zylstra]

That is the code which writes the user records to the intermediate data files. Basically you need to compare this ordering to the ordering that exists in your xf_user table in your XF database. The fields are tab-delimited. This was last aligned for XF 1.4 I believe. Maybe your XF forum has some addons which changed the ordering.

Fresh install, no addons. But I'll check out the code and try the migration again. Thanks!

 

[eva2000]

Nope. It's not a matter of grabbing some code from the official importer. The two importers handle the data differently. This big board importer uses data files to get the fastest possible performance, and those files are order-sensitive by nature.

If I get around to rewriting this importer then I would implement some system in the Export to automatically detect your column orders in the XF tables and then use those orders when writing out the data files. This as opposed to having the orders hard-coded which is the current situation. This is not a minor change and it would require a complete rewrite.

for performance, would actually doing the auto detection and order sensitive checks against the resulting created data files' data contents be faster and easier ?

 

[Jake Bunce]

for performance, would actually doing the auto detection and order sensitive checks against the resulting created data files' data contents be faster and easier ?

You can specify a column list during the import, but that requires knowing the order of the exported columns in the data file. I think modifying the export to match the order of the database is best, then the import order can remain implicit.

My first thought is to wrap all export queries in an outer query which enforces the order of columns. That would be very clean but there may be performance problems.

SELECT sub.post_id, sub.title, sub.dateline
FROM (
	<ORIGINAL QUERY>
) AS sub

The outer select list would be assembled based on the order of the columns in the XF database.

I haven't even started this. It's just a general idea I have.

 

[zylstra]

Here are the issues I'm still having:

1) No custom user fields are imported. I tried rewriting the import code for my instance to no avail.
2) No moderators or administrators are imported.
3) Categories that are deeper than the first level are listed as forums, not categories.

 

[Jake Bunce]

Here are the issues I'm still having:

1) No custom user fields are imported. I tried rewriting the import code for my instance to no avail.
2) No moderators or administrators are imported.
3) Categories that are deeper than the first level are listed as forums, not categories.

1) There is the code in the Export file which queries the vB data, and there is also this line in the Import file to actually import that data:

importTable('xf_user_field_value');

Make sure this line is uncommented.

2) This importer doesn't import those admin / mod records. You just have to manually re-add those statuses in XF after the first import:

Admin CP -> Users
> Moderators
> Administrators

Then they will persist.

3) Interestingly I never noticed that before. I can see in the Export code that it assumes categories and forums based on the parent level which is probably correct for most forums but it's not actually looking at the designated setting for this in vB:

// Nodes
$start = microtime(true);
echo " nodes";
exec('mysql -h' . $this->slaveDbHost . ' ' . $this->sourceDb . ' ' . $this->sourceDBuser . ' ' . $this->sourceDBpassword . ' -N -q -e "SET NAMES binary;SELECT SQL_NO_CACHE forum.forumid AS node_id, REPLACE(forum.title, \'&\', \'&\'), forum.description, IF(LENGTH(forum.link) > 0, SUBSTRING(forum.link, 51), \'\\\\\\N\') AS node_name, IF(forum.parentid = -1, \'Category\', IF(LENGTH(forum.link) > 0, \'LinkForum\', \'Forum\')) AS node_type_id, IF(forum.parentid = -1, 0, forum.parentid) AS parent_node_id, forum.displayorder AS display_order, IF(forum.options & 1, 1, 0) AS display_in_list, \'0\' AS lft, \'0\' AS rgt, \'\' AS depth, forum.styleid AS style_id, forum.styleid AS effective_style_id FROM ' . self::$tablePrefix . 'forum as forum ORDER BY forum.forumid" ' . self::$extraOutCommand . ' > ' . self::$dataDir . 'xf_node.txt');
exec(self::$sedCommand . " -i 's/\\\\N/N/g' " . self::$dataDir . "xf_node.txt");
echo " (" . number_format(intval(shell_exec('wc -l ' . self::$dataDir . 'xf_node.txt'))) . ' records in ' . number_format(microtime(true) - $start, 2) . "s)...\r\n";

// Forums
$start = microtime(true);
echo " forums";
exec('mysql -h' . $this->slaveDbHost . ' ' . $this->sourceDb . ' ' . $this->sourceDBuser . ' ' . $this->sourceDBpassword . ' -N -q -e "SET NAMES binary;SELECT SQL_NO_CACHE forum.forumid AS node_id, forum.threadcount AS discussion_count, forum.replycount AS message_count, forum.lastpostid AS last_post_id, forum.lastpost AS last_post_date, ' . (self::$isVB4 ? 'forum.lastposterid' : "'0'") . ' AS last_post_user_id, forum.lastposter AS last_post_username, forum.lastthread AS last_thread_title, IF(forum.options & 8, 1, 0) AS moderate_threads, IF(forum.options & 8, 1, 0) AS moderate_replies, IF(forum.options & 2, 1, 0) AS allow_posting, 1 AS allow_poll, IF(forum.options & 4096, \'1\', \'0\') AS count_messages, \'1\' AS find_new, \'\' AS prefix_cache, \'0\' AS default_prefix_id, \'last_post_date\' AS default_sort_order, \'desc\' AS default_sort_direction, 0 AS list_date_limit_days, \'0\' AS require_prefix, \'thread\' AS allowed_watch_notifications FROM ' . self::$tablePrefix . 'forum as forum WHERE forum.parentid > 0 ORDER BY forum.forumid" ' . self::$extraOutCommand . ' > ' . self::$dataDir . 'xf_forum.txt');
echo " (" . number_format(intval(shell_exec('wc -l ' . self::$dataDir . 'xf_forum.txt'))) . ' records in ' . number_format(microtime(true) - $start, 2) . "s)...\r\n";

For the first red piece you should use this to match vB categories:

NOT forum.options & 4

For the second red piece you should use this to match forums:

forum.options & 4

vB uses a bitfield for its forum options. The "4" bit is for the "Act as Forum" setting which is what defines a forum or category in vB.

 

[zylstra]

1) I checked that the line was uncommented and that the xf_user_field_value.txt did have data in it. ... Ah, I think I just figured it out. I have to create the custom user fields first, huh? Then do the import?
2) OK. I think grabbing some tables from the standard import may work too. I'll try that and report back eventually.
3) For this I may try the standard import tables or I may change them manually. There are only 30 or so at most.

 

[zylstra]

I'm not sure if this is a big board importer issue or a general Xenforo issue. I analyze a user's permission to view a node, and the answer is yes. Then I test that user's permission and cannot see the node. Why is that?

 

[zylstra]

Sadik B recommended I toggle/save a usergroup and node permission back and forth. Apparently that rebuilds the permissions to the stated values.

 

[zylstra]

I've been working on this for the past 12 hours, and it's way past my bedtime, and what I wrote is very unfinished, but I'm excited to share it:

The Unofficial Guide for VBulletin to Xenforo Migration with the Big Board Importer

I hope you like it! Feel free to help me improve it. What do you think @Jake Bunce ? Perhaps a link to it in your Usage section on the Overview page will be helpful for future migraters.

 

[thedude]

I've been working on this for the past 12 hours, and it's way past my bedtime, and what I wrote is very unfinished, but I'm excited to share it:

The Unofficial Guide for VBulletin to Xenforo Migration with the Big Board Importer

I hope you like it! Feel free to help me improve it. What do you think @Jake Bunce ?

Nice start!

I'll try to isolate the changes I made so unicode characters worked properly (we rewrote big parts of the importer). In the meantime, do the test I mentioned here by creating a thread with extended characters in the title, post, usertitle of the vb version of the site to see if you're affected by that issue once it imports into the xf database.

 

[zylstra]

I did that test and I wasn't affected, but I can certainly put it in the guide.

 

[eva2000]

I've been working on this for the past 12 hours, and it's way past my bedtime, and what I wrote is very unfinished, but I'm excited to share it:

The Unofficial Guide for VBulletin to Xenforo Migration with the Big Board Importer

I hope you like it! Feel free to help me improve it. What do you think @Jake Bunce ?

nice guide there.. might want to add bigboard impoter's technical requirements too.. php requiring pcntl extension which might not be installed in common php setups

Importer Technical Requirements

• Linux or similar type of server (this will not work on Windows)
• PHP with PCNTL extensions compiled into it
• sed, split, & PERL need to be installed on server (pretty standard for any server)

 

[zylstra]

3) Interestingly I never noticed that before. I can see in the Export code that it assumes categories and forums based on the parent level which is probably correct for most forums but it's not actually looking at the designated setting for this in vB

I recommend keeping it this way. Here the BBI does better than the standard importer, since categories are useless - basically forums without most of the functionality of forums.

 

[CreationNation]

Heya @Jake Bunce - I recall seeing you post about this but can't find it.

I am done on my dev server, so now am going to import to my new server area for my VB > XF conversion. Once I import, set permissions, do the addons, tweak, styles, etc. - do I then run the import script again (with live forum paused) to grab any new users/posts, then I can launch? Mainly my question is about, getting those users/posts since previous import (may be a couple days apart).

 

[Alpha1]

I've been working on this for the past 12 hours, and it's way past my bedtime, and what I wrote is very unfinished, but I'm excited to share it:

The Unofficial Guide for VBulletin to Xenforo Migration with the Big Board Importer

I hope you like it! Feel free to help me improve it.

Nice! I hope to see chapter 2 expanded.

 

[zylstra]

I may be modifying it a bit in the future since I haven't done my migration yet. If you give me some pointers as to what you would like to see in the Intro (chapter 2) I may expand it.

 

[CreationNation]

I'm getting a lot of issues that I didn't experience on my initial export to the dev server. Can anyone shed some light on these?

When exporting posts:
postssed: couldn't write 296 items to stdout: Broken pipe
sh: line 1: 29124 Done
[mysql query output]
29125 Done(4)
[bbcode stuff]
29126 Killed
iconv -f LATIN1 -t UTF8 > /home/[mypath]/forum/importData/xf_post.txt
rm: cannot remove `/home/[mypath]/forum/importData/xf_post_*': No such file or directory

 

[thedude]

I'm getting a lot of issues that I didn't experience on my initial export to the dev server. Can anyone shed some light on these?

When exporting posts:
postssed: couldn't write 296 items to stdout: Broken pipe
sh: line 1: 29124 Done
[mysql query output]
29125 Done(4)
[bbcode stuff]
29126 Killed
iconv -f LATIN1 -t UTF8 > /home/[mypath]/forum/importData/xf_post.txt
rm: cannot remove `/home/[mypath]/forum/importData/xf_post_*': No such file or directory

Search this thread for "broken pipe" and you'll find the posts about how to resolve it

 

[CreationNation]

Thanks. I had searched before but must have got sidetracked.

I see it mentions running iconv -l and making sure UTF8 is in there (it is). I'm scratching my head on my I may have to use the chunks method when it seemed to do ok before, but maybe I will look into that.