vBulletin Big Board Importer [vBulletin 3 + vBulletin 4] [Paid] 1.5.0

[thedude]

In Exporter.php, this line

'response' => str_replace('\t', ' ', trim($option)),

should actually be

'response' => str_replace("\t", ' ', trim($option)),

(needed double quotes on the \t ....at least for vb 3 the vote options can contain a tab literal)

 

[Jake Bunce]

A bug in xf_user column ordering. This is the corrected Export line:

					fwrite($fp_user, str_replace('\\', '\\\\', "$user[userid]	" . html_entity_decode($user['username']) . "	$user[email]		" . ($user['customtitle'] ? html_entity_decode($user['usertitle']) : '') . "	1	0	" . $importModel->resolveTimeZoneOffset($user['timezoneoffset'], $user['options'] & 64) . "	1	1	$user[usergroupid]	$user[membergroupids]	$userGroupDisplay	$permissionComboId	$user[posts]	$user[pmunread]	$user[joindate]	$user[lastactivity]	0	0	$user[avatar_date]	192	192		" . ($user['usergroupid'] == 1 ? 'email_confirm' : 'valid') . "	" . ($user['usergroupid'] == 4 ? '1' : '0') . "	" . ($user['usergroupid'] == 3 ? '1' : '0') . "	" . ($is_banned ? 1 : 0) . "	$user[like_count]	$user[ipoints]	" . (($user['usergroupid'] == 4 OR $user['usergroupid'] == 3) ? '1' : '0')) . "\n");

There was previously too many tabs between email and customtitle.

@Slavik

 

[Slavik]

Slavik updated vBulletin Big Board Importer [vBulletin 3 + vBulletin 4] with a new update entry:

Bug Fixes

Various assorted fixes.

Read the rest of this update entry...

 

[arn]

Can someone guide me as to which Caches must be rebuilt before opening the forum to the public?

I presume these must be:

- User Caches
- Forum Information

But, it's not clear to me if all the others must be rebuilt before re-opening a forum after import.

thanks
arn

 

[Slavik]

Basically run them all, apart from dont tick the recalculate post positions.

 

[arn]

Basically run them all, apart from dont tick the recalculate post positions.

I planned on running them all eventually... but my question was more to what needs to be run before opening to public. We are doing a migration and want to reduce our downtime as much as possible.

arn

 

[Slavik]

Users forums and threads.

 

[Jake Bunce]

Definitely users and forums. I don't believe threads need to be rebuilt... it doesn't hurt, but it shouldn't be required.

 

[arn]

thanks for replies...

When importing users, my moderators/admins aren't automatically inserted into the Moderator and Admin lists in the control panel. It appears these lists are maintained by xf_moderator and xf_admin tables. How are these supposed to be populated after import? I rebuilt user caches but this didn't work. The users have "is_admin" and "is_moderator" fields as "1" in their xf_user table.

 

[thedude]

thanks for replies...

When importing users, my moderators/admins aren't automatically inserted into the Moderator and Admin lists in the control panel. It appears these lists are maintained by xf_moderator and xf_admin tables. How are these supposed to be populated after import? I rebuilt user caches but this didn't work. The users have "is_admin" and "is_moderator" fields as "1" in their xf_user table.

Look at them in your vb installation. If their primary usergroup was administrator or moderator, then what you need to do after the import in xf is change their primaries to Registered User and then tick off Moderator for the mods and Administrator and Moderator for the admin.

 

[arn]

Look at them in your vb installation. If their primary usergroup was administrator or moderator, then what you need to do after the import in xf is change their primaries to Registered User and then tick off Moderator for the mods and Administrator and Moderator for the admin.

ok, so it's simply a manual process after you import to make them true moderators/admins? I don't mind doing that, it's just hard to tell if it's a botched import or that's how the import works.

 

[thedude]

ok, so it's simply a manual process after you import to make them true moderators/admins? I don't mind doing that, it's just hard to tell if it's a botched import or that's how the import works.

No it's working properly. The issue is actually how you have it setup on the vb side. I know this because we had the same problem. Apparently the correct way to have it on vb is the same as I described... Both mods and admins as registered users, with the mod/admin as their secondary group. That would allow the xf big board importer to import it properly.

However at this point, I didn't wanna make such a change in the vb install since I didn't know what any side effects would be, so I just do the manual change in xf after the import.

 

[arn]

No it's working properly. The issue is actually how you have it setup on the vb side. I know this because we had the same problem. Apparently the correct way to have it on vb is the same as I described... Both mods and admins as registered users, with the mod/admin as their secondary group. That would allow the xf big board importer to import it properly.

However at this point, I didn't wanna make such a change in the vb install since I didn't know what any side effects would be, so I just do the manual change in xf after the import.

Ah, I think we are talking about different things. So, right, I get that. but did your Moderators appear in this list without any manual intervention? Moderators are a special class beyond just placing them as a 2ndary user group.

In the screenshot, I'd expect to see a dozen other usernames, but it just has mine.

 

[thedude]

Ah, I think we are talking about different things. So, right, I get that. but did your Moderators appear in this list without any manual intervention? Moderators are a special class beyond just placing them as a 2ndary user group.

In the screenshot, I'd expect to see a dozen other usernames, but it just has mine.

You are correct. I just double checked our test server and there were no other moderators added to the xf_moderator table other than the admin.

When you add a moderator to it, via Create New Moderator, there are a ton of permissions you can choose to set, so it's not realistically possible for the importer to automatically pick and choose the permissions for you on an import.

I added this step to my post import check-list. Good catch

 

[Jake Bunce]

thanks for replies...

When importing users, my moderators/admins aren't automatically inserted into the Moderator and Admin lists in the control panel. It appears these lists are maintained by xf_moderator and xf_admin tables. How are these supposed to be populated after import? I rebuilt user caches but this didn't work. The users have "is_admin" and "is_moderator" fields as "1" in their xf_user table.

Yes, moderators and admins are a special class beyond just group memberships. Their special admin and moderator permissions are defined in:

Admin CP -> Users
> Moderators
> Administrators

With respect to this big board importer, it doesn't do any permissions (groups, mods, admins, or forum-specific permissions). You need to manually recreate all permissions in XF as appropriate. It's a one-time task. The permissions will persist since repeated imports are run against the same installation of XF. So... you recreate the mod/admin entries for those users in XF. Those entries are recorded in the database by user_id, and the user_ids for imported users are preserved each time you import. So on the next import those mod/admin entries are automatically associated again. You just need to rebuild the user cache after each import in order to update the cached permission values for the newly imported users.

 

[Jake Bunce]

Look at them in your vb installation. If their primary usergroup was administrator or moderator, then what you need to do after the import in xf is change their primaries to Registered User and then tick off Moderator for the mods and Administrator and Moderator for the admin.

You have a choice here. The default XF behavior is to put all users in Registered as the primary group. The advantage of this is that you can use that group to define common permissions for all users. But it is perfectly valid to have different primary groups for different users.

This importer will preserve the group memberships from vB, and vB's default behavior is to use different primary groups for mods / admins. You can keep this arrangement in XF, you just need to setup your group permissions appropriately.

 

[arn]

With respect to this big board importer, it doesn't do any permissions (groups, mods, admins, or forum-specific permissions). You need to manually recreate all permissions in XF as appropriate. It's a one-time task. The permissions will persist since repeated imports are run against the same installation of XF. So... you recreate the mod/admin entries for those users in XF. Those entries are recorded in the database by user_id, and the user_ids for imported users are preserved each time you import. So on the next import those mod/admin entries are automatically associated again. You just need to rebuild the user cache after each import in order to update the cached permission values for the newly imported users.

Ah, I think that was the missing piece for me. I have been redoing imports into an empty Xenforo install.

I guess the procedure should be: Import, set up permissions, and reimport overtop of your Xenforo install? No need to empty the Xenforo installation each time?

arn

 

[Jake Bunce]

I guess the procedure should be: Import, set up permissions, and reimport overtop of your Xenforo install? No need to empty the Xenforo installation each time?

Correct.

 

[arn]

Try emptying this setting in the Export script:

    // USE THIS SETTING TO SPECIFY AN EXTRA PIPE FOR ALL OUTPUT RECORDS
    // EXAMPLE IS TO PIPE OUTPUT THROUGH iconv IF CHARACTER ENCODING CHANGES ARE NEEDED
    // '| iconv -f LATIN1 -t UTF8'
    public static $extraOutCommand = '| iconv -f LATIN1 -t UTF8';

A broken pipe isn't necessarily caused by iconv, but removing it has fixed this problem for me before.

@Jake Bunce @David Denton - Not sure if you ever resolved this. But I had a similar issue with sed broken pipe. It turns out that iconv does poorly with large data sets and will run out of memory. The solution is to replace it with iconv-chunks, which saves to temp files along the way.

I used it like this in Export.php

       public static $extraOutCommand = '| /path/to/iconv-chunks - -f LATIN1 -t UTF8';

If you remove iconv completely, it screws up character sets in the import.

 

[arn]

The only outstanding issue with using this importer for XF 1.4 is the xf_ip table. It was actually XF 1.3 which added support for IPv6, and this importer will need the various IP sections rewritten to conform to the new format. If you don't care about importing your IP logs then it doesn't matter. I have commented out this line in the Import.php script so it doesn't import the IPs in the meantime:

//importTable('xf_ip');

When I get time to do this... my inclination will probably be to import IPs using the old format and then copy some code from the XF 1.3 upgrade script to make the appropriate conversions after-the-fact.

Jake,

Any idea how soon this would be coming? I'd much prefer to keep IP log data. It will introduce headaches along the way I think for our mod team if all IP history is wiped.

Besides xf_ip, there's also an extra ip_id column in xf_conversation_message that isn't being filled into. That may be on purpose, but the exporter doesn't even put a dummy value in for the export. I've padded it in the meanwhile, to stop the Warnings on import.

arn