Resource icon

vBulletin Big Board Importer [vBulletin 3 + vBulletin 4] [Paid] 1.5.0

No permission to buy ($150.00)

Slavik

XenForo moderator
Staff member
Slavik submitted a new resource:

vBulletin Big Board Importer [vBulletin 3 + vBulletin 4] - CLI based vBulletin Importer for large boards.

Today i'm happy to release the vBulletin Big Board Importer. This importer is based off the A.R.F.I CLI importer by Shawn at @digitalpoint and has been extensively modified by @Jake Bunce and @Slavik to expand its capabilities to include the following:
  • Imports from vBulletin 3
  • Imports from vBulletin 4
  • Attachments from the FileSystem or Database
  • Avatars from the FileSystem or Database
  • Incremental Attachment Imports
  • Optional Incremental Post Imports

This...

Read more about this resource...
 
Today i'm happy to release the vBulletin Big Board Importer (Smaller boards can use the free importers available for VB3 which is built into XenForo or VB4 at this link). This Big Board importer is based off the A.R.F.I CLI importer by Shawn at @digitalpoint and has been extensively modified by @Jake Bunce and @Slavik to expand its capabilities to include the following:
  • Imports from vBulletin 3
  • Imports from vBulletin 4
  • Attachments from the FileSystem or Database
  • Avatars from the FileSystem or Database
  • Incremental Attachment Imports
  • Optional Incremental Post Imports

This importer has had a lot of work put into it, and is priced as such at $150 which is broken down as follows:
  • $30 to Shawn Hogan for the original licensing fee for A.R.F.I
  • $60 to Slavik for alterations made to support generic vBulletin4 imports
  • $60 to Jake Bunce for alterations made to support vBulletin3 imports, tidying up a lot of the code and combining everything into 2 amazingly easy to use files.

Special thanks also to @Mike and @Robbo for input and assitance in getting things working.


Importer Technical Requirements
  • Linux or similar type of server (this will not work on Windows)
  • PHP with PCNTLextensions compiled into it
  • sed, split, & PERL need to be installed on server (pretty standard for any server)

Importer Board Requirements
  • The source database user MUST have GRANT ALL priviliges.
  • The files should be run on the database server for best speed (Remote MySQL will still work, just slower due to network bottleneck).
  • You may experience errors if the vBulletin default tables have been altered. If this is the case, you will need to revert them back to their default state.

Usage
  1. Install XenForo 1.1.5.
  2. Upload the Importer files to your XenForo directory.
  3. Open the Export.php and fill in the required details as commented from lines 22 through to 70
  4. Run the Exporter via CLI.
  5. Open the Importer.php file and fill in the relevent database informationand the temp data directory on line 45. Save the changes.
  6. Run the Importer via CLI.
  7. When finished, log into your AdminCP and rebuild all caches.

Whilst this importer is relatively easy to use and follow, it may still require a good level of knowledge in PHP, MySQL and Server Admin to tackle any issues you may encounter, as such, support is provided on a "Best Effort" basis rather than as a guarantee.

Managed imports are available on request at additoinal cost dependent on the forum size.

This addon must be purchased at DigitalPoint.com.
 
Last edited:
Why do you have to install xf 1.0.0 first?

Extra columns were added to a few XenForo tables between 1.0.0 and 1.1.4, and are required to be in an absolute fixed order. If you install 1.1.4, these columns may actually end up in a different order, and the importer wont be able to load the data in correctly.

Jake fixed this, so just install 1.1.5 directly
 
Last edited:
How does XF 1.1.5 factor in to this? Do that upgrade after everything else is functioning properly with 1.1.4?
 
Why do you have to install xf 1.0.0 first?

Hmm. I didn't.

One thing I fixed was some columns that were being inserted out of order with the default XF 1.1.5 installation into which I was importing. That could be related. When you upgrade to a new version with schema changes (versus installing to the new version straight up) it can sometimes result in a different order of columns. So the changes I made might have removed the requirement to install 1.0 first.
 
Why do you have to install xf 1.0.0 first?

See Jakes reply below!

How does XF 1.1.5 factor in to this? Do that upgrade after everything else is functioning properly with 1.1.4?

See Jakes reply, just use 1.1.5 from the outset now!

Hmm. I didn't.

One thing I fixed was some columns that were being inserted out of order with the default XF 1.1.5 installation into which I was importing. That could be related. When you upgrade to a new version with schema changes (versus installing to the new version straight up) it can sometimes result in a different order of columns. So the changes I made might have removed the requirement to install 1.0 first.


Well see how awesome Jake is? He even fixed this "bug".
 
Like this, but definitely please remove the unnecessary steps that just delay the import (if they haven't been yet). I'll be buying this as soon as 1.2 is out but I want to install that and import without having to **** around with one XF install and 2 subsequent upgrades.

Also, I don't get this requirement:

  • The default vBulletin tables must be unaltered by addons. If extra columns have been added, these will need to be dropped before importing.
Why aren't extra columns just ignored?
 
Like this, but definitely please remove the unnecessary steps that just delay the import (if they haven't been yet). I'll be buying this as soon as 1.2 is out but I want to install that and import without having to **** around with one XF install and 2 subsequent upgrades.

Also, I don't get this requirement:

  • The default vBulletin tables must be unaltered by addons. If extra columns have been added, these will need to be dropped before importing.
Why aren't extra columns just ignored?

I would expect them to be ignored. I see no reason why they would conflict. The export selects source columns by name.
 
Like this, but definitely please remove the unnecessary steps that just delay the import (if they haven't been yet). I'll be buying this as soon as 1.2 is out but I want to install that and import without having to **** around with one XF install and 2 subsequent upgrades.

Also, I don't get this requirement:

  • The default vBulletin tables must be unaltered by addons. If extra columns have been added, these will need to be dropped before importing.
Why aren't extra columns just ignored?

The importer is what it is, the requirements and considerations that enable it to be so fast are a more than fair tradeoff for a few extra steps added.

You may not have to remove those columns, however in 1 test I performed, an extra column in the vb_post table caused a whole load of errors until I dropped it, so have recommended to do this ahead of the export to reduce the likelyhood of any problems occuring.

Feel free to try and do an import without dropping them, but, if it does throw errors about it, you'll know why.
 
Like this, but definitely please remove the unnecessary steps that just delay the import (if they haven't been yet). I'll be buying this as soon as 1.2 is out but I want to install that and import without having to **** around with one XF install and 2 subsequent upgrades.

Also, I don't get this requirement:

  • The default vBulletin tables must be unaltered by addons. If extra columns have been added, these will need to be dropped before importing.
Why aren't extra columns just ignored?

Give it a test run with your vbulletin tables as they are. I didn't have to remove any added columns (vbseo, vaultwiki, etc). It seems in our case extra columns were ignored.
 
The importer is what it is, the requirements and considerations that enable it to be so fast are a more than fair tradeoff for a few extra steps added.

You may not have to remove those columns, however in 1 test I performed, an extra column in the vb_post table caused a whole load of errors until I dropped it, so have recommended to do this ahead of the export to reduce the likelyhood of any problems occuring.

Feel free to try and do an import without dropping them, but, if it does throw errors about it, you'll know why.

Actually, it is what you make it. ;)

And if source columns are picked by name, I can't see how additional ones could be problematic, but maybe I'm missing something.

And sure, I'll try without dropping them first and then see how that goes. Thanks for the info!
 
For those interested, here are the configurable settings I added to the top of the exporter:

Code:
	//Enter your vBulletin Forum details here
	protected $sourceDb = 'dbname';
	protected $slaveDbHost = 'localhost';	
	protected $sourceDBuser = '-udbuser';	
	protected $sourceDBpassword = '-pdbpass';	

	// IS VB4? IF NOT THEN VB3
	public static $isVB4 = false;

	// IF TRUE THEN IT CHECKS THE XF DATABASE FOR THE HIGHEST POSTID AND ONLY IMPORTS NEW POSTS FROM VB HIGHER THAN THAT ID
	public static $incrementalPosts = true;

	// PATH TO STORE INTERMEDIATE DATA
	public static $dataDir = '/home/path/public_html/importData/';

	// VB'S STORAGE DIRECTORIES
	// IF EMPTY ('') THEN IT LOOKS TO THE DATABASE FOR THE FILE DATA
	public static $avatarFile = '/home/path/public_html/forums/customavatars/';
	public static $profilePicFile = '/home/path/public_html/forums/customprofilepics/';
	public static $attachFile = '/home/path/attachments/';

	// USE THIS SETTING TO SPECIFY AN EXTRA PIPE FOR ALL OUTPUT RECORDS
	// EXAMPLE IS TO PIPE OUTPUT THROUGH iconv IF CHARACTER ENCODING CHANGES ARE NEEDED
	// '| iconv -f LATIN1 -t UTF8'
	public static $extraOutCommand = '| iconv -f LATIN1 -t UTF8';

	// MAP VB GROUP IDS TO XF GROUP IDS
	public static $groupMap = array(
				1 => 1,
				2 => 2,
				3 => 1,
				4 => 1,
				5 => 4,
				6 => 3,
				7 => 4,
				11 => 5,
				13 => 6, 
				28 => 7,
				29 => 8,
				30 => 9,
				31 => 10, 
				32 => 11,
				33 => 12,
				35 => 13,
				40 => 14,
				42 => 15,
				43 => 16,
				44 => 17,
				45 => 18,
			);

My testing has been limited to a vB3 forum with file system avatars and attachments. I haven't actually tested vB4 or db avatars / attachments since I made these changes, but it should work. I just added extra settings and conditions for the changes I had to make to support vB3 and file system avatars / attachments. Of course I am here to help with any specific problems you may encounter.
 
Just curious, now I'm not saying this is overpriced. I'm just wondering why importers like this cost so much?

Time. A LOT of hours went into making it. However many hours Shawn used to build the original. Add on a few weeks for my contributions, add on a couple more for Jakes.

Cost is also relative. 99% of all boards who will ever need this importer will be profit generating, and the cost of several days or more downtime is considered "cheap" in comparison to the cost of this importer
 
Couple of questions
  1. How much faster is this over the standard VB 3 importer?
  2. Does it handle VB3 paid subscriptions?
 
Couple of questions
  1. How much faster is this over the standard VB 3 importer?
  2. Does it handle VB3 paid subscriptions?

1) 18 million posts in 2 hours. I am guessing the web-based importer would take 3 days.

2) It imports the subscription records (xf_user_upgrade_active, xf_user_upgrade_expired, xf_user_upgrade_log) but not the subscriptions themselves (xf_user_upgrade). It appears the intention of this importer is that you recreate the paid subs in the XF Admin CP, paying special attention to the unique ids.

I will look into adding subs, but this may be difficult to generalize to all forums. There are individual considerations that you need to address. For example, vB supports multiple different payment processors but XF only supports Paypal. Callbacks for recurring payments will fail unless you redirect those requests.

The additional group memberships for the subscriptions are imported though.
 
2) It imports the subscription records (xf_user_upgrade_active, xf_user_upgrade_expired, xf_user_upgrade_log) but not the subscriptions themselves (xf_user_upgrade). It appears the intention of this importer is that you recreate the paid subs in the XF Admin CP, paying special attention to the unique ids.

I will look into adding subs, but this may be difficult to generalize to all forums. There are individual considerations that you need to address. For example, vB supports multiple different payment processors but XF only supports Paypal. Callbacks for recurring payments will fail unless you redirect those requests.

The additional group memberships for the subscriptions are imported though.

Thanks Jake. I only have a single paid subscription (non recurring) and use Paypal.
 
Top Bottom