1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

vBulletin Big Board Importer [vBulletin 3 + vBulletin 4] [Paid] 1.5.0

CLI based vBulletin Importer for large boards.

  1. Slavik

    Slavik XenForo Moderator Staff Member

    Slavik submitted a new resource:

    vBulletin Big Board Importer [vBulletin 3 + vBulletin 4] - CLI based vBulletin Importer for large boards.

    Read more about this resource...
     
    Digital Doctor likes this.
  2. Slavik

    Slavik XenForo Moderator Staff Member

    Today i'm happy to release the vBulletin Big Board Importer (Smaller boards can use the free importers available for VB3 which is built into XenForo or VB4 at this link). This Big Board importer is based off the A.R.F.I CLI importer by Shawn at @digitalpoint and has been extensively modified by @Jake Bunce and @Slavik to expand its capabilities to include the following:
    • Imports from vBulletin 3
    • Imports from vBulletin 4
    • Attachments from the FileSystem or Database
    • Avatars from the FileSystem or Database
    • Incremental Attachment Imports
    • Optional Incremental Post Imports

    This importer has had a lot of work put into it, and is priced as such at $150 which is broken down as follows:
    • $30 to Shawn Hogan for the original licensing fee for A.R.F.I
    • $60 to Slavik for alterations made to support generic vBulletin4 imports
    • $60 to Jake Bunce for alterations made to support vBulletin3 imports, tidying up a lot of the code and combining everything into 2 amazingly easy to use files.

    Special thanks also to @Mike and @Robbo for input and assitance in getting things working.


    Importer Technical Requirements
    • Linux or similar type of server (this will not work on Windows)
    • PHP with PCNTLextensions compiled into it
    • sed, split, & PERL need to be installed on server (pretty standard for any server)

    Importer Board Requirements
    • The source database user MUST have GRANT ALL priviliges.
    • The files should be run on the database server for best speed (Remote MySQL will still work, just slower due to network bottleneck).
    • You may experience errors if the vBulletin default tables have been altered. If this is the case, you will need to revert them back to their default state.

    Usage
    1. Install XenForo 1.1.5.
    2. Upload the Importer files to your XenForo directory.
    3. Open the Export.php and fill in the required details as commented from lines 22 through to 70
    4. Run the Exporter via CLI.
    5. Open the Importer.php file and fill in the relevent database informationand the temp data directory on line 45. Save the changes.
    6. Run the Importer via CLI.
    7. When finished, log into your AdminCP and rebuild all caches.

    Whilst this importer is relatively easy to use and follow, it may still require a good level of knowledge in PHP, MySQL and Server Admin to tackle any issues you may encounter, as such, support is provided on a "Best Effort" basis rather than as a guarantee.

    Managed imports are available on request at additoinal cost dependent on the forum size.

    This addon must be purchased at DigitalPoint.com.
     
    Last edited: Jun 20, 2013
    MattW and Alien like this.
  3. Matthew Hawley

    Matthew Hawley Well-Known Member

    Why do you have to install xf 1.0.0 first?
     
  4. Slavik

    Slavik XenForo Moderator Staff Member

    Extra columns were added to a few XenForo tables between 1.0.0 and 1.1.4, and are required to be in an absolute fixed order. If you install 1.1.4, these columns may actually end up in a different order, and the importer wont be able to load the data in correctly.

    Jake fixed this, so just install 1.1.5 directly
     
    Last edited: Jun 18, 2013
  5. Alien

    Alien Well-Known Member

    How does XF 1.1.5 factor in to this? Do that upgrade after everything else is functioning properly with 1.1.4?
     
  6. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    Hmm. I didn't.

    One thing I fixed was some columns that were being inserted out of order with the default XF 1.1.5 installation into which I was importing. That could be related. When you upgrade to a new version with schema changes (versus installing to the new version straight up) it can sometimes result in a different order of columns. So the changes I made might have removed the requirement to install 1.0 first.
     
  7. Slavik

    Slavik XenForo Moderator Staff Member

    See Jakes reply below!

    See Jakes reply, just use 1.1.5 from the outset now!


    Well see how awesome Jake is? He even fixed this "bug".
     
  8. Gladius

    Gladius Well-Known Member

    Like this, but definitely please remove the unnecessary steps that just delay the import (if they haven't been yet). I'll be buying this as soon as 1.2 is out but I want to install that and import without having to **** around with one XF install and 2 subsequent upgrades.

    Also, I don't get this requirement:

    • The default vBulletin tables must be unaltered by addons. If extra columns have been added, these will need to be dropped before importing.
    Why aren't extra columns just ignored?
     
  9. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    I would expect them to be ignored. I see no reason why they would conflict. The export selects source columns by name.
     
    Gladius likes this.
  10. The Forum Heroes

    The Forum Heroes Well-Known Member

    Nice job guys!
     
  11. Slavik

    Slavik XenForo Moderator Staff Member

    The importer is what it is, the requirements and considerations that enable it to be so fast are a more than fair tradeoff for a few extra steps added.

    You may not have to remove those columns, however in 1 test I performed, an extra column in the vb_post table caused a whole load of errors until I dropped it, so have recommended to do this ahead of the export to reduce the likelyhood of any problems occuring.

    Feel free to try and do an import without dropping them, but, if it does throw errors about it, you'll know why.
     
  12. kontrabass

    kontrabass Well-Known Member

    Give it a test run with your vbulletin tables as they are. I didn't have to remove any added columns (vbseo, vaultwiki, etc). It seems in our case extra columns were ignored.
     
    Gladius and Slavik like this.
  13. Gladius

    Gladius Well-Known Member

    Actually, it is what you make it. ;)

    And if source columns are picked by name, I can't see how additional ones could be problematic, but maybe I'm missing something.

    And sure, I'll try without dropping them first and then see how that goes. Thanks for the info!
     
  14. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    For those interested, here are the configurable settings I added to the top of the exporter:

    Code:
    	//Enter your vBulletin Forum details here
    	protected $sourceDb = 'dbname';
    	protected $slaveDbHost = 'localhost';	
    	protected $sourceDBuser = '-udbuser';	
    	protected $sourceDBpassword = '-pdbpass';	
    
    	// IS VB4? IF NOT THEN VB3
    	public static $isVB4 = false;
    
    	// IF TRUE THEN IT CHECKS THE XF DATABASE FOR THE HIGHEST POSTID AND ONLY IMPORTS NEW POSTS FROM VB HIGHER THAN THAT ID
    	public static $incrementalPosts = true;
    
    	// PATH TO STORE INTERMEDIATE DATA
    	public static $dataDir = '/home/path/public_html/importData/';
    
    	// VB'S STORAGE DIRECTORIES
    	// IF EMPTY ('') THEN IT LOOKS TO THE DATABASE FOR THE FILE DATA
    	public static $avatarFile = '/home/path/public_html/forums/customavatars/';
    	public static $profilePicFile = '/home/path/public_html/forums/customprofilepics/';
    	public static $attachFile = '/home/path/attachments/';
    
    	// USE THIS SETTING TO SPECIFY AN EXTRA PIPE FOR ALL OUTPUT RECORDS
    	// EXAMPLE IS TO PIPE OUTPUT THROUGH iconv IF CHARACTER ENCODING CHANGES ARE NEEDED
    	// '| iconv -f LATIN1 -t UTF8'
    	public static $extraOutCommand = '| iconv -f LATIN1 -t UTF8';
    
    	// MAP VB GROUP IDS TO XF GROUP IDS
    	public static $groupMap = array(
    				1 => 1,
    				2 => 2,
    				3 => 1,
    				4 => 1,
    				5 => 4,
    				6 => 3,
    				7 => 4,
    				11 => 5,
    				13 => 6, 
    				28 => 7,
    				29 => 8,
    				30 => 9,
    				31 => 10, 
    				32 => 11,
    				33 => 12,
    				35 => 13,
    				40 => 14,
    				42 => 15,
    				43 => 16,
    				44 => 17,
    				45 => 18,
    			);
    
    My testing has been limited to a vB3 forum with file system avatars and attachments. I haven't actually tested vB4 or db avatars / attachments since I made these changes, but it should work. I just added extra settings and conditions for the changes I had to make to support vB3 and file system avatars / attachments. Of course I am here to help with any specific problems you may encounter.
     
    Gladius and Slavik like this.
  15. Matthew Hawley

    Matthew Hawley Well-Known Member

    Just curious, now I'm not saying this is overpriced. I'm just wondering why importers like this cost so much?
     
  16. Slavik

    Slavik XenForo Moderator Staff Member

    Time. A LOT of hours went into making it. However many hours Shawn used to build the original. Add on a few weeks for my contributions, add on a couple more for Jakes.

    Cost is also relative. 99% of all boards who will ever need this importer will be profit generating, and the cost of several days or more downtime is considered "cheap" in comparison to the cost of this importer
     
    feldon30, DeltaHF, Mark3121 and 3 others like this.
  17. 0ptima

    0ptima Well-Known Member

    Couple of questions
    1. How much faster is this over the standard VB 3 importer?
    2. Does it handle VB3 paid subscriptions?
     
  18. Jake Bunce

    Jake Bunce XenForo Moderator Staff Member

    1) 18 million posts in 2 hours. I am guessing the web-based importer would take 3 days.

    2) It imports the subscription records (xf_user_upgrade_active, xf_user_upgrade_expired, xf_user_upgrade_log) but not the subscriptions themselves (xf_user_upgrade). It appears the intention of this importer is that you recreate the paid subs in the XF Admin CP, paying special attention to the unique ids.

    I will look into adding subs, but this may be difficult to generalize to all forums. There are individual considerations that you need to address. For example, vB supports multiple different payment processors but XF only supports Paypal. Callbacks for recurring payments will fail unless you redirect those requests.

    The additional group memberships for the subscriptions are imported though.
     
  19. 0ptima

    0ptima Well-Known Member

    Thanks Jake. I only have a single paid subscription (non recurring) and use Paypal.
     
  20. 0ptima

    0ptima Well-Known Member

    Will this be updated to include post edit history for XF 1.2?
     
    Alfa1 likes this.

Share This Page