vBulletin Big Board Importer [vBulletin 3 + vBulletin 4] [Paid] 1.5.0

[Jake Bunce]

Hi,

I'm getting the 'Sed' related error while exporting posts. "sed: couldn't write 225 items to stdout: Broken pipe".
Does any one know what does it mean?


Here is the output

1. ################################################
2. ### XenForo Big Board vB3/vB4 -> XF Importer ###
3. ### Original version created by Shawn Hogan ###
4. ### Modified by Slavik and Jake Bunce ###
5. ### Version Number: 1.2.0 ###
6. ################################################
7. 
   
8. 1 - Posts
9. 2 - Users
10. 3 - Threads
11. 4 - Private Messages
12. 5 - Avatars
13. 6 - Attachments
14. 7 - Forums
15. 8 - Miscellaneous
16. 9 - Everything
17. 
    
18. Make Selection [1-9]: 1
19. 
    
20. exporting posts...
21. postsWarning: Using a password on the command line interface can be insecure.
22. sed: couldn't write 225 items to stdout: Broken pipe

Try emptying this setting in the Export script:

	// USE THIS SETTING TO SPECIFY AN EXTRA PIPE FOR ALL OUTPUT RECORDS
	// EXAMPLE IS TO PIPE OUTPUT THROUGH iconv IF CHARACTER ENCODING CHANGES ARE NEEDED
	// '| iconv -f LATIN1 -t UTF8'
	public static $extraOutCommand = '| iconv -f LATIN1 -t UTF8';

A broken pipe isn't necessarily caused by iconv, but removing it has fixed this problem for me before.

 

[David Denton]

I don't see this error after increasing the RAM on my machine.

 

[David Denton]

After data import I don't see Node to user group permissions are set. Is this not part of this plugin?

Thanks,

 

[Jake Bunce]

After data import I don't see Node to user group permissions are set. Is this not part of this plugin?

Thanks,

Permissions are not imported. You need to manually setup your permissions. But once they are setup then they will persist through any number of test imports.

 

[NikC]

Has anyone tested this to see if it works with xF 1.4 yet guys?

 

[Jake Bunce]

Has anyone tested this to see if it works with xF 1.4 yet guys?

I haven't. But after reviewing the upgrade steps I believe there is only one issue. 1.4 drops xf_poll.multiple, so you need to remove $poll[multiple] from this code (and the tab before it).

$fp = fopen('' . self::$dataDir . 'xf_poll.txt', 'w');
		foreach ($polls as $poll)
		{
			fwrite($fp, "$poll[poll_id]	$poll[content_type]	$poll[content_id]	$poll[question]	$poll[responses]	$poll[voter_count]	$poll[public_votes]	$poll[multiple]	$poll[close_date]\n");
		}

Multiple votes is now specified by xf_poll.max_votes = 0, so the code can be adjusted to specify this when multiple is true.

 

[CMGDevel]

This product definitely has my attention as I will be importing many vb databases over the next several months. Definitely will be needed for the bigger ones. What I'm not clear on is if it adds any import features over and above the stock importer. For example does it replicate permissions, usergroup promotions and recurring subscriptions?

 

[Jake Bunce]

This product definitely has my attention as I will be importing many vb databases over the next several months. Definitely will be needed for the bigger ones. What I'm not clear on is if it adds any import features over and above the stock importer. For example does it replicate permissions, usergroup promotions and recurring subscriptions?

The stock importer does the same stuff and is the recommended importer. This big board importer is just much faster for extremely large forums (millions of posts).

The stock importer does permissions. The big board importer does not.

Neither importer does promotions.

Recurring subscriptions are a problem for both importers due to the Paypal callbacks.

 

[Superfluous]

Is it compatible with 1.4? (I've just read half this thread for various issues and haven't noticed this question answered, although I'm sure it has to be somewhere!)

My first try took 3.5 hours with a 7 million post forum and failed miserably. My admin account didn't even have rights to view the forums. I'm going again now after giving it a completely clean install.

 

[Superfluous]

Just realised that I was leaving out a trailing slash when defining the directories (i.e. public static $dataDir = '/var/www/vhosts/***.com/***/importData/' Hopefully fixing that will give me a cleaner result this time.

Anyway - does anyone know if there is any issue in performing a few of the steps at the same time using different SSH sessions (for instance export posts, and users and threads all individually?).

Thanks.

 

[Jake Bunce]

Anyway - does anyone know if there is any issue in performing a few of the steps at the same time using different SSH sessions (for instance export posts, and users and threads all individually?).

The Export script already forks itself when you run all steps (option 9).

 

[Jake Bunce]

Is it compatible with 1.4? (I've just read half this thread for various issues and haven't noticed this question answered, although I'm sure it has to be somewhere!)

My first try took 3.5 hours with a 7 million post forum and failed miserably. My admin account didn't even have rights to view the forums. I'm going again now after giving it a completely clean install.

Yes it works with 1.4:

https://xenforo.com/community/threa...-3-vbulletin-4-paid.52796/page-37#post-835654

After running the Import you need to rebuild the user cache:

Admin CP -> Tools -> Rebuild Caches

Rebuilding the user cache will calculate the permission combination ids in the user records thereby granting users access from their usergroups. If you still don't have access at that point then you probably need to setup permissions for your usergroups.

Also, you need to rebuild the forum information in the same area following the Import.

 

[Superfluous]

Thanks for the quick reply, very helpful.

Unfortunately after the import I can no longer access the Admin CP because the account doesn't have permissions (to rebuild permissions!) - any ideas?

Edit, I've also tried adding my user ID to superAdmins, but I'm not even sure I'm using the right ID with the merged user (I've tried new and old)

Edit 2, I've confirmed that the merge ID is the right one, but I'm still not getting in. I've even tried SQL updating my user_group_id in the users table to 3, but no joy.

 

[Ridemonkey]

Edit 2, I've confirmed that the merge ID is the right one, but I'm still not getting in. I've even tried SQL updating my user_group_id in the users table to 3, but no joy.

Modify the is_admin field to 1 in the Users table.

 

[Superfluous]

Modify the is_admin field to 1 in the Users table.

Genius.

 

[Ridemonkey]

I was in the same spot when I did my import. Modified the user group, nothing, and finally realized that I hadn't examined the table carefully enough to find the is_admin column.

 

[Superfluous]

Yes, certainly a learning curve. Out of general curiosity (as I'll figure this out eventually, I'm guessing) but when I'm finished doing my trial run of importing, when I import again does it only import new things, or does it do the entire process from scratch?

I ask because I've been rebuilding 'user' caches for about 3 hours alone now, and I'm starting to get worried that this may actually be a full day of downtime, even with the big board importer.

 

[Jake Bunce]

Yes, certainly a learning curve. Out of general curiosity (as I'll figure this out eventually, I'm guessing) but when I'm finished doing my trial run of importing, when I import again does it only import new things, or does it do the entire process from scratch?

I ask because I've been rebuilding 'user' caches for about 3 hours alone now, and I'm starting to get worried that this may actually be a full day of downtime, even with the big board importer.

It's from scratch into your existing XF installation. It's the same installation so styles, permissions, etc are preserved during a reimport, but the imported content is nuked and redone such as users and posts.

Yes, you need to re-run the user cache after each import.

 

[Ridemonkey]

I ask because I've been rebuilding 'user' caches for about 3 hours alone now, and I'm starting to get worried that this may actually be a full day of downtime, even with the big board importer.

3 hours? Yikes - how many users?

I brought my site back online long before all my caches were rebuilt. I did rebuild all user caches (IIRC, I drastically increased the number of users per page to something like 1000), but then I disabled search, made sure my .htaccess redirects were in place, and brought the site back up. Posted an announcement to expect weirdness and slowness, and went about rebuilding caches while the site was online.

 

[Superfluous]

3 hours? Yikes - how many users?

I brought my site back online long before all my caches were rebuilt. I did rebuild all user caches (IIRC, I drastically increased the number of users per page to something like 1000), but then I disabled search, made sure my .htaccess redirects were in place, and brought the site back up. Posted an announcement to expect weirdness and slowness, and went about rebuilding caches while the site was online.

I think something's going very wrong - I've only got about 25k users and it took 3 hours to rebuild them. I've been rebuilding my threads for about 5 hours now and it's only on thread 175k at the minute. None of the avatars are appearing beside the users and Profile Messages don't seem to have been imported. I'm on a Xeon 6 core, 32gb RAM dedicated box btw (although it's an almost freshly imaged box, might need some tweeking for server settings such mysql, php etc).

I'm getting thousands of these errors in the log as I think the bots are hitting my temp domain

ErrorException: unserialize(): Error at offset 0 of 14 bytes - library/XenForo/Model/Forum.php:253
Generated By: Unknown Account, A moment ago

Stack Trace

#0 [internal function]: XenForo_Application::handlePhpError(8, 'unserialize(): ...', '/var/www/vhosts...', 253, Array)

Anyone have any idea if this is the problem?