vBulletin Big Board Importer [vBulletin 3 + vBulletin 4] [Paid] 1.5.0

Jake Bunce · May 8, 2016

RBenard said:
After fixing this, should I reinstall xenforo and point the importer to that install? The xf was a new install anyways or would that mess up the import somehow?

Reinstalling XF is not necessary. Just fix the error and re-run the Import.php file. The Import.php file is designed to be run multiple times against the same XF installation. Each time it will empty the relevant tables and re-import the content. That way you can keep redoing it until there are no more problems.

elsparkodiablo · May 10, 2016

Getting: DEBUG: RUNNING WITH FILE: /tmp/avatar_temp and IMAGE TYPE: 6 when encountering one user, any input? I've deleted that users avatar, changed it, still get same results.

elsparkodiablo · May 11, 2016

As an update: it would appear that the importer is choking on the avatars. We used to have avatars stored in the database and moved them to the filesystem as soon as it was an option in vB. Looking at the avatars it's choking on, it seems that vb has them all labeled as .gif while some are actually bitmaps.

In the code I'm seeing:

static $types = array(
1 => 'GIF',
2 => 'JPEG',
3 => 'PNG'
);

Is there a way to add bitmaps in there without breaking it?

Jake Bunce · May 11, 2016

elsparkodiablo said:
As an update: it would appear that the importer is choking on the avatars. We used to have avatars stored in the database and moved them to the filesystem as soon as it was an option in vB. Looking at the avatars it's choking on, it seems that vb has them all labeled as .gif while some are actually bitmaps.

In the code I'm seeing:

static $types = array(
1 => 'GIF',
2 => 'JPEG',
3 => 'PNG'
);

Is there a way to add bitmaps in there without breaking it?

I remember that allowing bitmaps caused errors with some of the image functions so they were excluded intentionally.

Here are some code ideas:

http://stackoverflow.com/questions/...e-from-bmp-with-mime-type-image-x-ms-bmp-in-p

Otherwise bitmaps just aren't supported and those users would need to re-upload.

elsparkodiablo · May 11, 2016

Ok, rather than attempting to recreate and change people's avatars, maybe it'd make more sense that if there's a problem with an avatar to skip over it? Might be something to consider as a bug fix - then just output a list of user ids who had their avatars skipped because they are bmps or other file types that are disallowed.

elsparkodiablo · May 12, 2016

Ok, so it seems like I should have created usergroups and upgrades prior to importing, then mapped them.

Here's the default array from the export.php code for usergroups

Code:

public static $groupMap = array(
                1 => 1,
                2 => 2,
                3 => 1,
                4 => 1,
                5 => 4,
                6 => 3,
                7 => 4,
                11 => 5,
                13 => 6,
                28 => 7,
                29 => 8,
                30 => 9,
                31 => 10,
                32 => 11,
                33 => 12,
                35 => 13,
                40 => 14,
                42 => 15,
                43 => 16,
                44 => 17,
                45 => 18,
            );

Here's how my usergroups are now mapped to the newly created, appropriate level user upgrades in the import script

Code:

public static $groupMap = array(
                1 => 1,  // unregistered 
                2 => 2, // default registered users
                3 => 1, // people who need to confirm their email
                4 => 1, // COPPA needing approval
                5 => 4, // Supermods in vb becoming regular mods
                6 => 3, // Administrators
                7 => 4, // Moderators
                22 => 5, // gold member
                27 => 6, // platinum member
                23 => 8, // Dealer level paid subscription
                29 => 7, // basic paid subscription
                30 => 1, // people who have had emails to them bounce
                25 => 10, // banned by moderators
                34 => 11, // pg level platinum member
                24 => 12, // knifemaker member
                28 => 13, // pirate member
                40 => 14,
                42 => 15,
                43 => 16,
                44 => 17,
                45 => 18,
            );

This should work correctly, yes?

Jake Bunce · May 12, 2016

elsparkodiablo said:
This should work correctly, yes?

Yep

elsparkodiablo · May 12, 2016

Thanks Jake.

elsparkodiablo · May 12, 2016

Ok, since I screwed up and didn't map the usergroups properly before exporting previously, we're doing a new export. What files do we need to delete to make sure that a complete export is done?

Jake Bunce · May 12, 2016

elsparkodiablo said:
Ok, since I screwed up and didn't map the usergroups properly before exporting previously, we're doing a new export. What files do we need to delete to make sure that a complete export is done?

The Export.php script will automatically overwrite the intermediate data files when you run it again.

BusOLuv · May 12, 2016

Jake Bunce said:
The Export.php script will automatically overwrite the intermediate data files when you run it again.

Hello,

I am working with elsparkodiablo and taking care of some PHP in the background. We are on the latest version of MariaDB on a CentOS 7.2 box with PHP 5.4.16.

In attempting to work through some issues with Export.php, I jockeyed some of the data in data and internal_data directories. As well as re moving the temp files in /tmp. We have no run an Import yet. Is there a way I can completely clear things out so that the Export starts from scratch?

BusOLuv · May 12, 2016

Also,

Several of the temp files are unable to execute the mysql command provided as well:

# cat xf_conversation_message.txt
mysql Ver 15.1 Distrib 5.5.47-MariaDB, for Linux (x86_64) using readline 5.1
Copyright (c) 2000, 2015, Oracle, MariaDB Corporation Ab and others.

Usage: mysql [OPTIONS] [database]

Default options are read from the following files in the given order:
/etc/mysql/my.cnf /etc/my.cnf ~/.my.cnf
The following groups are read: mysql client client-server client-mariadb
The following options may be given as the first argument:
--print-defaults Print the program argument list and exit.
--no-defaults Don't read default options from any option file.
--defaults-file=# Only read default options from the given file #.
--defaults-extra-file=# Read this file after the global files are read.

-?, --help Display this help and exit.
-I, --help Synonym for -?
--abort-source-on-error
Abort 'source filename' operations in case of errors
--auto-rehash Enable automatic rehashing. One doesn't need to use
'rehash' to get table and field completion, but startup
and reconnecting may take a longer time. Disable with
--disable-auto-rehash.
(Defaults to on; use --skip-auto-rehash to disable.)
-A, --no-auto-rehash
No automatic rehashing. One has to use 'rehash' to get
table and field completion. This gives a quicker start of
mysql and disables rehashing on reconnect.
--auto-vertical-output
Automatically switch to vertical output mode if the
result is wider than the terminal width.
-B, --batch Don't use history file. Disable interactive behavior.
(Enables --silent.)
--character-sets-dir=name
Directory for character set files.
--column-type-info Display column type information.
-c, --comments Preserve comments. Send comments to the server. The
default is --skip-comments (discard comments), enable
with --comments.
-C, --compress Use compression in server/client protocol.
-#, --debug[=#] This is a non-debug version. Catch this and exit.
--debug-check Check memory and open file usage at exit.
-T, --debug-info Print some debug info at exit.
-D, --database=name Database to use.
--default-character-set=name
Set the default character set.
--delimiter=name Delimiter to be used.
-e, --execute=name Execute command and quit. (Disables --force and history
file.)
-E, --vertical Print the output of a query (rows) vertically.
-f, --force Continue even if we get an SQL error. Sets
abort-source-on-error to 0
-G, --named-commands
Enable named commands. Named commands mean this program's
internal commands; see mysql> help . When enabled, the
named commands can be used from any line of the query,
otherwise only from the first line, before an enter.
Disable with --disable-named-commands. This option is
disabled by default.
-i, --ignore-spaces Ignore space after function names.
--init-command=name SQL Command to execute when connecting to MySQL server.
Will automatically be re-executed when reconnecting.
--local-infile Enable/disable LOAD DATA LOCAL INFILE.
-b, --no-beep Turn off beep on error.
-h, --host=name Connect to host.
-H, --html Produce HTML output.
-X, --xml Produce XML output.
--line-numbers Write line numbers for errors.
(Defaults to on; use --skip-line-numbers to disable.)
-L, --skip-line-numbers
Don't write line number for errors.
-n, --unbuffered Flush buffer after each query.
--column-names Write column names in results.
(Defaults to on; use --skip-column-names to disable.)
-N, --skip-column-names
Don't write column names in results.
--sigint-ignore Ignore SIGINT (CTRL-C).
-o, --one-database Ignore statements except those that occur while the
default database is the one named at the command line.
--pager[=name] Pager to use to display results. If you don't supply an
option, the default pager is taken from your ENV variable
PAGER. Valid pagers are less, more, cat [> filename],
etc. See interactive help (\h) also. This option does not
work in batch mode. Disable with --disable-pager. This
option is disabled by default.
-p, --password[=name]
Password to use when connecting to server. If password is
not given it's asked from the tty.
-P, --port=# Port number to use for connection or 0 for default to, in
order of preference, my.cnf, $MYSQL_TCP_PORT,
/etc/services, built-in default (3306).
--progress-reports Get progress reports for long running commands (like
ALTER TABLE)
(Defaults to on; use --skip-progress-reports to disable.)
--prompt=name Set the mysql prompt to this value.
--protocol=name The protocol to use for connection (tcp, socket, pipe,
memory).
-q, --quick Don't cache result, print it row by row. This may slow
down the server if the output is suspended. Doesn't use
history file.
-r, --raw Write fields without conversion. Used with --batch.
--reconnect Reconnect if the connection is lost. Disable with
--disable-reconnect. This option is enabled by default.
(Defaults to on; use --skip-reconnect to disable.)
-s, --silent Be more silent. Print results with a tab as separator,
each row on new line.
-S, --socket=name The socket file to use for connection.
--ssl Enable SSL for connection (automatically enabled with
other flags).
--ssl-ca=name CA file in PEM format (check OpenSSL docs, implies
--ssl).
--ssl-capath=name CA directory (check OpenSSL docs, implies --ssl).
--ssl-cert=name X509 cert in PEM format (implies --ssl).
--ssl-cipher=name SSL cipher to use (implies --ssl).
--ssl-key=name X509 key in PEM format (implies --ssl).
--ssl-verify-server-cert
Verify server's "Common Name" in its cert against
hostname used when connecting. This option is disabled by
default.
-t, --table Output in table format.
--tee=name Append everything into outfile. See interactive help (\h)
also. Does not work in batch mode. Disable with
--disable-tee. This option is disabled by default.
-u, --user=name User for login if not current user.
-U, --safe-updates Only allow UPDATE and DELETE that uses keys.
-U, --i-am-a-dummy Synonym for option --safe-updates, -U.
-v, --verbose Write more. (-v -v -v gives the table output format).
-V, --version Output version information and exit.
-w, --wait Wait and retry if connection is down.
--connect-timeout=# Number of seconds before connection timeout.
--max-allowed-packet=#
The maximum packet length to send to or receive from
server.
--net-buffer-length=#
The buffer size for TCP/IP and socket communication.
--select-limit=# Automatic limit for SELECT when using --safe-updates.
--max-join-size=# Automatic limit for rows in a join when using
--safe-updates.
--secure-auth Refuse client connecting to server if it uses old
(pre-4.1.1) protocol.
--server-arg=name Send embedded server this as a parameter.
--show-warnings Show warnings after every statement.
--plugin-dir=name Directory for client-side plugins.
--default-auth=name Default authentication client-side plugin to use.
--binary-mode By default, ASCII '\0' is disallowed and '\r\n' is
translated to '\n'. This switch turns off both features,
and also turns off parsing of all clientcommands except
\C and DELIMITER, in non-interactive mode (for input
piped to mysql or loaded using the 'source' command).
This is necessary when processing output from mysqlbinlog
that may contain blobs.

Variables (--variable-name=value)
and boolean options {FALSE|TRUE} Value (after reading options)
--------------------------------- ----------------------------------------
abort-source-on-error FALSE
auto-rehash TRUE
auto-vertical-output FALSE
character-sets-dir (No default value)
column-type-info FALSE
comments FALSE
compress FALSE
debug-check FALSE
debug-info FALSE
database (No default value)
default-character-set auto
delimiter ;
vertical FALSE
force FALSE
named-commands FALSE
ignore-spaces FALSE
init-command (No default value)
local-infile FALSE
no-beep FALSE
host 199.231.142.11
html FALSE
xml FALSE
line-numbers TRUE
unbuffered FALSE
column-names FALSE
sigint-ignore FALSE
port 0
progress-reports FALSE
prompt \N [\d]>
quick TRUE
raw FALSE
reconnect FALSE
socket (No default value)
ssl FALSE
ssl-ca (No default value)
ssl-capath (No default value)
ssl-cert (No default value)
ssl-cipher (No default value)
ssl-key (No default value)
ssl-verify-server-cert FALSE
table FALSE
user (No default value)
safe-updates FALSE
i-am-a-dummy FALSE
connect-timeout 0
max-allowed-packet 16777216
net-buffer-length 16384
select-limit 1000
max-join-size 1000000
secure-auth FALSE
show-warnings FALSE
plugin-dir (No default value)
default-auth (No default value)
binary-mode FALSE

Jake Bunce · May 13, 2016

BusOLuv said:
Hello,

I am working with elsparkodiablo and taking care of some PHP in the background. We are on the latest version of MariaDB on a CentOS 7.2 box with PHP 5.4.16.

In attempting to work through some issues with Export.php, I jockeyed some of the data in data and internal_data directories. As well as re moving the temp files in /tmp. We have no run an Import yet. Is there a way I can completely clear things out so that the Export starts from scratch?

Empty the importData directory specified in this setting in the Export.php file:

Code:

	// PATH TO STORE INTERMEDIATE DATA
	public static $dataDir = '/home/sites/yoursite/public_html/community/importData/';

You could also delete imported attachments and avatars, but those are designed to be run incrementally. Empty XF's /data and /internal_data/attachments directories, then truncate XF's xf_attachment table.

Jake Bunce · May 13, 2016

BusOLuv said:
Also,

Several of the temp files are unable to execute the mysql command provided as well:

# cat xf_conversation_message.txt
mysql Ver 15.1 Distrib 5.5.47-MariaDB, for Linux (x86_64) using readline 5.1
Copyright (c) 2000, 2015, Oracle, MariaDB Corporation Ab and others.

Usage: mysql [OPTIONS] [database]

...

It seems that xf_conversation_message.txt contains an error from the mysql command that is supposed to write to it. One of your MySQL settings near the top of the Export.php file is probably wrong and is creating an invalid command. Search for the file name and you will find this code:

Code:

		$start = microtime(true);
		echo "          conversation message";
		exec('mysql -h' . $this->slaveDbHost . ' ' . $this->sourceDb . ' ' . $this->sourceDBuser . ' ' . $this->sourceDBpassword . ' -N -q -e "SET NAMES binary;SET @ip_id = ' . $existingIpCount . ';SELECT SQL_NO_CACHE pmtext.pmtextid AS message_id, pmtext.pmtextid AS conversation_id, pmtext.dateline AS message_date, pmtext.fromuserid AS user_id, pmtext.fromusername AS username, pmtext.message, 0 AS attach_count FROM ' . self::$tablePrefix . 'pmtext as pmtext ORDER BY pmtext.pmtextid"' . $this->_getReplaceBBCodeCommand() . ' ' . self::$extraOutCommand . ' > ' . self::$dataDir . 'xf_conversation_message.txt');
		echo " (" . number_format(intval(shell_exec('wc -l ' . self::$dataDir . 'xf_conversation_message.txt'))) . ' records in ' .  number_format(microtime(true) - $start, 2) . "s)...\r\n";

The mysql command is a concatenation which includes several settings from near the top of the file. Check these settings in particular:

Code:

	//Enter your vBulletin Forum details here
	protected $sourceDb = 'DBNAME';
	protected $slaveDbHost = 'localhost';	
	protected $sourceDBuser = '-uUSER';	
	protected $sourceDBpassword = '-pPASS';

	...

	// USE THIS SETTING TO SPECIFY AN EXTRA PIPE FOR ALL OUTPUT RECORDS
	// EXAMPLE IS TO PIPE OUTPUT THROUGH iconv IF CHARACTER ENCODING CHANGES ARE NEEDED
	// '| iconv -f LATIN1 -t UTF8'
	public static $extraOutCommand = '| iconv -f LATIN1 -t UTF8';

BusOLuv · May 13, 2016

Thanks, Jake, I'll give this a try on the re-run. For the mysql command, I definitely saw the standard mysql commands with the -e flag and the > redirect. Our connection values are correct, so let me do some digging and I will report on the mysql. I'll echo the full command it is trying to run and try it separately myself.

elsparkodiablo · Jun 2, 2016

@Slavik we're having issues with our export / import on our test server. What's your timeline of availability for paid work?

Slavik · Jun 2, 2016

elsparkodiablo said:
@Slavik we're having issues with our export / import on our test server. What's your timeline of availability for paid work?

How large is your forum?

elsparkodiablo · Jun 2, 2016

21gb database. 1.2 mil threads, 15mil posts, 300k users.

Slavik · Jun 2, 2016

elsparkodiablo said:
21gb database. 1.2 mil threads, 15mil posts, 300k users.

I can do next week mid week?

elsparkodiablo · Jun 2, 2016

Looks like my guy had to fix some issues with the code so that the latest MariaDB would work. He'll post more soon.

vBulletin Big Board Importer [vBulletin 3 + vBulletin 4] [Paid] 1.5.0

Well-known member

Active member

Active member

Well-known member

Active member

Active member

Well-known member

Active member

Active member

Well-known member

New member

New member

Well-known member

Well-known member

New member

Active member

XenForo moderator

Active member

XenForo moderator

Active member

Similar threads

We value your privacy