Resource icon
vBulletin Big Board Importer [vBulletin 3 + vBulletin 4] [Paid] 1.5.0
- Thread starter Slavik
- Start date
Bossman
Member
Indeed, the memory limit was the issue for the export. We upped it to 512MB and the export works ok now, much obliged for pointing me in the right direction.
As a sidenote, I think it would be ok to have the PHP errors displayed by default in a future version of the script, for the sake of expediency when dealing with issues like this one. We used
PHP:
ini_set('display_errors',1);
ini_set('display_startup_errors',1);
error_reporting(-1);at the top for good measure
Jake Bunce
Well-known member
FYI, I sent my latest copy of this importer to Slavik. Off the top of my head:
1) Fixed bug in attachment system that affects vB4 file system attachments.
2) Fixed bug in attachment system regarding empty batches.
3) Removed hard-coded DP stuff for link-forums.
4) Added option to resurrect sent PMs from vB. Currently they are resurrected, and the option allows you to disable that if you want.
5) Added setting to specify the usergroupid of the banned group in vB to allow for proper importing of banned users.
1) Fixed bug in attachment system that affects vB4 file system attachments.
2) Fixed bug in attachment system regarding empty batches.
3) Removed hard-coded DP stuff for link-forums.
4) Added option to resurrect sent PMs from vB. Currently they are resurrected, and the option allows you to disable that if you want.
5) Added setting to specify the usergroupid of the banned group in vB to allow for proper importing of banned users.
Jake Bunce
Well-known member
I did not add dbtech likes. It would be a simple addition though. Just duplicate the block of code for reputation and change the table/field names appropriately.
Hornstar
Well-known member
I tried doing that but I was not sure how to get both the likes and thanks from dbtech imported. I posted the schema a couple pages back, are you able to help?
Slavik updated vBulletin Big Board Importer [vBulletin 3 + vBulletin 4] with a new update entry:
1.2.0
Read the rest of this update entry...
1.2.0
Read the rest of this update entry...
M
Member 10517
Guest
Is it possible to import the vB4 CMS articles?
Hi all.
I have managed to conduct the import process successfully, but I have one last issue: some usernames are not encoded correctly like the one shown below.
Any idea about that?
By the way, the import process took around 12 hours. Is that OK? My forum stats:
Thank you in advance.
I have managed to conduct the import process successfully, but I have one last issue: some usernames are not encoded correctly like the one shown below.
Any idea about that?
By the way, the import process took around 12 hours. Is that OK? My forum stats:
Thank you in advance.
Jake Bunce
Well-known member
vB4 CMS articles are actually stored as threads in a hidden forum. So they should come across as threads.
Jake Bunce
Well-known member
12 hours is much too long for 4 million posts, but if you have lots of attachments then that could explain it. This importer does attachments incrementally such that subsequent runs (after having already imported attachments) should be much faster.
Jake Bunce
Well-known member
Your usernames are encoded as entities which is a problem. Ideally you need a program to replace those entities with utf-8 equivalents. I don't know of such a program, but it could be called using $extraOutCommand in the Export script.
Bossman
Member
While we're on the subject of charsets, we've come across an issue when doing test imports for one of the forums we want to move to XF. Namely, the special characters of our language.
By default, they look like this:
The problem is that the binary import puts them back in MySQL like this:
Which means that if we do a standard search and replace, there's no way for us to differentiate between the lowercase and uppercase special characters
Is there really no way we can do the export / import using utf8 as charset?
By default, they look like this:
The problem is that the binary import puts them back in MySQL like this:
Which means that if we do a standard search and replace, there's no way for us to differentiate between the lowercase and uppercase special characters
Is there really no way we can do the export / import using utf8 as charset?
imthebest
Well-known member
Could you elaborate a bit more about what does PM resurrection means?
Bossman
Member
Jake Bunce
Well-known member
While we're on the subject of charsets, we've come across an issue when doing test imports for one of the forums we want to move to XF. Namely, the special characters of our language.
By default, they look like this:
The problem is that the binary import puts them back in MySQL like this:
Which means that if we do a standard search and replace, there's no way for us to differentiate between the lowercase and uppercase special characters
Is there really no way we can do the export / import using utf8 as charset?
What is the source character encoding? Use this setting in the Export script to specify that:
Code:
// USE THIS SETTING TO SPECIFY AN EXTRA PIPE FOR ALL OUTPUT RECORDS
// EXAMPLE IS TO PIPE OUTPUT THROUGH iconv IF CHARACTER ENCODING CHANGES ARE NEEDED
// '| iconv -f LATIN1 -t UTF8'
public static $extraOutCommand = '| iconv -f LATIN1 -t UTF8';Also heed the warning when you run the Export script:
Bossman
Member
We have utf8 encoding (charset and collation), so ultimately we had to use
in the export script to get the output right
Nevertheless, at first glance it seems that everything's in order, encoding wise, so at least that's one more problem solved.
The only section of the Export which keeps failing though is the attachments one.
Although we have over 20.000 attachments in the filesystem, the Export scripts tells us that
with a lot of
up until that point.
The question is: is this generated by a "bad" attachment, or are we doing something wrong in the config file? Because there's no way the importer can go through 20.000+ attachments (with 12GB of space used) in only 15s. Even with a nice server like ours
P.S: when trying to export the Attachments, the script leaves behind an attachment_tmp file in the importData dir and it does process 143 files out of the 20k+ we have. Also, the avatar export seems to be working as expected, so I'm not 100% sure it's a path issue.
PHP:
public static $extraOutCommand = '| iconv -f UTF8 -t UTF8';in the export script to get the output right
Nevertheless, at first glance it seems that everything's in order, encoding wise, so at least that's one more problem solved.
The only section of the Export which keeps failing though is the attachments one.
Although we have over 20.000 attachments in the filesystem, the Export scripts tells us that
Code:
updating attachment view count
Total time for attachments: 15.13swith a lot of
Code:
!!MISSING DATA!!29224 !!MISSING DATA!!29225 !!MISSING DATA!!29226 !!MISSING DATA!!29227 !!MISSING DATA!!29228 !!MISSING DATA!!29229 !!MISSING DATA!!29230 !!MISSING DATA!!29231 !!MISSING DATA!!29232 !!MISSING DATA!!29233 !!MISSING DATA!!29234 !!MISSING DATA!!29235 !!MISSING DATA!!29236 !!MISSING DATA!!29237 !!MISSING DATA!!29238 !!MISSING DATA!!29249 !!MISSING DATA!!29250 !!MISSING DATA!!29251 !!MISSING DATA!!29252 !!MISSING DATA!!29253 !!MISSING DATA!!29254 !!MISSING DATA!!29255 !!MISSING DATA!!29256 !!MISSING DATA!!29257 !!MISSING DATA!!29258 !!MISSING DATA!!29259 !!MISSING DATA!!29260 !!MISSING DATA!!29262 !!MISSING DATA!!29263 !!MISSING DATA!!29264 !!MISSING DATA!!29265 !!MISSINGup until that point.
The question is: is this generated by a "bad" attachment, or are we doing something wrong in the config file? Because there's no way the importer can go through 20.000+ attachments (with 12GB of space used) in only 15s. Even with a nice server like ours
P.S: when trying to export the Attachments, the script leaves behind an attachment_tmp file in the importData dir and it does process 143 files out of the 20k+ we have. Also, the avatar export seems to be working as expected, so I'm not 100% sure it's a path issue.
Last edited:
Jake Bunce
Well-known member
We have utf8 encoding (charset and collation), so ultimately we had to use
PHP:public static $extraOutCommand = '| iconv -f UTF8 -t UTF8';
in the export script to get the output right
Nevertheless, at first glance it seems that everything's in order, encoding wise, so at least that's one more problem solved.
The only section of the Export which keeps failing though is the attachments one.
Although we have over 20.000 attachments in the filesystem, the Export scripts tells us that
Code:updating attachment view count Total time for attachments: 15.13s
with a lot of
Code:!!MISSING DATA!!29224 !!MISSING DATA!!29225 !!MISSING DATA!!29226 !!MISSING DATA!!29227 !!MISSING DATA!!29228 !!MISSING DATA!!29229 !!MISSING DATA!!29230 !!MISSING DATA!!29231 !!MISSING DATA!!29232 !!MISSING DATA!!29233 !!MISSING DATA!!29234 !!MISSING DATA!!29235 !!MISSING DATA!!29236 !!MISSING DATA!!29237 !!MISSING DATA!!29238 !!MISSING DATA!!29249 !!MISSING DATA!!29250 !!MISSING DATA!!29251 !!MISSING DATA!!29252 !!MISSING DATA!!29253 !!MISSING DATA!!29254 !!MISSING DATA!!29255 !!MISSING DATA!!29256 !!MISSING DATA!!29257 !!MISSING DATA!!29258 !!MISSING DATA!!29259 !!MISSING DATA!!29260 !!MISSING DATA!!29262 !!MISSING DATA!!29263 !!MISSING DATA!!29264 !!MISSING DATA!!29265 !!MISSING
up until that point.
The question is: is this generated by a "bad" attachment, or are we doing something wrong in the config file? Because there's no way the importer can go through 20.000+ attachments (with 12GB of space used) in only 15s. Even with a nice server like ours
P.S: when trying to export the Attachments, the script leaves behind an attachment_tmp file in the importData dir and it does process 143 files out of the 20k+ we have. Also, the avatar export seems to be working as expected, so I'm not 100% sure it's a path issue.
Attachments are incremental meaning it only imports new attachments since the last run. If you want to run them all again then empty xf_attachment and xf_attachment_data, and delete the attach directories inside of data and internal_data.
"!!MISSING DATA!!" means exactly that. Confirm the correct attachment path at the top of the export script.
Bossman
Member
I did, the path is correct. And the script keeps processing only 112 attachments out of the 22k available every time
I tried moving the vb attach folder inside and then outside the xf docroot.
I tried with and without the trailing slash for the attach path inside the export script.
I defined all the attach types in public function attachments.
The end result is always the same. 112 files get processed, the rest are "missing data"
What's interesting though is that these 112 attachments that do get processed are all jpg. Nothing else. No gifs, no pngs, no zips, nothing. Just jpg.
Last edited:
Hornstar
Well-known member
I have the same problem.I did, the path is correct. And the script keeps processing only 112 attachments out of the 22k available every time
I tried moving the vb attach folder inside and then outside the xf docroot.
I tried with and without the trailing slash for the attach path inside the export script.
I defined all the attach types in public function attachments.
The end result is always the same. 112 files get processed, the rest are "missing data"
What's interesting though is that these 112 attachments that do get processed are all jpg. Nothing else. No gifs, no pngs, no zips, nothing. Just jpg.
Similar threads
- Question
- Question
