Resource icon
vBulletin Big Board Importer [vBulletin 3 + vBulletin 4] [Paid] 1.5.0
- Thread starter Slavik
- Start date
Hi. I've been attempting to get this working however I get the following error:
It's the first time running but it's saying the files are missing?
Code:
exporting posts...
postssh: aBULW: command not found
sh: /home/sites/mixspace/public_html/importData/xf_post.txt: No such file or directory
rm: cannot remove `/home/sites/mixspace/public_html/importData/xf_post_*': No such file or directory
wc: /home/sites/mixspace/public_html/importData/xf_post.txt: No such file or directory
(0 records in 0.01s)...
split: cannot open `/home/sites/mixspace/public_html/importData/xf_post.txt' for reading: No such file or directory
An exception occurred: unlink(/home/sites/mixspace/public_html/importData/xf_post.txt): No such file or directory in /home/mixspace/public_html/Export.php on line 821
#0 [internal function]: XenForo_Application::handlePhpError(2, 'unlink(/home/si...', '/home/mixspace/...', 821, Array)
#1 /home/mixspace/public_html/Export.php(821): unlink('/home/sites/mix...')
#2 /home/mixspace/public_html/Export.php(1900): DigitalPointExporter->posts()
#3 {main}It's the first time running but it's saying the files are missing?
Adam Bloch
Member
@NixFifty:
[While you're waiting for more competent replies!] I think it's normal to have 'No such file or directory' warnings on first run; I think the exporter tries to remove any intermediate files written in previous-runs when a selection is made - although I'm clearly no expert in this!
Again, with the caution that I'm no expert (I seem to be able only to export MySQL quick-help with the script!), I'd suggest that perhaps the 'postssh: aBULW: command not found' line is more of a problem - exception handling might be an issue too; I might be getting the wrong end of the stick, but it appears to be dying because previously written intermediate files aren't present - which seems strange.
Again, please take all the above with a pinch of salt - I'm a MySQL/PHP fiddler, some distance short of being fluent, and a long way from being a guru! - Maybe it's a useful perspective, maybe it's nonsense :/
@Slavik:
I built two new clean CentOS 6.5 VMs, the first - CentOS 6.5, updates to out-of-the-box packages complete, latest stable LAMP from EPEL/Remi repos - As previously, I just get quick-help dumped to file.
The second was as above, but with Percona55 (from Percona's yum repo) in place of standard MySQL...
*cries*
I must be missing something crucial which is pretty low on the food chain, but as per my caution to NixFifty above, I'm not completely green, but I'm by no means a guru - I'd really appreciate any further advice you can offer.
Thanks again,
A
[While you're waiting for more competent replies!] I think it's normal to have 'No such file or directory' warnings on first run; I think the exporter tries to remove any intermediate files written in previous-runs when a selection is made - although I'm clearly no expert in this!
Again, with the caution that I'm no expert (I seem to be able only to export MySQL quick-help with the script!), I'd suggest that perhaps the 'postssh: aBULW: command not found' line is more of a problem - exception handling might be an issue too; I might be getting the wrong end of the stick, but it appears to be dying because previously written intermediate files aren't present - which seems strange.
Again, please take all the above with a pinch of salt - I'm a MySQL/PHP fiddler, some distance short of being fluent, and a long way from being a guru! - Maybe it's a useful perspective, maybe it's nonsense :/
@Slavik:
I built two new clean CentOS 6.5 VMs, the first - CentOS 6.5, updates to out-of-the-box packages complete, latest stable LAMP from EPEL/Remi repos - As previously, I just get quick-help dumped to file.
The second was as above, but with Percona55 (from Percona's yum repo) in place of standard MySQL...
*cries*
I must be missing something crucial which is pretty low on the food chain, but as per my caution to NixFifty above, I'm not completely green, but I'm by no means a guru - I'd really appreciate any further advice you can offer.
Thanks again,
A
Last edited:
Bossman
Member
Solution for people trying to import attachments and getting the dreaded "MISSING DATA!!!" error, even though everything seems to be in order*
* meaning that the folder path is ok and has the / at the end in the export.php file + the user and group permissions are set accordingly
There are two folder structures that vBulletin can use when storing attachments in the filesystem:
1. The default one that looks like
So if the userid is 1234, the attachment path for a file will be /public_html/attachment_folder/1/2/3/4/1.attach.
This is what both the official importer and the Big Board Importer (BBI) look for by default.
2. A legacy one, from times long forgotten, where the user id isn't broken up into digits sub-folders and looks like
So in the real world it would translate to /public_html/attachment_folder/1234/1.attach
Since the BBI doesn't recognize the #2 folder structure by default, you'll get the MISSING DATA!!! error.
Thankfully, @Mike provided us with a solution for the default importer which we then applied to the BBI:
1. Open the export.php file
2. Go to line 1307 to find the following code block:
3. Replace said code block with:
4. Profit
For those who want to understand what we did, we changed the
code (the one that tells the importer to "break apart" the user id when creating the attachment path) with
thus telling the importer to read the user id whole.
We then added a
at the front of the user id to make sure that the path would be generated correctly (yes, including the period, that's what's called a concatenation operator. You can read more about it here).
Otherwise, the generated path would have been
/public_html/attachment_folder1234/1.attach (notice how the userid and attachment path are fused).
-----------------------------------------------------------------------------------
If you're having the same attachments issues when using the official importer:
1. Open library/XenForo/Importer/vBulletin4x.php
2. Go to line 92
3. Change
to
4. Profit
Once again, all credit goes to @Mike for pointing us in the right direction regarding the fix for this problem.
* meaning that the folder path is ok and has the / at the end in the export.php file + the user and group permissions are set accordingly
There are two folder structures that vBulletin can use when storing attachments in the filesystem:
1. The default one that looks like
PHP:
/attachment_folder/u/s/e/r/i/d/{file-id}.attachSo if the userid is 1234, the attachment path for a file will be /public_html/attachment_folder/1/2/3/4/1.attach.
This is what both the official importer and the Big Board Importer (BBI) look for by default.
2. A legacy one, from times long forgotten, where the user id isn't broken up into digits sub-folders and looks like
PHP:
/{path-you-set-in-the-vbulletin-options}/userid/{file-id}.attachSo in the real world it would translate to /public_html/attachment_folder/1234/1.attach
Since the BBI doesn't recognize the #2 folder structure by default, you'll get the MISSING DATA!!! error.
Thankfully, @Mike provided us with a solution for the default importer which we then applied to the BBI:
1. Open the export.php file
2. Go to line 1307 to find the following code block:
PHP:
if (self::$attachFile AND file_exists(self::$attachFile . implode('/', preg_split('//', $attachment['user_id'], -1, PREG_SPLIT_NO_EMPTY)) . '/' . strval($attachment['filesysid']) . '.attach'))
{
$attachment['file'] = file_get_contents(self::$attachFile . implode('/', preg_split('//', $attachment['user_id'], -1, PREG_SPLIT_NO_EMPTY)) . '/' . strval($attachment['filesysid']) . '.attach');
}
else3. Replace said code block with:
PHP:
if (self::$attachFile AND file_exists(self::$attachFile . '/' . $attachment['user_id'] . '/' . strval($attachment['filesysid']) . '.attach'))
{
$attachment['file'] = file_get_contents(self::$attachFile . '/' . $attachment['user_id'] . '/' . strval($attachment['filesysid']) . '.attach');
}
else4. Profit
For those who want to understand what we did, we changed the
PHP:
implode('/', preg_split('//', $attachment['user_id'], -1, PREG_SPLIT_NO_EMPTY))code (the one that tells the importer to "break apart" the user id when creating the attachment path) with
PHP:
$attachment['user_id']thus telling the importer to read the user id whole.
We then added a
PHP:
'/'.at the front of the user id to make sure that the path would be generated correctly (yes, including the period, that's what's called a concatenation operator. You can read more about it here).
Otherwise, the generated path would have been
/public_html/attachment_folder1234/1.attach (notice how the userid and attachment path are fused).
-----------------------------------------------------------------------------------
If you're having the same attachments issues when using the official importer:
1. Open library/XenForo/Importer/vBulletin4x.php
2. Go to line 92
3. Change
PHP:
implode('/', str_split($attachment['filedata_userid']))to
PHP:
$attachment['filedata_userid']4. Profit
Once again, all credit goes to @Mike for pointing us in the right direction regarding the fix for this problem.
@NixFifty have you got all the requirements installed as listed in the resource description?
@Adam Bloch I will fire up a VM when I can to test, and note down what I do.
@Bossman Most people experiencing the missing data error will be due to the attachment being deleted, but not the reference to it in the database, however, thanks for posting the fix in the exceptionally rare case someone else was using this particular beta version attachment structure of vB.
@Adam Bloch I will fire up a VM when I can to test, and note down what I do.
@Bossman Most people experiencing the missing data error will be due to the attachment being deleted, but not the reference to it in the database, however, thanks for posting the fix in the exceptionally rare case someone else was using this particular beta version attachment structure of vB.
To my knowledge, yes.
Jake Bunce
Well-known member
^what Slavik said^
That particular attachment scheme is from pre-release versions of vB 3.0 (ie very old and very rare). This is actually explained in the vB Admin CP when you enable debug mode to access hidden settings:
As you discovered, this importer doesn't account for this possibility. It is very unlikely that anyone else will ever need your fix, but it's legitimate none-the-less.
Adam Bloch
Member
@Slavik
Woah there. Delay that. Don't sweat over a VM on my account just yet.
The penny has just dropped...
First of all, an admission... I'm not "A"; in actual fact, I'm R(ob) [windowtitle on terminal screenies might've given that one up, anyway] - Adam's right-hand and down-a-bit man; Adam owns the site, buys the stuff, and tinkers with things. I'm the SSH urchin. It's normally easier just to use his credentials and sign A than explain it all.
He'd dropped the two script files into our 'live' XenForo installation before I was roped in, and since then, I've been working with those. Scratching my head really hard about this, I thought I'd see if I could get hold of the original package and start from the beginning - just in case.
It's at that point I noticed that in the section of your script where parameters are entered, the database user & password variables need to have the command syntax inside their values.
I feel 10% shame, and 90% frustration (which I'll be airing to Adam at my first opportunity!).
I'm about to put all the details into the Exporter from scratch, and run again - I'm sure you'll agree it's not unreasonable to expect I'll get over the first hurdle now!
My sincerest apologies for posting what's turned out to be a red-herring issue; although I can't help thinking that sticking the -u/-p syntax in the variables' values is pretty dirty
maybe if you update the script in future, you might append the parameters to pre-formatted syntax.
Once again, my apologies; and rest assured Adam will be made to feel the pain of all our wasted time!
R
EDIT:
*tears of joy*
...I'm trying my level best to see the funny side.
Apologies and thanks again,
R
(I guess you might want to delete my posts from the thread)
Woah there. Delay that. Don't sweat over a VM on my account just yet.
The penny has just dropped...
First of all, an admission... I'm not "A"; in actual fact, I'm R(ob) [windowtitle on terminal screenies might've given that one up, anyway] - Adam's right-hand and down-a-bit man; Adam owns the site, buys the stuff, and tinkers with things. I'm the SSH urchin. It's normally easier just to use his credentials and sign A than explain it all.
He'd dropped the two script files into our 'live' XenForo installation before I was roped in, and since then, I've been working with those. Scratching my head really hard about this, I thought I'd see if I could get hold of the original package and start from the beginning - just in case.
It's at that point I noticed that in the section of your script where parameters are entered, the database user & password variables need to have the command syntax inside their values.
I feel 10% shame, and 90% frustration (which I'll be airing to Adam at my first opportunity!).
I'm about to put all the details into the Exporter from scratch, and run again - I'm sure you'll agree it's not unreasonable to expect I'll get over the first hurdle now!
My sincerest apologies for posting what's turned out to be a red-herring issue; although I can't help thinking that sticking the -u/-p syntax in the variables' values is pretty dirty
maybe if you update the script in future, you might append the parameters to pre-formatted syntax.Once again, my apologies; and rest assured Adam will be made to feel the pain of all our wasted time!
R
EDIT:
*tears of joy*
...I'm trying my level best to see the funny side.
Apologies and thanks again,
R
(I guess you might want to delete my posts from the thread)
Last edited:
We were getting this error when trying to run the Export script. Turned out our DB password had an ampersand in it thus it was causing the error that the command wasn't found. Changing the password to an alpha-numeric string solved this for us. Moral of the story, make sure you don't have any ampersands in your password.
I get this when trying to import avatars:
Error Info
ErrorException: file_put_contents(importData/avatar_temp): failed to open stream: Is a directory - Export.php:1528
Generated By: Unknown Account, 4 minutes ago
Stack Trace
#0 [internal function]: XenForo_Application::handlePhpError(2, 'file_put_conten...', '/home/mixspace/...', 1528, Array)
#1 /home/mixspace/public_html/Export.php(1528): file_put_contents('importData/avat...', '??????JFIF?????...')
#2 /home/mixspace/public_html/Export.php(1894): DigitalPointExporter->avatars()
#3 {main}
Error Info
ErrorException: file_put_contents(importData/avatar_temp): failed to open stream: Is a directory - Export.php:1528
Generated By: Unknown Account, 4 minutes ago
Stack Trace
#0 [internal function]: XenForo_Application::handlePhpError(2, 'file_put_conten...', '/home/mixspace/...', 1528, Array)
#1 /home/mixspace/public_html/Export.php(1528): file_put_contents('importData/avat...', '??????JFIF?????...')
#2 /home/mixspace/public_html/Export.php(1894): DigitalPointExporter->avatars()
#3 {main}
Adam Bloch
Member
Yeah, same error if they're removed.
Adam Bloch
Member
@NixFifty
Again, I'm sorry if my suggestions are clumsy/dumbass - I'm working on the 'another pair of eyes' principle - if you'd rather I shut up, just say
Just had a look at line (~)1528 in the exporter, and there's something about throwing errors for non-image files. I notice there's a JFIF???? bit in your error - do you have users with JFIF files as avatars? Maybe you could temporarily move them out of the filesystem and see if it gets past the error?
R
Again, I'm sorry if my suggestions are clumsy/dumbass - I'm working on the 'another pair of eyes' principle - if you'd rather I shut up, just say
Just had a look at line (~)1528 in the exporter, and there's something about throwing errors for non-image files. I notice there's a JFIF???? bit in your error - do you have users with JFIF files as avatars? Maybe you could temporarily move them out of the filesystem and see if it gets past the error?
R
Adam Bloch
Member
I'm having fun trying to import now...
I get various warnings about unexpected or missing values in certain columns/rows, but I'm guessing they're not critical as the thread & post counts look right; but when it comes to users I'm getting an error that the user authentication table can't be repaired - this is something to do with InnoDB, right?
I'm quite lost from the outset on this one - after the import I can log-in to the XF Admin CP with my ex-VB credentials, but most of the controls are missing, and at the front end I get 'You do not have permission to view this page or perform this action.' wherever I try to go.
I'll start reading this thread from the front, but if anyone's got a suggestion while I am, I'd be most grateful.
Thanks,
R
EDIT: Thinking out loud (I'm sorry to do it in the thread!)... Should I be switching tables to MyISAM before I run the import?
R
I get various warnings about unexpected or missing values in certain columns/rows, but I'm guessing they're not critical as the thread & post counts look right; but when it comes to users I'm getting an error that the user authentication table can't be repaired - this is something to do with InnoDB, right?
I'm quite lost from the outset on this one - after the import I can log-in to the XF Admin CP with my ex-VB credentials, but most of the controls are missing, and at the front end I get 'You do not have permission to view this page or perform this action.' wherever I try to go.
I'll start reading this thread from the front, but if anyone's got a suggestion while I am, I'd be most grateful.
Thanks,
R
EDIT: Thinking out loud (I'm sorry to do it in the thread!)... Should I be switching tables to MyISAM before I run the import?
R
Last edited:
Jake Bunce
Well-known member
Some harmless errors during the import can be normal as the data is not "strict". For example, some integer values are inserted as nulls. It still imports but it will throw an error.
The xf_user_authenticate table should be InnoDB. Please confirm this.
The lack of permissions for your imported admin user on the frontend and in the Admin CP is probably just admin and group permissions needing to be setup. First make yourself a super admin to ensure that you have full access to the Admin CP:
http://xenforo.com/help/config-php-options/
That will give you access to the groups where you can setup your frontend permissions:
Admin CP -> Users -> List User Groups
Jake Bunce
Well-known member
Delete "importData/avatar_temp", then make sure "importData" is fully writable (chmod 777). Then try running the user step again.
Adam Bloch
Member
@Jake Bunce
Thanks for your reply, Jake.
xf_user_authenticate is indeed InnoDB - in case it's useful to know, I've been creating an empty database at the command line, installing XenForo, then running the script; so I guess all the tables are as intended.
I didn't think to add my ex-VB user ID into the config (doh!) - once that was done all the Admin CP came back
I've since been trying to rebuild search index and caches; with a few hiccups so far.
I started at the top of the page and figured I'd just rebuild everything from top to bottom. Search index rebuild got through about 1.1M posts and then ground to a halt - maybe I should reduce the chunk size or introduce a delay? (Although there's no load on the server beside itself - I'm testing on a VM, and while it doesn't have huge grunt, it's got an Ivy Bridge CPU core and 4GB RAM to itself).
Most of everything else seemed to run OK, although one or two were suspiciously quick (blink and you'll miss it).
However, when I came to rebuild conversations and rebuild daily statistics, I got the following error (same for both):
After that the front end wouldn't respond at all until I restarted Apache.
I'm getting much closer though... Before I log in there are no forums visible (I guess this is an issue of permissions?), but I can log-in at the front end, and when I do forums, threads & posts seem to be where they should be.
Thanks again for your help (and any more pointers you can give),
R
Thanks for your reply, Jake.
xf_user_authenticate is indeed InnoDB - in case it's useful to know, I've been creating an empty database at the command line, installing XenForo, then running the script; so I guess all the tables are as intended.
I didn't think to add my ex-VB user ID into the config (doh!) - once that was done all the Admin CP came back
I've since been trying to rebuild search index and caches; with a few hiccups so far.
I started at the top of the page and figured I'd just rebuild everything from top to bottom. Search index rebuild got through about 1.1M posts and then ground to a halt - maybe I should reduce the chunk size or introduce a delay? (Although there's no load on the server beside itself - I'm testing on a VM, and while it doesn't have huge grunt, it's got an Ivy Bridge CPU core and 4GB RAM to itself).
Most of everything else seemed to run OK, although one or two were suspiciously quick (blink and you'll miss it).
However, when I came to rebuild conversations and rebuild daily statistics, I got the following error (same for both):
Code:
Server Error
Mysqli statement execute error : Got error 28 from storage engine
Zend_Db_Statement_Mysqli->_execute() in Zend/Db/Statement.php at line 297
Zend_Db_Statement->execute() in Zend/Db/Adapter/Abstract.php at line 479
Zend_Db_Adapter_Abstract->query() in XenForo/Model.php at line 219
XenForo_Model->fetchAllKeyed() in XenForo/Model/Conversation.php at line 116
XenForo_Model_Conversation->getConversationRecipients() in XenForo/DataWriter/ConversationMaster.php at line 417
XenForo_DataWriter_ConversationMaster->rebuildRecipients() in XenForo/Deferred/Conversation.php at line 28
XenForo_Deferred_Conversation->execute() in XenForo/Model/Deferred.php at line 252
XenForo_Model_Deferred->runDeferred() in XenForo/Model/Deferred.php at line 378
XenForo_Model_Deferred->_runInternal() in XenForo/Model/Deferred.php at line 331
XenForo_Model_Deferred->run() in XenForo/ControllerAdmin/Tools.php at line 147
XenForo_ControllerAdmin_Tools->actionRunDeferred() in XenForo/FrontController.php at line 347
XenForo_FrontController->dispatch() in XenForo/FrontController.php at line 134
XenForo_FrontController->run() in /var/www/html/xen/admin.php at line 13After that the front end wouldn't respond at all until I restarted Apache.
I'm getting much closer though... Before I log in there are no forums visible (I guess this is an issue of permissions?), but I can log-in at the front end, and when I do forums, threads & posts seem to be where they should be.
Thanks again for your help (and any more pointers you can give),
R
Adam Bloch
Member
@Slavik I'll have a poke about, but I'm not sure that's it - the VM's currently 16.3GB and I'd initially allocated 40GB. Are there any other likely reasons? (just in case it's relevant, I've set InnoDB to auto-extend)
Will get the spanners out and report back once I've double-checked.
Thanks (again!)
R
EDIT: You might well have been on the money. VM's got disk space available, but I figured I'd drop the VB database just to see what happened, and it seems to be doing the conversation rebuild now. As is said around these parts, it's cob.
Could it have been a memory issue? (Thinking that storage might apply to RAM for InnoDB tables?)
Thanks again again!
Will get the spanners out and report back once I've double-checked.
Thanks (again!)
R
EDIT: You might well have been on the money. VM's got disk space available, but I figured I'd drop the VB database just to see what happened, and it seems to be doing the conversation rebuild now. As is said around these parts, it's cob.
Could it have been a memory issue? (Thinking that storage might apply to RAM for InnoDB tables?)
Thanks again again!
Last edited:
Jake Bunce
Well-known member
@Adam Bloch
It can be a specific partition on the disk that is full. SSH into your server and run the "df" command to view all partitions and their available space.
As for the search rebuild... I hope that you are using the Enhanced Search addon with Elasticsearch. The default (Fulltext) search engine doesn't scale well beyond 1 million posts or so.
It can be a specific partition on the disk that is full. SSH into your server and run the "df" command to view all partitions and their available space.
As for the search rebuild... I hope that you are using the Enhanced Search addon with Elasticsearch. The default (Fulltext) search engine doesn't scale well beyond 1 million posts or so.
Similar threads
- Question
- Question
