Jake Bunce
Well-known member
To confirm, yes you can. Attachments are fully incremental with this importer. Each time you re-run the Export it only does new attachments. Considerable time can be saved by running attachments beforehand.
Resource icon
vBulletin Big Board Importer [vBulletin 3 + vBulletin 4] [Paid] 1.5.0
- Thread starter Slavik
- Start date
To confirm, yes you can. Attachments are fully incremental with this importer. Each time you re-run the Export it only does new attachments. Considerable time can be saved by running attachments beforehand.
Jake Bunce
Well-known member
Here is the latest attach function containing two bug fixes, one for vB4 file system attachments, and one for handling empty batches:
Code:
public function attachments ()
{
static $types = array(
1 => 'GIF',
2 => 'JPEG',
3 => 'PNG'
);
$globalStart = microtime(true);
$start = $this->db->fetchOne('
SELECT MAX(attachment_id)
FROM xf_attachment
');
$maxvB = $this->db->fetchOne('SELECT MAX(attachmentid) FROM ' . $this->sourceDb . '.attachment');
$fileName = '' . self::$dataDir . 'attachment_tmp';
$attachmentModel = XenForo_Model::create('XenForo_Model_Attachment');
$emptyBatch = false;
while (
// WHERE attachment.attachmentid BETWEEN ' . intval($start) . ' AND ' . intval($start + 999) . ' AND attachment.contenttypeid = 1
$attachments = $this->db->fetchAll('
SELECT attachment.attachmentid AS attachment_id, attachment.userid AS user_id, ' . (self::$isVB4 ? 'contentid' : 'postid') . ' AS content_id, attachment.dateline AS attachment_date, ' . (self::$isVB4 ? 'filedata.filedata' : 'attachment.filedata') . ' AS file, ' . (self::$isVB4 ? 'filedata.thumbnail_filesize' : 'attachment.thumbnail_filesize') . ', attachment.filename, ' . (self::$isVB4 ? 'filedata.extension' : 'attachment.extension') . ' AS extension, ' . (self::$isVB4 ? 'filedata.filedataid' : 'attachment.attachmentid') . ' AS filesysid
FROM ' . $this->sourceDb . '.attachment
' . (self::$isVB4 ? 'LEFT JOIN ' . $this->sourceDb . '.filedata ON (filedata.filedataid = attachment.filedataid)' : '') . '
WHERE attachment.attachmentid BETWEEN ' . intval($start) . ' AND ' . intval($start + 999) . (self::$isVB4 ? ' AND attachment.contenttypeid = 1' : '') . '
' . (self::$isVB4 ? 'GROUP BY attachment.filedataid' : '') . '
ORDER BY attachment.attachmentid
')
OR $emptyBatch = (($start + 999) < $maxvB)
)
{
if ($emptyBatch)
{
echo "\r\n EMPTY BATCH (no 'post' attachments in range)\r\n";
$start = $start + 1000;
$emptyBatch = false;
continue;
}
echo "\r\n " . number_format(count($attachments)) . " new attachments\r\n";
foreach ($attachments as $attachment)
{
echo $attachment['attachment_id'] . ' ';
if (self::$attachFile AND file_exists(self::$attachFile . implode('/', preg_split('//', $attachment['user_id'], -1, PREG_SPLIT_NO_EMPTY)) . '/' . strval($attachment['filesysid']) . '.attach'))
{
$attachment['file'] = file_get_contents(self::$attachFile . implode('/', preg_split('//', $attachment['user_id'], -1, PREG_SPLIT_NO_EMPTY)) . '/' . strval($attachment['filesysid']) . '.attach');
}
else
{
$attachment['file'] = $attachment['file'];
}
if (!$attachment['file'])
{
echo '!!MISSING DATA!!';
continue;
}
if ($attachment['thumbnail_filesize'] > 0 AND $attachment['extension'] != 'bmp')
{
file_put_contents($fileName, $attachment['file']);
$tempThumbFile = tempnam(XenForo_Helper_File::getTempDir(), 'xf');
$imageInfo = getimagesize($fileName);
if ($imageInfo === false)
{
echo '!!BAD!!';
$attachment['width'] = 0;
$attachment['height'] = 0;
$attachment['thumbnail_width'] = 0;
$attachment['thumbnail_height'] = 0;
$attachment['thumbnail_file'] = '';
$attachment['file_type'] = '';
}
else
{
$attachment['width'] = $imageInfo[0];
$attachment['height'] = $imageInfo[1];
$attachment['thumbnail_width'] = 0;
$attachment['thumbnail_height'] = 0;
$attachment['thumbnail_file'] = '';
$attachment['file_type'] = strtoupper(image_type_to_extension($imageInfo[2], false));
$image = XenForo_Image_Abstract::createFromFile($fileName, $imageInfo[2]);
if ($image)
{
if ($image->thumbnail(XenForo_Application::get('options')->attachmentThumbnailDimensions))
{
$image->output($imageInfo[2], $tempThumbFile);
$attachment['thumbnail_file'] = file_get_contents($tempThumbFile);
unlink($tempThumbFile);
}
else
{
$attachment['thumbnail_file'] = $attachment['file'];
}
$attachment['thumbnail_width'] = $image->getWidth();
$attachment['thumbnail_height'] = $image->getHeight();
unset($image);
}
}
}
else
{
$attachment['width'] = 0;
$attachment['height'] = 0;
$attachment['thumbnail_width'] = 0;
$attachment['thumbnail_height'] = 0;
$attachment['thumbnail_file'] = '';
$attachment['file_type'] = '';
}
if (self::$extraOutCommand)
{
$attachFileNameTmp = array();
exec('echo $\'' . addslashes($attachment['filename']) . '\' ' . self::$extraOutCommand , $attachFileNameTmp);
$attachFileName = $attachFileNameTmp[0];
}
else
{
$attachFileName = $attachment['filename'];
}
$this->db->query('
INSERT INTO xf_attachment_data
(user_id, upload_date, filename, file_size, file_hash, width, height, thumbnail_width, thumbnail_height, attach_count)
VALUES
(' . $attachment['user_id'] . ', ' . $attachment['attachment_date'] . ', "' . addslashes($attachFileName) . '", ' . strlen($attachment['file']) . ', "' . addslashes(md5($attachment['file'])) . '", ' . $attachment['width'] . ', ' . $attachment['height'] . ', ' . $attachment['thumbnail_width'] . ', ' . $attachment['thumbnail_height'] . ', 1)
');
$attachmentDataId = $this->db->lastInsertId();
$this->db->query('
INSERT INTO xf_attachment
(attachment_id, data_id, content_type, content_id, attach_date, temp_hash, unassociated, view_count)
VALUES
(' . $attachment['attachment_id'] . ', ' . $attachmentDataId . ', "post", ' . $attachment['content_id'] . ', ' . $attachment['attachment_date'] . ', "", 0, 0)
ON DUPLICATE KEY UPDATE
data_id = VALUES(data_id), content_type = VALUES(content_type), content_id = VALUES(content_id), attach_date = VALUES(attach_date), temp_hash = VALUES(temp_hash), unassociated = VALUES(unassociated), view_count = VALUES(view_count)
');
// SKIP THIS AND WRITE TO FILE INSTEAD
$path = $attachmentModel->getAttachmentDataFilePath(array(
'data_id' => $attachmentDataId,
'file_hash' => md5($attachment['file']),
));
XenForo_Helper_File::createDirectory(dirname($path));
file_put_contents($path, $attachment['file']);
if ($attachment['thumbnail_width'])
{
$path = $attachmentModel->getAttachmentThumbnailFilePath(array(
'data_id' => $attachmentDataId,
'file_hash' => md5($attachment['file']),
));
XenForo_Helper_File::createDirectory(dirname($path), true);
file_put_contents($path, $attachment['thumbnail_file']);
}
}
unset ($attachments);
$start = $start + 1000;
//break;
}
echo "\r\n updating attachment view count";
$this->db->query('
UPDATE xf_attachment
LEFT JOIN ' . $this->sourceDb . '.attachment ON (attachment.attachmentid = xf_attachment.attachment_id)
SET xf_attachment.view_count = attachment.counter
');
$this->db->closeConnection();
echo "\r\nTotal time for attachments: " . number_format(microtime(true) - $globalStart, 2) . "s\r\n";
}
Jake Bunce
Well-known member
Attachments are actually written directly to XF's data and internal_data directories, and appropriate records written to XF's database. They don't get written out to the importData directory like other types.
Jake Bunce
Well-known member
Your shell connection is timing out? You would have to increase that timeout. Or use a program like 'screen' if it's installed on your server:
Code:
screen -S blahThat will start a screen. Run the Export script in your screen session.
To detach use Ctrl-a d
You can reattach using:
Code:
screen -r blahThat way the process is running in screen which is independent of your shell connection. If you timeout then you can reattach to your screen.
I suggest outputting to a file when using screen:
Code:
php -f Export.php > outputfile.txt
If using putty, you can also go to the connection option and add in 180 to send null packets to prevent timeouts.
Marcel
Active member
Right, had a few problems getting Export.php going, but sorted it all out, *almost* got it running and whoah! It's fast.
Insanely fast!
However, I'm having a problem Exporting Users.
I've upped the value in xcache.ini and restarted httpd, but it's still crapping out.
Is this something specific to our server that's not right, or is it something in the code that's throwing me a curveball?
Thanks
Insanely fast!
However, I'm having a problem Exporting Users.
Code:
exporting users...
user & user authenticate
An exception occurred: xcache_set(): xcache.var_size is either 0 or too small to enable var data caching in Zend/Cache/Backend/Xcache.php on line 134I've upped the value in xcache.ini and restarted httpd, but it's still crapping out.
Is this something specific to our server that's not right, or is it something in the code that's throwing me a curveball?
Thanks
Not yet, however you can just alter one of the other likes import code, just change the feild names to the ones your system uses.
Marcel
Active member
Importer almost fully functional for our forums
I do have a question. Could I have found a bug (Well, more of an oversight).
We have a forum that has no parent (Our welcome forum).
Export.php exports it as a category.
I think I've narrowed down the relevant code.
In public function forums (), the nodes section, you have an IF conditional that states if a forum has a parentid of -1, Export as a category, else a forum.
Well our welcome forum has no parent and therefore is exported as a category.
I did try adding a further conditional (eg AND forum.forumid = 11). Not sure if it's correct, but I did manage to get it working and exporting as a forum (I checked by viewing the txt file).
However, when I import it, it imports as a forum, but it doesn't 'work' when I try to edit or view it (I just get an error "node does not exist" or something like that).
Cheers.
I do have a question. Could I have found a bug (Well, more of an oversight).
We have a forum that has no parent (Our welcome forum).
Export.php exports it as a category.
I think I've narrowed down the relevant code.
In public function forums (), the nodes section, you have an IF conditional that states if a forum has a parentid of -1, Export as a category, else a forum.
Well our welcome forum has no parent and therefore is exported as a category.
I did try adding a further conditional (eg AND forum.forumid = 11). Not sure if it's correct, but I did manage to get it working and exporting as a forum (I checked by viewing the txt file).
However, when I import it, it imports as a forum, but it doesn't 'work' when I try to edit or view it (I just get an error "node does not exist" or something like that).
Cheers.
Importer almost fully functional for our forums
I do have a question. Could I have found a bug (Well, more of an oversight).
We have a forum that has no parent (Our welcome forum).
Export.php exports it as a category.
I think I've narrowed down the relevant code.
In public function forums (), the nodes section, you have an IF conditional that states if a forum has a parentid of -1, Export as a category, else a forum.
Well our welcome forum has no parent and therefore is exported as a category.
I did try adding a further conditional (eg AND forum.forumid = 11). Not sure if it's correct, but I did manage to get it working and exporting as a forum (I checked by viewing the txt file).
However, when I import it, it imports as a forum, but it doesn't 'work' when I try to edit or view it (I just get an error "node does not exist" or something like that).
Cheers.
The easiest way would be to install waindigos import tools, and import the forums on their own, and comment out the import forums section of the import script.
Similar threads
- Question
- Question
