The import system

[Robbo]

I have just written a few importers and one was for iTrader from vBulletin to XenTrader. iTrader stored user IDs for any feedback given but it never stored usernames. This meant I need the import log from the initial vBulletin import. What I think you must expect is for me to extend the vBulletin importers and add my step in, I could of done that but some people may have already imported and some people may have moved to vBulletin and keep the data of old mods waiting for ports.

So I made the more standalone importer (used all the systems, just didn't extend the current importers) and was able to do some little hacky things to be able to get it to work. I will still have to make it clear to users about import logs so they don't stuff things up and lose their data.

So when configuring the importer it asks for the import log archive table name from the initial vBulletin import and then I do the hacky part...

define('IMPORT_LOG_TABLE', $this->_config['importlog']);

I'm calling this at the start of my step, you can see why it is hacky... I don't like using define() at all, let alone in a method. And just in case you ask, when validating the config I do make sure the table is valid, I don't just inject the table name and hope.

So what I would like is a better way to handle this kind of stuff, keep your define() stuff for redirecting scripts but add something a little nicer for importers. Simply having XenForo_Model_Import::getImportContentMap() accept a third argument for the import log table would work. And then a XenForo_Model_Import::importLogExists() or something for validation.

And another thing I would like. After completing the import the controller executes some cache rebuilds. Since the caches that are being rebuilt don't have anything to do with my imports I would like to be able to skip it easily. Additionally, I would like to be able to inject my own cache rebuilds.
So basically I would like to be able to modify...

$caches = array(
'User', 'Thread', 'Poll', 'Forum'
);

in the Import controller easily with extending it. Add to the importer getCachesToRebuild() or even just let me store them in the config and grab it from the session inthe controller.

Sorry for wall of text, I hope I made sense.

 

[Robbo]

Here is the importer just in case you need to look at it.

<?php

class XenTrader_Importer_ITrader extends XenTrader_Importer_Abstract
{
protected $_sourceDb;

protected $_prefix;

protected $_config;

protected $_charset = 'windows-1252';

public static function getName()
{
return 'iTrader';
}

public function configure(XenForo_ControllerAdmin_Abstract $controller, array &$config)
{
if ($config)
{
if ($errors = $this->validateConfiguration($config))
return $controller->responseError($errors);

$this->_bootstrap($config);

return true;
}

$configPath = getcwd() . '/includes/config.php';
if (file_exists($configPath) && is_readable($configPath))
{
$config = array();
include($configPath);

$viewParams = array('input' => $config);
}
else
{
$viewParams = array('input' => array
(
'MasterServer' => array
(
'servername' => 'localhost',
'port' => 3306,
'username' => '',
'password' => '',
),
'Database' => array
(
'dbname' => '',
'tableprefix' => ''
),
'Mysqli' => array
(
'charset' => ''
),
));
}

return $controller->responseView('XenForo_ViewAdmin_Import_vBulletin_Config', 'xentrader_import_itrader_config', $viewParams);
}

public function validateConfiguration(array &$config)
{
$errors = array();

if (preg_match('/[^a-z0-9_]/i', $config['importlog']))
{
$errors[] = new XenForo_Phrase('error_table_name_illegal');
}
else
{
try
{
$table = $this->_db->describeTable($config['importlog']);
if (!isset($table['content_type']) || !isset($table['old_id']) || !isset($table['new_id']))
$errors[] = new XenForo_Phrase('xentrader_import_log_table_invalid');
}
catch (Zend_Db_Exception $e)
{
$errors[] = $e->getMessage();
}
}

$config['db']['prefix'] = preg_replace('/[^a-z0-9_]/i', '', $config['db']['prefix']);

try
{
$db = Zend_Db::factory('mysqli',
array(
'host' => $config['db']['host'],
'port' => $config['db']['port'],
'username' => $config['db']['username'],
'password' => $config['db']['password'],
'dbname' => $config['db']['dbname'],
'charset' => $config['db']['charset']
)
);
$db->getConnection();
}
catch (Zend_Db_Exception $e)
{
$errors[] = new XenForo_Phrase('source_database_connection_details_not_correct_x', array('error' => $e->getMessage()));
}

if ($errors)
{
return $errors;
}

try
{
$db->query('
SELECT userid
FROM ' . $config['db']['prefix'] . 'user
LIMIT 1
');
}
catch (Zend_Db_Exception $e)
{
if ($config['db']['dbname'] === '')
{
$errors[] = new XenForo_Phrase('please_enter_database_name');
}
else
{
$errors[] = new XenForo_Phrase('table_prefix_or_database_name_is_not_correct');
}
}

if (!$errors)
{
$defaultLanguageId = $db->fetchOne('
SELECT value
FROM ' . $config['db']['prefix'] . 'setting
WHERE varname = \'languageid\'
');
$defaultCharset = $db->fetchOne('
SELECT charset
FROM ' . $config['db']['prefix'] . 'language
WHERE languageid = ?
', $defaultLanguageId);
if (!$defaultCharset || str_replace('-', '', strtolower($defaultCharset)) == 'iso88591')
{
$config['charset'] = 'windows-1252';
}
else
{
$config['charset'] = strtolower($defaultCharset);
}
}

return $errors;
}

public function getSteps()
{
return array(
'feedback' => array(
'title' => new XenForo_Phrase('xentrader_import_feedback')
)
);
}

protected function _bootstrap(array $config)
{
if ($this->_sourceDb)
{
// already run
return;
}

@set_time_limit(0);

$this->_config = $config;

$this->_sourceDb = Zend_Db::factory('mysqli',
array(
'host' => $config['db']['host'],
'port' => $config['db']['port'],
'username' => $config['db']['username'],
'password' => $config['db']['password'],
'dbname' => $config['db']['dbname'],
'charset' => $config['db']['charset']
)
);

$this->_prefix = preg_replace('/[^a-z0-9_]/i', '', $config['db']['prefix']);

if (!empty($config['charset']))
{
$this->_charset = $config['charset'];
}
}

public function stepFeedback($start, array $options)
{
define('IMPORT_LOG_TABLE', $this->_config['importlog']);

$options = array_merge(array(
'limit' => 200,
'max' => false
), $options);

$sDb = $this->_sourceDb;
$prefix = $this->_prefix;

$importModel = $this->_importModel;

if ($options['max'] === false)
{
$options['max'] = $sDb->fetchOne('
SELECT MAX(rateid)
FROM ' . $prefix . 'itrader
');
}

$feedback = $sDb->fetchAll($sDb->limit(
'
SELECT *
FROM ' . $prefix . 'itrader
WHERE rateid > ' . $sDb->quote($start) . '
ORDER BY rateid
', $options['limit']
));

if (!$feedback)
return true;

$next = 0;
$total = 0;

$userIds = array();
$threadIds = array();
foreach ($feedback as $k => $fb)
{
$userIds[] = $fb['rateduserid'];
$userIds[] = $fb['userid'];

$feedback[$k]['threadid'] = $this->_convertUrlToThreadId($fb['dealurl']);
$threadIds[] = $feedback[$k]['threadid'];
}
$userIdMap = $importModel->getImportContentMap('user', $userIds);
$threadIdMap = $importModel->getImportContentMap('thread', $threadIds);

XenForo_Db::beginTransaction();

foreach ($feedback as $fb)
{
$toUserId = $this->_mapLookUp($userIdMap, $fb['rateduserid']);
if (!$toUserId)
continue;

$next = $fb['rateid'];

$fromUserId = $this->_mapLookUp($userIdMap, $fb['userid']);

$threadId = $this->_mapLookUp($threadIdMap, $fb['threadid']);

switch ($fb['buyselltrade'])
{
case 1: $type = 'buy'; break;
case 2: $type = 'sell'; break;
case 3: $type = 'trade'; break;
}

$import = array(
'from_user_id' => $fromUserId,
'to_user_id' => $toUserId,
'from_username' => $this->_getUsernameFromId($fromUserId),
'to_username' => $this->_getUsernameFromId($toUserId),
'thread_id' => $threadId,
'thread_title' => $this->_getThreadTitleFromId($threadId),
'feedback' => $this->_convertToUtf8($fb['subject']),
'rating' => $fb['rating'],
'type' => $type,
'feedback_date' => $fb['dateline'],
);

if (empty($import['from_username']))
$import['from_username'] = 'guest';

if (empty($import['to_username']))
$import['to_username'] = 'guest';

if ($importModel->importFeedback($fb['rateid'], $import))
$total++;
}

XenForo_Db::commit();

$this->_session->incrementStepImportTotal($total);

return array($next, $options, $this->_getProgressOutput($next, $options['max']));
}

protected function _convertUrlToThreadId($url)
{
$parsed = parse_url(html_entity_decode(strtolower($url)));

if (!$parsed || empty($parsed['scheme']) || $parsed['scheme'] != 'http')
return 0;

$friendUrls = $this->_sourceDb->fetchOne('SELECT value FROM ' . $this->_prefix . 'setting WHERE varname = \'friendlyurl\'');

$threadId = 0;
if (!$friendUrls)
{
$arr = explode('&', $parsed['query']);

foreach ($arr as $v)
{
$pairs = explode('=', $v);
if ($pairs[0] == 't')
{
$threadId = $pairs[1];
break;
}
}
}
else
{
if ($friendUrls === 1)
$parsed = $parsed['path'] . $parsed['query'];
else
$parsed = $parsed['path'];

$parts = explode('&', $parsed);

foreach ($parts as $part)
{
switch ($friendUrls)
{
case 1: $id = explode('-', str_replace('showthread.php', '', $part)); break;
case 2: $id = explode('-', str_replace('showthread.php/', '', $part)); break;
case 3: $id = explode('-', str_replace('threads/', '', $part)); break;
}

if ($id[0] && is_numeric($id[0]))
{
$threadId = $id[0];
break;
}
}

}

return $threadId;
}
}

The class it extends simply points $this->_importModel to an extended version of the model.

 

[Robbo]

I thought more about this. What would be nice is if we could hook into importers. So for the above stuff I would hook into the vBulletin importer and add my step. Or, allow to extend it like models etc. That way I could simply add a xentrader feedback step which requires users and threads... What would be really cool is if you could extend in a way so that if they are using another importer it adds the step but if they have already used that importer it will just take the extended steps, add to its own importer and ask for an import archive. Hard to explain what I mean, if I was to bring up some code would it be considered? Anyone read this? :/

 

[ragtek]

That's already possible?!

http://xenforo.com/community/threads/guide-extend-the-importers.5825/

 

[Robbo]

Oh well there you go, don't know why I never thought of that. First stuff still applies though.