1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

The import system

Discussion in 'XenForo Development Discussions' started by Robbo, Jun 22, 2011.

  1. Robbo

    Robbo Well-Known Member

    I have just written a few importers and one was for iTrader from vBulletin to XenTrader. iTrader stored user IDs for any feedback given but it never stored usernames. This meant I need the import log from the initial vBulletin import. What I think you must expect is for me to extend the vBulletin importers and add my step in, I could of done that but some people may have already imported and some people may have moved to vBulletin and keep the data of old mods waiting for ports.

    So I made the more standalone importer (used all the systems, just didn't extend the current importers) and was able to do some little hacky things to be able to get it to work. I will still have to make it clear to users about import logs so they don't stuff things up and lose their data.

    So when configuring the importer it asks for the import log archive table name from the initial vBulletin import and then I do the hacky part...
    PHP:
    define('IMPORT_LOG_TABLE'$this->_config['importlog']);
    I'm calling this at the start of my step, you can see why it is hacky... I don't like using define() at all, let alone in a method. And just in case you ask, when validating the config I do make sure the table is valid, I don't just inject the table name and hope.

    So what I would like is a better way to handle this kind of stuff, keep your define() stuff for redirecting scripts but add something a little nicer for importers. Simply having XenForo_Model_Import::getImportContentMap() accept a third argument for the import log table would work. And then a XenForo_Model_Import::importLogExists() or something for validation.

    And another thing I would like. After completing the import the controller executes some cache rebuilds. Since the caches that are being rebuilt don't have anything to do with my imports I would like to be able to skip it easily. Additionally, I would like to be able to inject my own cache rebuilds.
    So basically I would like to be able to modify...
    PHP:
    $caches = array(
    'User''Thread''Poll''Forum'
    );
    in the Import controller easily with extending it. Add to the importer getCachesToRebuild() or even just let me store them in the config and grab it from the session inthe controller.

    Sorry for wall of text, I hope I made sense.
     
  2. Robbo

    Robbo Well-Known Member

    Here is the importer just in case you need to look at it.

    PHP:
    <?php

    class XenTrader_Importer_ITrader extends XenTrader_Importer_Abstract
    {
    protected 
    $_sourceDb;

    protected 
    $_prefix;

    protected 
    $_config;

    protected 
    $_charset 'windows-1252';

    public static function 
    getName()
    {
    return 
    'iTrader';
    }

    public function 
    configure(XenForo_ControllerAdmin_Abstract $controller, array &$config)
    {
    if (
    $config)
    {
    if (
    $errors $this->validateConfiguration($config))
    return 
    $controller->responseError($errors);

    $this->_bootstrap($config);

    return 
    true;
    }

    $configPath getcwd() . '/includes/config.php';
    if (
    file_exists($configPath) && is_readable($configPath))
    {
    $config = array();
    include(
    $configPath);

    $viewParams = array('input' => $config);
    }
    else
    {
    $viewParams = array('input' => array
    (
    'MasterServer' => array
    (
    'servername' => 'localhost',
    'port' => 3306,
    'username' => '',
    'password' => '',
    ),
    'Database' => array
    (
    'dbname' => '',
    'tableprefix' => ''
    ),
    'Mysqli' => array
    (
    'charset' => ''
    ),
    ));
    }

    return 
    $controller->responseView('XenForo_ViewAdmin_Import_vBulletin_Config''xentrader_import_itrader_config'$viewParams);
    }

    public function 
    validateConfiguration(array &$config)
    {
    $errors = array();

    if (
    preg_match('/[^a-z0-9_]/i'$config['importlog']))
    {
    $errors[] = new XenForo_Phrase('error_table_name_illegal');
    }
    else
    {
    try
    {
    $table $this->_db->describeTable($config['importlog']);
    if (!isset(
    $table['content_type']) || !isset($table['old_id']) || !isset($table['new_id']))
    $errors[] = new XenForo_Phrase('xentrader_import_log_table_invalid');
    }
    catch (
    Zend_Db_Exception $e)
    {
    $errors[] = $e->getMessage();
    }
    }

    $config['db']['prefix'] = preg_replace('/[^a-z0-9_]/i'''$config['db']['prefix']);

    try
    {
    $db Zend_Db::factory('mysqli',
    array(
    'host' => $config['db']['host'],
    'port' => $config['db']['port'],
    'username' => $config['db']['username'],
    'password' => $config['db']['password'],
    'dbname' => $config['db']['dbname'],
    'charset' => $config['db']['charset']
    )
    );
    $db->getConnection();
    }
    catch (
    Zend_Db_Exception $e)
    {
    $errors[] = new XenForo_Phrase('source_database_connection_details_not_correct_x', array('error' => $e->getMessage()));
    }

    if (
    $errors)
    {
    return 
    $errors;
    }

    try
    {
    $db->query('
    SELECT userid
    FROM ' 
    $config['db']['prefix'] . 'user
    LIMIT 1
    '
    );
    }
    catch (
    Zend_Db_Exception $e)
    {
    if (
    $config['db']['dbname'] === '')
    {
    $errors[] = new XenForo_Phrase('please_enter_database_name');
    }
    else
    {
    $errors[] = new XenForo_Phrase('table_prefix_or_database_name_is_not_correct');
    }
    }

    if (!
    $errors)
    {
    $defaultLanguageId $db->fetchOne('
    SELECT value
    FROM ' 
    $config['db']['prefix'] . 'setting
    WHERE varname = \'languageid\'
    '
    );
    $defaultCharset $db->fetchOne('
    SELECT charset
    FROM ' 
    $config['db']['prefix'] . 'language
    WHERE languageid = ?
    '
    $defaultLanguageId);
    if (!
    $defaultCharset || str_replace('-'''strtolower($defaultCharset)) == 'iso88591')
    {
    $config['charset'] = 'windows-1252';
    }
    else
    {
    $config['charset'] = strtolower($defaultCharset);
    }
    }

    return 
    $errors;
    }

    public function 
    getSteps()
    {
    return array(
    'feedback' => array(
    'title' => new XenForo_Phrase('xentrader_import_feedback')
    )
    );
    }

    protected function 
    _bootstrap(array $config)
    {
    if (
    $this->_sourceDb)
    {
    // already run
    return;
    }

    @
    set_time_limit(0);

    $this->_config $config;

    $this->_sourceDb Zend_Db::factory('mysqli',
    array(
    'host' => $config['db']['host'],
    'port' => $config['db']['port'],
    'username' => $config['db']['username'],
    'password' => $config['db']['password'],
    'dbname' => $config['db']['dbname'],
    'charset' => $config['db']['charset']
    )
    );

    $this->_prefix preg_replace('/[^a-z0-9_]/i'''$config['db']['prefix']);

    if (!empty(
    $config['charset']))
    {
    $this->_charset $config['charset'];
    }
    }

    public function 
    stepFeedback($start, array $options)
    {
    define('IMPORT_LOG_TABLE'$this->_config['importlog']);

    $options array_merge(array(
    'limit' => 200,
    'max' => false
    ), $options);

    $sDb $this->_sourceDb;
    $prefix $this->_prefix;

    $importModel $this->_importModel;

    if (
    $options['max'] === false)
    {
    $options['max'] = $sDb->fetchOne('
    SELECT MAX(rateid)
    FROM ' 
    $prefix 'itrader
    '
    );
    }

    $feedback $sDb->fetchAll($sDb->limit(
    '
    SELECT *
    FROM ' 
    $prefix 'itrader
    WHERE rateid > ' 
    $sDb->quote($start) . '
    ORDER BY rateid
    '
    $options['limit']
    ));

    if (!
    $feedback)
    return 
    true;

    $next 0;
    $total 0;

    $userIds = array();
    $threadIds = array();
    foreach (
    $feedback as $k => $fb)
    {
    $userIds[] = $fb['rateduserid'];
    $userIds[] = $fb['userid'];

    $feedback[$k]['threadid'] = $this->_convertUrlToThreadId($fb['dealurl']);
    $threadIds[] = $feedback[$k]['threadid'];
    }
    $userIdMap $importModel->getImportContentMap('user'$userIds);
    $threadIdMap $importModel->getImportContentMap('thread'$threadIds);

    XenForo_Db::beginTransaction();

    foreach (
    $feedback as $fb)
    {
    $toUserId $this->_mapLookUp($userIdMap$fb['rateduserid']);
    if (!
    $toUserId)
    continue;

    $next $fb['rateid'];

    $fromUserId $this->_mapLookUp($userIdMap$fb['userid']);

    $threadId $this->_mapLookUp($threadIdMap$fb['threadid']);

    switch (
    $fb['buyselltrade'])
    {
    case 
    1$type 'buy'; break;
    case 
    2$type 'sell'; break;
    case 
    3$type 'trade'; break;
    }

    $import = array(
    'from_user_id' => $fromUserId,
    'to_user_id' => $toUserId,
    'from_username' => $this->_getUsernameFromId($fromUserId),
    'to_username' => $this->_getUsernameFromId($toUserId),
    'thread_id' => $threadId,
    'thread_title' => $this->_getThreadTitleFromId($threadId),
    'feedback' => $this->_convertToUtf8($fb['subject']),
    'rating' => $fb['rating'],
    'type' => $type,
    'feedback_date' => $fb['dateline'],
    );

    if (empty(
    $import['from_username']))
    $import['from_username'] = 'guest';

    if (empty(
    $import['to_username']))
    $import['to_username'] = 'guest';

    if (
    $importModel->importFeedback($fb['rateid'], $import))
    $total++;
    }

    XenForo_Db::commit();

    $this->_session->incrementStepImportTotal($total);

    return array(
    $next$options$this->_getProgressOutput($next$options['max']));
    }

    protected function 
    _convertUrlToThreadId($url)
    {
    $parsed parse_url(html_entity_decode(strtolower($url)));

    if (!
    $parsed || empty($parsed['scheme']) || $parsed['scheme'] != 'http')
    return 
    0;

    $friendUrls $this->_sourceDb->fetchOne('SELECT value FROM ' $this->_prefix 'setting WHERE varname = \'friendlyurl\'');

    $threadId 0;
    if (!
    $friendUrls)
    {
    $arr explode('&'$parsed['query']);

    foreach (
    $arr as $v)
    {
    $pairs explode('='$v);
    if (
    $pairs[0] == 't')
    {
    $threadId $pairs[1];
    break;
    }
    }
    }
    else
    {
    if (
    $friendUrls === 1)
    $parsed $parsed['path'] . $parsed['query'];
    else
    $parsed $parsed['path'];

    $parts explode('&'$parsed);

    foreach (
    $parts as $part)
    {
    switch (
    $friendUrls)
    {
    case 
    1$id explode('-'str_replace('showthread.php'''$part)); break;
    case 
    2$id explode('-'str_replace('showthread.php/'''$part)); break;
    case 
    3$id explode('-'str_replace('threads/'''$part)); break;
    }

    if (
    $id[0] && is_numeric($id[0]))
    {
    $threadId $id[0];
    break;
    }
    }

    }

    return 
    $threadId;
    }
    }
    The class it extends simply points $this->_importModel to an extended version of the model.
     
  3. Robbo

    Robbo Well-Known Member

    I thought more about this. What would be nice is if we could hook into importers. So for the above stuff I would hook into the vBulletin importer and add my step. Or, allow to extend it like models etc. That way I could simply add a xentrader feedback step which requires users and threads... What would be really cool is if you could extend in a way so that if they are using another importer it adds the step but if they have already used that importer it will just take the extended steps, add to its own importer and ask for an import archive. Hard to explain what I mean, if I was to bring up some code would it be considered? Anyone read this? :/
     
  4. ragtek

    ragtek Guest

  5. Robbo

    Robbo Well-Known Member

    Oh well there you go, don't know why I never thought of that. First stuff still applies though.
     

Share This Page