Only import first 500 posts

Banxix

Member
So I try to import vB4 data to XF 2.1.0 Beta 4 and only first 499 posts are imported.
It ended at 499 rows. (self::MAX_POSTS)

Since then, no other posts were imported. But the counter in CLI keep increasing. I only noticed the strange thing when it was up to 2 millions but showed 0.00 percent.
In processlist of mysql, there are a lot of "TRUNCATE TABLE xf_css_cache" queries, and it kept happening.

Threads, profile posts and other tables above are imported ok.
 
Last edited:

Banxix

Member
This is really hard to debug because I had to re-run prequence steps in order to get posts. Trying to find a way to directly import posts.

The counter increase mean $newId should return true, but there is no post inserted.
PHP:
if ($newId = $import->save($post['postid']))
{
    $state->imported++;
    $total++;
}
I will atempt to dump $newId.
 

MaGeFH

Active member
This happens when the importer step fails to set a new value for $step->start_at. Make sure it gets set properly.
 

Slavik

XenForo moderator
Staff member
It would appear if one of the first threads is more than 500 posts this causes it to restart.

A simple fix for now is to increase const MAX_POSTS = 500;
in
/src/addons/XFI/Import/Importer on line 22

to a higher number.
 

Banxix

Member
It would appear if one of the first threads is more than 500 posts this causes it to restart.

A simple fix for now is to increase const MAX_POSTS = 500;
in
/src/addons/XFI/Import/Importer on line 22

to a higher number.
I think I have submitted my fix in this post but somehow it was moderated and got rejected.

A proper fix is to give $state->startAfter in stepPosts a value != 0 to prevent the loop.

Below I assign -1 to $state->startAfter

PHP:
public function stepPosts(StepState $state, array $stepConfig, $maxTime, $limit = 50)
{
    if ($state->startAfter == 0)
    {
        // just in case these are lying around, get rid of them before continue...
        unset($state->extra['postDateStart'], $state->extra['postPosition']);

        // set this to a more sensible value so it won't clear out postDateStart again
        $state->startAfter = -1;
    }
    ...
 
Top