Fixed Loop importing first 500 posts

[Banxix]

So I try to import vB4 data to XF 2.1.0 Beta 4 and only first 499 posts are imported.
It ended at 499 rows. (self::MAX_POSTS)

Since then, no other posts were imported. But the counter in CLI keep increasing. I only noticed the strange thing when it was up to 2 millions but showed 0.00 percent.
In processlist of mysql, there are a lot of "TRUNCATE TABLE xf_css_cache" queries, and it kept happening.

Threads, profile posts and other tables above are imported ok.

 

[Kelico]

 

[Banxix]

This is really hard to debug because I had to re-run prequence steps in order to get posts. Trying to find a way to directly import posts.

The counter increase mean $newId should return true, but there is no post inserted.

if ($newId = $import->save($post['postid']))
{
    $state->imported++;
    $total++;
}

I will atempt to dump $newId.

 

[MaGeFH]

This happens when the importer step fails to set a new value for $step->start_at. Make sure it gets set properly.

 

[joeldaviddc]

I have the same problem, but trying to import from vb 5.4.5 to xenforo 2.1.2

 

[Banxix]

I have the same problem, but trying to import from vb 5.4.5 to xenforo 2.1.2

Send a ticket for support. Fix for this is easy.

 

[joeldaviddc]

ok, thanks

 

[Slavik]

It would appear if one of the first threads is more than 500 posts this causes it to restart.

A simple fix for now is to increase const MAX_POSTS = 500;
in
/src/addons/XFI/Import/Importer on line 22

to a higher number.

 

[Banxix]

It would appear if one of the first threads is more than 500 posts this causes it to restart.

A simple fix for now is to increase const MAX_POSTS = 500;
in
/src/addons/XFI/Import/Importer on line 22

to a higher number.

I think I have submitted my fix in this post but somehow it was moderated and got rejected.

A proper fix is to give $state->startAfter in stepPosts a value != 0 to prevent the loop.

Below I assign -1 to $state->startAfter

public function stepPosts(StepState $state, array $stepConfig, $maxTime, $limit = 50)
{
    if ($state->startAfter == 0)
    {
        // just in case these are lying around, get rid of them before continue...
        unset($state->extra['postDateStart'], $state->extra['postPosition']);

        // set this to a more sensible value so it won't clear out postDateStart again
        $state->startAfter = -1;
    }
    ...

 

[Mike]

This should be sorted for the next XFI release. Strictly speaking, I don't think this code is actually necessary. However, I have left it, but gated the unset behind ($state->startAfter == 0 && $state->imported == 0), which should ensure that it only runs on the very first pass (as intended).