XF 2.1 CLI post import progress in 2.1.6 broken?

[mazdas247]

In 2.1.6 I am scared about the progress of imports:

- Step 15 of 25: Posts 00:57:35 [461,583] 0.03%
and shortly after:
- Step 15 of 25: Posts 00:55:06 [440,970] 0.03%

0.03% is completely wrong. I have 5.5M post, this should be more like 8%.

The rate is approx 147 post per sec, and so that means 10H, which is how long it took on 2.1.5, but 2.1.5 displayed proper statistics during the import.

Did something fundamentally change in how you calculate the percentage in the new importers? I noticed heavy changes in Importer/vBulletin.php, and that looks like stepPosts is just completely gone. Point is, should I worry, because if it's really 0.03%, it will take 4+ months to complete?

 

[mazdas247]

Edit - 2.1.6 patch 1

 

[mazdas247]

Further "progress" update:

- Step 15 of 25: Posts 03:57:30 [2,000,203] 99.80%

There's some bad math going on here for sure.

 

[mazdas247]

FWIW, i have a huge discontinuity in my thread/post numbers, so if the percentage is calculated based on id of last - id of first and assuming they are sequential, that would explain the behaviour.

 

[Paul B]

I was going to post about that last night but got distracted.

How huge is huge?
It must be massive to account for those percentages.

 

[mazdas247]

Stupid big, billions? Not sure why vb did that.

 

[Chris D]

That would certainly account for it.

There is an edge case/known issue as well if you are running the import with multiple processes.

But on the whole this is expected. The most performant way of tracking the progress during an import is based on the max ID of a given table.

 

[mazdas247]

Just curious what's wrong with a simple SELECT COUNT(*) ... within threads, post?

 

[MySiteGuy]

Just curious what's wrong with a simple SELECT COUNT(*) ... within threads, post?

select count(*) is an "expensive" query with InnoDB table types.
.