vBulletin 3.8.x importer seems to be stuck in loop on posts step

[Jake B.]

I've got a vBulletin database with around 6 million posts, the posts step has been running for about 6 hours now and has gotten to this point:

 - Step 16 of 26: Posts                     06:08:09 [10,547,242] 39.47%

It's imported about 98% of the posts based on the counts in the xf_post / post tables:

MariaDB [(none)]> select max(postid) from vbulletin_copy.post;
+-------------+
| max(postid) |
+-------------+
|     6607320 |
+-------------+
1 row in set (0.001 sec)

MariaDB [(none)]> select max(post_id) from xenforo.xf_post;
+--------------+
| max(post_id) |
+--------------+
|      6607320 |
+--------------+
1 row in set (0.001 sec)

MariaDB [(none)]> select count(postid) from vbulletin_copy.post;
+---------------+
| count(postid) |
+---------------+
|       6598710 |
+---------------+
1 row in set (4.643 sec)

MariaDB [(none)]> select count(post_id) from xenforo.xf_post;
+----------------+
| count(post_id) |
+----------------+
|        6494041 |
+----------------+
1 row in set (5.474 sec)

It's running with 100 processes currently, not sure if that's relevant:

php cmd.php xf:import --processes=100

 

[Kirby]

Wow, that is an insanely (?) large pool and does take way too lang.
With that many processes I would't be too surprised if you run into concurrency issues (deadlocks, etc.)

Usually I run imports at 8-16 processes and it has never taken that many hours, not even with 10+ million posts.

I'd check MySQL process list to see what's currently going on.
Are there any errors that have been logged?

 

[Jake B.]

Ah yeah, that was a typo - it's 10, not 100 processes, not sure how I managed that twice.

No errors logged, no crashes or anything like that. I'm going to try running it again with only a single process to see if it did run into something with multiple processes running tomorrow.

And just as a quick update on the process:

 - Step 16 of 26: Posts                     07:06:40 [14,574,803] 39.58%

Exactly the same number of rows in xf_post

MariaDB [(none)]> select count(post_id) from xenforo.xf_post;
+----------------+
| count(post_id) |
+----------------+
|        6494041 |
+----------------+
1 row in set (6.658 sec)

Haven't done too much debugging yet, but it's clearly no longer actually doing anything. Hopefully I should have some time to dig through and figure out what is actually going on tomorrow

 

[Chris D]

Not sure there's enough here yet to suggest there is definitely a bug.

Although I'm slightly confused by some of the output here (which may indicate a bug... )

Firstly, [10,547,242] in the command output is essentially how many times we've called $state->imported++; in that step. Obviously that shouldn't exceed 6598710 (the count inside the post table).

Second, the max ID from the post table and the xf_post table is the same. If retain IDs has been enabled then the import should be finished.

My bet is stopping the import and then re-running the command may sort it, or if that doesn't work, drop it down to a single process to see if it gets moving.

 

[Jake B.]

My bet is stopping the import and then re-running the command may sort it, or if that doesn't work, drop it down to a single process to see if it gets moving.

I've actually stopped and resumed it several times throughout the day just to see if something would happen and it didn't, but I kept it set to the 10 processes each time. I've just stopped and resumed with a single process so we'll see what happens:

$ time php cmd.php xf:import
Starting import from vBulletin 3.7, 3.8...
 - Step 16 of 26: Posts                     07:10:17 [14,986,120] 39.58%

 

[Chris D]

Although if that 10 million count is rising (which it seems to be) and the amount of content in your database isn't increasing then something is wrong so I don't think it's going to get further.

That does seem like it's stuck in a loop.

 

[Jake B.]

 - Step 16 of 26: Posts                     09:45:42 [14,986,608] 39.58%

Just went back to check and looks like it stopped incrementing entirely shortly after my previous post. I definitely think this migration is a lost cause . Going to reinstall XF tomorrow and try again with 5 processes instead of 10 and see what happens. Anything from the database that would help determine what exactly happened? I can save a dump of it to go back on if anything from it is useful

 

[Jake B.]

Alright, I've got it running again and it's up to the posts step:

 - Step 16 of 26: Posts                     00:18:30 [1,329,067] 21.05%

we'll see what happens

 

[Chris D]

If it fails from here, ideally we'd need a copy of the database submitted to us via a ticket so we can look into it further.

 

[Jake B.]

Okay, sounds good. Will see what happens on this run through, and if there are issues I'll check with them on getting DB access sent over

 

[Jake B.]

MariaDB [(none)]> select max(post_id) from xenforo.xf_post;
+--------------+
| max(post_id) |
+--------------+
|      6607320 |
+--------------+
1 row in set (0.002 sec)

MariaDB [(none)]> select max(postid) from vbulletin.post;
+-------------+
| max(postid) |
+-------------+
|     6607320 |
+-------------+
1 row in set (0.001 sec)

- Step 16 of 26: Posts 03:06:05 [6,493,847] 38.86%

Looks like we're in the same boat as before. The counter hasn't incremented to the full number of posts in vBulletin, but the max post ID is the same on both XF and vBulletin so I think we're in the same place. I'll let it run for a bit longer until it hits at least 7 million, but will be asking them if we can send SSH access to this server to have a look, will probably be creating the ticket from my XF account rather than theirs to make sure I can get email notifications if that's okay. We have access to their account so I'll include their account email so you can verify the license when I do that

 

[Claudio]

MariaDB [(none)]> select max(post_id) from xenforo.xf_post;
+--------------+
| max(post_id) |
+--------------+
|      6607320 |
+--------------+
1 row in set (0.002 sec)

MariaDB [(none)]> select max(postid) from vbulletin.post;
+-------------+
| max(postid) |
+-------------+
|     6607320 |
+-------------+
1 row in set (0.001 sec)

- Step 16 of 26: Posts 03:06:05 [6,493,847] 38.86%

Looks like we're in the same boat as before. The counter hasn't incremented to the full number of posts in vBulletin, but the max post ID is the same on both XF and vBulletin so I think we're in the same place. I'll let it run for a bit longer until it hits at least 7 million, but will be asking them if we can send SSH access to this server to have a look, will probably be creating the ticket from my XF account rather than theirs to make sure I can get email notifications if that's okay. We have access to their account so I'll include their account email so you can verify the license when I do that

Hi Jake,

How did you manage to resolve this? We're dealing with the same situation on a big forum with more than 10 million posts.

 

[Paul B]

The solution for this particular issue is as I posted in the ticket.

If that didn't resolve it then its a different issue.

Is not due to the number of posts overall, it's due to one thread with over 100,000 posts.