XF 1.1 CLI Speed up? - 4m posts in 50 secs?

[MGSteve]

Has the CLI importer been tweaked, its just claimed to have imported 4m posts in 50 seconds?

01:37:43 Approximately 414,100 threads remaining to import.
01:38:13 Waiting for processes to terminate.......
01:38:33 All done. Imported 414,100 threads.

Although there only seems to be 9,000 odd records in the post table, which sort of gives it away perhaps
This did work in previous 1.1 betas, has anyone else run it on 1.1 recently?

 

[MGSteve]

ah, I think I've answered by own question. I got a bunch of vB error reports saying mysql failed to connect - "to many connections"... this probably was the same here.

Perhaps worth amending the scripts to cope with mysql failures on subsequent runs. i.e. bomb out with an error on the first run, but wait and retry on subsequent iterations.

 

[Mike]

It does - it tries to import the same thing 5 times before moving on.

 

[MGSteve]

with any time gap between them as the 32 threads failed within 50 seconds. They ran once and then failed to run again?

 

[Mike]

No, no gap.

 

[MGSteve]

May be worth adding a gap then, otherwise if its connection related, its really as good as having no retries at all.

It also said it imported 414,000 threads, which is a tad incorrect - unless it imports the threads first and then goes back and imports the posts for the threads, but I'd have thought it would have done it thread-by-thread.

 

[Mario Estrada]

21:39:03    Approximately 161,500 threads remaining to import.
21:39:33    Approximately 160,300 threads remaining to import.
21:40:03    Approximately 158,800 threads remaining to import.
21:40:33    Approximately 156,900 threads remaining to import.
21:41:03    Waiting for processes to terminate....
21:41:08    All done. Imported 161,500 threads.

Looks like i've ran into the same issue, any idea how we could get it to work?

 

[Floren]

This looks like a MySQL connection failure, related to too many connections. Setting a large number of connections into my.cnf is useless if you don't adjust the number of open files related to your mysql user. For example, 1024 open files will allow you only 214 connections.

 

[MGSteve]

Yeah, that is one way, but some people don't have access to the mySQL configurations if they're on a hosted / shared server.

I do, but its already set quite high anyway... I'm actually not sure why I got the out of connections error I did, the limit is not normally reached anyway (its in the hundreds).

But some degree of either handling or recovery (i.e. we can just run the script again to carry on) would be great.

Last thing I need is for this to go fine for 4m posts and then fall over right near the end!