1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

XF 1.1 CLI Speed up? - 4m posts in 50 secs?

Discussion in 'XenForo Questions and Support' started by MGSteve, Nov 29, 2011.

  1. MGSteve

    MGSteve Well-Known Member

    Has the CLI importer been tweaked, its just claimed to have imported 4m posts in 50 seconds?

    Although there only seems to be 9,000 odd records in the post table, which sort of gives it away perhaps ;)
    This did work in previous 1.1 betas, has anyone else run it on 1.1 recently?
  2. MGSteve

    MGSteve Well-Known Member

    ah, I think I've answered by own question. I got a bunch of vB error reports saying mysql failed to connect - "to many connections"... this probably was the same here.

    Perhaps worth amending the scripts to cope with mysql failures on subsequent runs. i.e. bomb out with an error on the first run, but wait and retry on subsequent iterations.
  3. Mike

    Mike XenForo Developer Staff Member

    It does - it tries to import the same thing 5 times before moving on.
  4. MGSteve

    MGSteve Well-Known Member

    with any time gap between them as the 32 threads failed within 50 seconds. They ran once and then failed to run again?
  5. Mike

    Mike XenForo Developer Staff Member

    No, no gap. :)
  6. MGSteve

    MGSteve Well-Known Member

    May be worth adding a gap then, otherwise if its connection related, its really as good as having no retries at all. :)

    It also said it imported 414,000 threads, which is a tad incorrect - unless it imports the threads first and then goes back and imports the posts for the threads, but I'd have thought it would have done it thread-by-thread.
  7. Mario Estrada

    Mario Estrada New Member

    21:39:03    Approximately 161,500 threads remaining to import.
    21:39:33    Approximately 160,300 threads remaining to import.
    21:40:03    Approximately 158,800 threads remaining to import.
    21:40:33    Approximately 156,900 threads remaining to import.
    21:41:03    Waiting for processes to terminate....
    21:41:08    All done. Imported 161,500 threads.
    Looks like i've ran into the same issue, any idea how we could get it to work?
  8. Floren

    Floren Well-Known Member

    This looks like a MySQL connection failure, related to too many connections. Setting a large number of connections into my.cnf is useless if you don't adjust the number of open files related to your mysql user. For example, 1024 open files will allow you only 214 connections.
  9. MGSteve

    MGSteve Well-Known Member

    Yeah, that is one way, but some people don't have access to the mySQL configurations if they're on a hosted / shared server.

    I do, but its already set quite high anyway... I'm actually not sure why I got the out of connections error I did, the limit is not normally reached anyway (its in the hundreds).

    But some degree of either handling or recovery (i.e. we can just run the script again to carry on) would be great.

    Last thing I need is for this to go fine for 4m posts and then fall over right near the end!

Share This Page