1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Is there some sort of speed limit for importing?

Discussion in 'Troubleshooting and Problems' started by NolF, Oct 6, 2010.

  1. NolF

    NolF Active Member

    I have a very large board (400k users, 1.7m posts) and the speed at which just users are being imported is eternally slow. Combine that with some duplicate entry errors and it will never finish :S

    Is there a trick to make it run faster? I have a dedicated Q6600 offpeak and the load is very low (~1)....
     
  2. graham_w

    graham_w Active Member

    When testing this afternoon it took 1 hour 45 minutes to import 750000 posts alone without the time taken for all the other items.
     
  3. NolF

    NolF Active Member

    Been playing with the importer open library\XenForo\Importer\Vbulletin.php

    Changing the value from 100 to 1000 or more makes it considerably faster
    PHP:
    'limit' => 100,
    There are four entries, one for Visitor Meeseges, Users, Threads, Polls. Change it as desired :)

    edit: ~2h for 750k posts, hopefully it won't take ~4h to import the ~2Gb worth of posts :/
     
  4. Mike

    Mike XenForo Developer Staff Member

    It will take a long time to import that much data. There's only so much that can be done. Aside from the data formats being different, consider that all the data has to be converted to UTF-8 as well.
     
  5. NolF

    NolF Active Member

    Fair points Mike, I hadn't considered the conversion :) Increasing the limit has helped too :D

    I'm wondering when the importer seemingly stops updating or the internet is disconnected, how does XenForo handle the F5? It seems to keep track of where it is in the import process but to what degree of accuracy? Would much data be lost?
     
  6. Mike

    Mike XenForo Developer Staff Member

    I suppose in theory it could try to import some dupes -- I'll try to check that. But if you refresh, it should pick up. The queries are almost always wrapped in a transaction, so if an error occurs, you can refresh without losing anything.

    I did get some dupe errors at one point with Chrome, but not with Firefox. I haven't figured out what's going on yet.
     
  7. NolF

    NolF Active Member

    That's excellent!! very cleaver solution :) I'll make sure not to use Chrome for the import once I figure out how to solve the annoying "Mysqli statement execute error : Incorrect string value: '\xAC~yaho...' for column 'account_name' at row 1" error :)
     
    Rob likes this.
  8. OperaManiac

    OperaManiac Well-Known Member

    i am facing my own unique problem. i cannot say that it is xenforo because i am facing the same problem on wordpress...

    on running import... it would land on a 404. no mysql overload error. no mysql down error. a simple 404.

    India Broadband Forum - Error

    The requested page could not be found.


    i cannot pinpoint any reason behind this mysterious problem i am facing on my server... wordpress too ends up taking me to a 404 on post publish.

    i would assume that if the mysql has been overloaded, it would give out a crash/toomanyconnections related error?
     
  9. Rob

    Rob Well-Known Member

    I got the same issue. Lots of people seems to have this same error at some point of the import process.... mine happens with threads and posts and it is caused by bad characters I'm sure.
     
  10. NolF

    NolF Active Member

    It's very likely. My users use lots of ACSII and Japanese characters so I wouldn't be surprised that there would be errors in the UTF-8 conversion. I hope it is something they can fix soon though, I want to play with this thing so badly :D
     
  11. steven s

    steven s Well-Known Member

    Mine continue to be with PMs and Forums. :(
     

Share This Page