1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Adopt the Digital Point importer as standard

Discussion in 'General XenForo Discussion and Feedback' started by Stuart Wright, Mar 20, 2013.

  1. Stuart Wright

    Stuart Wright Well-Known Member

    http://xenforo.com/community/resources/a-r-f-i-a-really-fast-vb4-importer.1397/
    Being worked on by Slavik.
    Please integrate this importer into the standard product.
    In order to set up AVForums on Xenforo, I would like to import a copy of the data so we can fully test it and the new functionality we will be writing on a large database.
    And when it comes to upgrading, closing the forums for an hour while we migrate is of course way less painful.
     
    Andrej, feldon30, kontrabass and 24 others like this.
  2. emc2

    emc2 Active Member

    ditto..
     
  3. klyde

    klyde Member

  4. Paul

    Paul Well-Known Member

  5. CyclingTribe

    CyclingTribe Well-Known Member

    Definitely a plus point for getting larger (vB) forums over to XF ... (y)
     
  6. MGSteve

    MGSteve Well-Known Member

    +1 we also need the importer to migrate Paid Subs over as well, as much as possible.
     
  7. Forsaken

    Forsaken Well-Known Member

    Unless you like the first post your +1 statements don't mean anything.
     
    SneakyDave and Neal like this.
  8. Rigel Kentaurus

    Rigel Kentaurus Well-Known Member

    It might be close to impossible to have that importer as an standard out of the box feature. The importer, as described, runs on CLI (a hurdle for most people), then it requires customization, and it is not the standard database read & write but builds files, parses, then batch imports, and I can see that going really bad with different server configurations (might work in one Linux Distro, definitely not on windows, maybe on Mac ... )

    So, as far as making it work for "everybody", I think I would re-read the text in red at the Digitalpoint site


    Screen Shot 2013-03-20 at 2.51.44 AM.png
     
  9. Jarod

    Jarod Active Member

  10. Digital Doctor

    Digital Doctor Well-Known Member

    What about Waindigo's ?
    http://xenforo.com/community/resources/import-tools-by-waindigo.1255/


    Ideally Xenforo would adopt some of the features of his importer ...

    I think lots of people would like to be able to redo steps, etc.
     
    Gladius and SchmitzIT like this.
  11. Paul

    Paul Well-Known Member

    I believe Slavik was implementing some stuff so that it would function like a standard migration tool to work with the "standard product"... as per first post by Stuart....
     
  12. Stuart Wright

    Stuart Wright Well-Known Member

    As I understand it, Slavik has been working on creating a more generic version of the importer.
    But yes, where certain specific server/setup requirements are needed, then perhaps the importer can check for those and refuse to run unless they are satisfied?

    The background for us is that some year(s) ago I did a test import which failed after 48 hours. Obviously from 48 hours down to less than one is a significant improvement :)
     
    thumped likes this.
  13. RoldanLT

    RoldanLT Well-Known Member

  14. RobParker

    RobParker Well-Known Member

    I don't think it makes any sense to make this "standard" as anyone who needs it is a pretty extreme use case and also requires a fair amount of technical expertise compared to a "standard" user. I think making it more generic (as Slavik has done) is extremely useful for people.

    The danger of making it a part of the core is that a lot of new users who neither need it or properly understand how to use it would then have problems which would need support.
     
    Mark3121 likes this.
  15. Stuart Wright

    Stuart Wright Well-Known Member

    Ok by 'Standard Product', I mean can it be adopted by the XF team, released and supported officially, as an additional resource if not in the standard product download?
    I know naff all about server configuration and don't touch MySQL other than through phpMyAdmin and I leave all that stuff to Tim at Nimbus Hosting as I'm too busy running the business and playing Black Ops II. ;)

    If there is a way to package this in a way that dummies like me can have a go with it, then that would be superb.

    We're likely going to need additional importing steps to import some additional data like iTrader ratings (if it's not imported by the addon), but an official importer will be good for us, many other forums and indeed Xenforo. How many forum platforms can boast importing ten million plus records in an hour or less? Awesome.
     
  16. shawn

    shawn Well-Known Member

    There is also a XF-written CLI, but it's also essentially unsupported/beta. Maybe there's a market for XF to offer customized import services instead. I don't know that it's realistic to think that the stock importer should be able to work quickly on +10MM post forums, especially if that importer is pushing the data through the stock XF datawriters.

    Waindigo's add-on is for web-based imports. I'm not sure what the realistic limitations of the web importer is, but I can't imagine it's much beyond 2MM posts. I did just shy of 1MM posts, but it took something like 8-10 hours.
     
  17. klyde

    klyde Member

    I also have to import a forum from vb4 with more than 16 millions of posts and i would like to do it as fast as possible. I will wait until 1.2 version to see the improvements. Migrations between forum software is not a easy decision.....
     
  18. Stuart Wright

    Stuart Wright Well-Known Member

    From vB3 to XF is an easy decision to make, it's the doing of it which is not so easy :)
     
    Thomas and CyclingTribe like this.
  19. Slavik

    Slavik XenForo Moderator Staff Member


    All going well it should be fully working by the end of the week, just need some more datasets to test it on. though it requires a pretty intimate knowledge of the importer itself, as various things (and core xenforo files) need modifications.

    I will be packaging it up with some information for the end user to use, and will probably be looking to sell it (unsupported) for around $150 I imagine as the time investment in it was pretty massive to be honest (much longer than I realised until I totaled it all up) or as a service on a quote-basis.
     
    Gladius likes this.
  20. Stuart Wright

    Stuart Wright Well-Known Member

    I'm happy to volunteer the AVForums database as a test subject provided, of course, we can ensure user data privacy.
     

Share This Page