1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

A.R.F.I (A Really Fast [vB4] Importer) [Paid] 1.0.0

The custom importer we built to migrate digitalpoint.com

  1. digitalpoint

    digitalpoint Well-Known Member

    digitalpoint submitted a new resource:

    A.R.F.I (A Really Fast [vB4] Importer) (version 1.0.0) - The custom importer we built to migrate digitalpoint.com

    Read more about this resource...
     
    euantor, Lisa, p4guru and 7 others like this.
  2. Adam Howard

    Adam Howard Well-Known Member

  3. Allan

    Allan Well-Known Member

    screens ?
     
  4. digitalpoint

    digitalpoint Well-Known Member

    It's a CLI based tool... the output is posted in the resource description to show you what it does.

    If you want screenshots of what it can do with vBulletin 4 data, take a look at this site. :) https://forums.digitalpoint.com/
     
    Lost and Shelley like this.
  5. Chris D

    Chris D XenForo Developer Staff Member

    Great work, digitalpoint. This will be well worth the money for big boards. Definitely wish I had got this when we did our import last year.

    Well done.
     
    Shelley likes this.
  6. Slavik

    Slavik XenForo Moderator Staff Member

    A heads up, getting this working will take some effort, however having had a play.

    Code:
    exporting posts...
              postsERROR 1054 (42S22) at line 1: Unknown column 'message_state' in 'field list'
    rm: cannot remove `/tmp/xf_post_*': No such file or directory
    wc: /tmp/xf_post.txt: No such file or directory
    (0 records in 0.09s)...
              post IPs (1,183,726 records in 16.59s)...
              likes (0 records in 0.04s)...
              post edit history (71,037 records in 16.16s)...
              reported posts (0 records in 0.93s)...
              reported post comments (0 records in 0.91s)...
              reported profile posts (0 records in 0.04s)...
              reported profile post comments (0 records in 0.05s)...
              reported private messages (0 records in 0.04s)...
              reported private message comments (0 records in 0.04s)...
    Total time for posts: 34.93s
    
    Thats pretty obscene.

    I'm going to go through this a section at a time and get it working for stock vBulletin boards (hopefully) and will pass on the changes to Shawn if he doesn't mind.
     
  7. bousaid

    bousaid Well-Known Member

    paid !! :(
     
    Adam Howard likes this.
  8. Slavik

    Slavik XenForo Moderator Staff Member

    Theres a free vb4 importer available ;)
     
    bousaid likes this.
  9. kkm323

    kkm323 Well-Known Member

    I am glad to see digital point making the move to xenforo :)
     
    bousaid likes this.
  10. Chris D

    Chris D XenForo Developer Staff Member

    As per his post the cost is for a month's paid membership at his site. I would guess that this allows you to download all of the other awesome stuff he has available.

    All seems pretty reasonable to me.

    Why shouldn't it be paid? A lot of hard work and effort has gone into it and the other premium content he has. And as Slavik says there's a free importer available which would fit most people's needs.
     
    Rudy, Shelley, bousaid and 1 other person like this.
  11. Digital Doctor

    Digital Doctor Well-Known Member

    $25 is essentially free. If your forum is big enough to need the importer ... you won't think $25 is a barrier.
     
  12. digitalpoint

    digitalpoint Well-Known Member

    To be honest the one fee to download anything (and lots of other site benefits) is more an arbitrary barrier just so I don't have to spend time trying to support people who don't actually *need* the stuff... Sphinx Search for vB4, Master/Slave DB support for XF, the really fast importer, etc... All stuff that small sites will never need, so trying to support a small site trying to use it for no reason other than *to* use it... Is a bit of a waste of everyone's time. That's the main reasoning about it.

    As far as if Slavik (or anyone else) forks the importer and makes a more generic one, I'm all for that... They can even sell it for their own profit if they want as long as they post it in our digital goods marketplace.

    The importer literally has hundreds of hours of work into it even in it's super raw form.
     
    zooki, HWS, AzzidReign and 8 others like this.
  13. SneakyDave

    SneakyDave Well-Known Member

    xF needs their entire import system CLI based, in my opinion. Great work!
     
  14. Slavik

    Slavik XenForo Moderator Staff Member

    I'll probably never meet the requirements to be able to post it there :p

    However, in terms of making it generic (and also more user friendly (however I found a catch which basically invalidated one of my changes, but meh, cba to edit it back)) ive gotten the posts and users done thus far.



    Code:
    exporting posts...
              post IPs (1,183,726 records in 16.62s)...
              likes (0 records in 0.04s)...
              post edit history (71,037 records in 16.14s)...
              reported posts (0 records in 0.88s)...
              reported post comments (0 records in 0.90s)...
              reported profile posts (0 records in 0.04s)...
              reported profile post comments (0 records in 0.04s)...
              reported private messages (0 records in 0.04s)...
              reported private message comments (0 records in 0.04s)...
    Total time for posts: 117.22s
    

    Code:
    exporting users...
              user & user authenticate (27,341 records in 6.85s)...
              user external auth (0 records in 0.34s)...
              user follow (22,295 records in 0.37s)...
              user ignored (166 records in 0.33s)...
              user notes (2 records in 0.31s)...
              user options (27,341 records in 0.43s)...
              user profile (27,341 records in 1.34s)...
              user privacy (27,341 records in 0.45s)...
              user warnings (1,725 records in 0.39s)...
              user bans (2,943 records in 0.49s)...
              warning action triggers (2,943 records in 0.42s)...
              skipping user fields (6,424 records in 0.16s)...
              user registration IPs (27,341 records in 0.72s)...
              user account confirmation IPs (1,646 records in 0.64s)...
              user change log (11,489 records in 0.68s)...
              user change log IPs (3,056 records in 0.47s)...
              profile posts (12,667 records in 0.63s)...
              profile post IPs (12,658 records in 0.50s)...
              user upgrades active (291 records in 0.33s)...
              user upgrades expired (487 records in 0.33s)...
              user upgrades log (883 records in 0.39s)...
    Total time for users: 16.58s
    *** Don't forget to rebuild user cache ***
    
     
  15. digitalpoint

    digitalpoint Well-Known Member

    Did you comment out the exporting of xf_post table? It's fairly important. :)
     
  16. Slavik

    Slavik XenForo Moderator Staff Member

    No... Why do you say that?

    I guess it may be because theres no postswc: /tmp/xf_post.txt output? But your importer breaks them down into xf_post_aa xf_post_ab etc... so there is actually no xf_post.txt by the time you go to count it because it's been removed I think.
     
  17. digitalpoint

    digitalpoint Well-Known Member

    Oh yeah, forgot I added the splitting late, and since I always did an "Everything" export the post output was never shown anyway since it was off in it's own forked thread.

    You could move this right below that exec (before the splitting) if you wanted the output:

    PHP:
    if ($this->pid['post'] !== 0)
    {
    echo 
    " (" number_format(intval(shell_exec('wc -l /tmp/xf_post.txt'))) . ' records in ' .  number_format(microtime(true) - $start2) . "s)...\r\n";
    }
     
  18. Slavik

    Slavik XenForo Moderator Staff Member

    Never thought of that, I just added a loop 5 times in to check if each individual file exists and then print the individual outputs...

    Code:
    exporting posts...
              posts
    showing first 5M output records...
     (1,000,000 records in 84.00s)...
     (183,726 records in 84.13s)...
              post IPs (1,183,726 records in 17.17s)...
              likes (0 records in 0.04s)...
              post edit history (71,037 records in 17.23s)...
              reported posts (0 records in 0.93s)...
              reported post comments (0 records in 0.90s)...
              reported profile posts (0 records in 0.04s)...
              reported profile post comments (0 records in 0.05s)...
              reported private messages (0 records in 0.04s)...
              reported private message comments (0 records in 0.04s)...
    Total time for posts: 120.59s
    
     
  19. digitalpoint

    digitalpoint Well-Known Member

    84 seconds for over a million posts... that's not bad, right? lol
     
  20. Slavik

    Slavik XenForo Moderator Staff Member

    Sucks, make it go faster :D
     
    SneakyDave, Alfa1 and Bob like this.

Share This Page