1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

S.A.F.A.R.I (Slavik's Amazingly Fast And Responsive Importer) [vB4] [Paid] [Deleted]

Discussion in 'Add-on Releases' started by Slavik, Mar 21, 2013.

Thread Status:
Not open for further replies.
  1. Slavik

    Slavik XenForo Moderator Staff Member

    Slavik submitted a new resource:

    S.A.F.A.R.I (Slavik's Amazingly Fast And Responsive Importer) [vB4] (version 1.0) - A Generic VB4 Importer Based on Shawn Hogans A.R.F.I

    Read more about this resource...
     
    Miner, Bob, Alfa1 and 1 other person like this.
  2. Slavik

    Slavik XenForo Moderator Staff Member

    Will be running some test shortly.
     
    ArnyVee likes this.
  3. Alfa1

    Alfa1 Well-Known Member

    Thanks Slavik!
     
  4. TheComputerGuy

    TheComputerGuy Active Member

    Does it work with vBulletin 3.8.x?
     
  5. Slavik

    Slavik XenForo Moderator Staff Member

    Untested at the moment
     
  6. Slavik

    Slavik XenForo Moderator Staff Member

    Test import of P8ntballer-Forums. 1.2M posts. Took several hours using web importer, excluding attachments, 5 minutes using CLI :)

    As expected the longest times is on attachments. Large forums with a lot of attachments are simply going to have to suck that up as part of the process. However, the good news is, this can be staggered, so we can run the attachments part before live import, so the actual import may only need to add a few hundred attachments on the day with the bulk done beforehand (y)


    Code:
    exporting posts...
              [forking process for posts]
              post IPs (1,183,726 records in 28.58s)...
              likes (0 records in 0.04s)...
              post edit history (71,037 records in 19.88s)...
              reported posts (0 records in 1.46s)...
              reported post comments (0 records in 0.95s)...
              reported profile posts (0 records in 0.17s)...
              reported profile post comments (0 records in 0.05s)...
              reported private messages (0 records in 0.03s)...
              reported private message comments (0 records in 0.04s)...
    Total time for posts: 51.24s
     
    exporting users...
              [forking process for users & user authenticate]
              user external auth (0 records in 0.09s)...
              user follow (22,295 records in 0.17s)...
              user ignored (166 records in 0.13s)...
              user notes (2 records in 0.04s)...
              user options (27,341 records in 0.13s)...
              user profile (27,341 records in 1.33s)...
              user privacy (27,341 records in 0.15s)...
              user warnings (1,725 records in 0.36s)...
              user bans (2,943 records in 0.25s)...
              warning action triggers (2,943 records in 0.13s)...
              skipping user fields (0 records in 0.02s)...
              user registration IPs (27,341 records in 0.33s)...
              user account confirmation IPs (1,646 records in 0.23s)...
              user change log (11,489 records in 0.22s)...
              user change log IPs (3,056 records in 0.16s)...
              profile posts (12,667 records in 0.20s)...
              profile post IPs (12,658 records in 0.18s)...
              user upgrades active (291 records in 0.08s)...
              user upgrades expired (487 records in 0.04s)...
              user upgrades log (883 records in 0.08s)...
    Total time for users: 4.32s
    *** Don't forget to rebuild user cache ***
     
    exporting threads...
              [forking process for threads]
              watched threads (295,118 records in 0.55s)...
              thread user post (611,466 records in 1.88s)...
              polls (19,589 records in 0.74s)...
              thread read (0 records in 0.08s)...
    Total time for threads: 3.26s
    *** Don't forget to rebuild poll cache ***
     
    exporting private messages...
              [forking process for conversation master]
              conversation message (182,233 records in 15.61s)...
              conversation recipient (288,037 records in 3.26s)...
              conversation user (199,111 records in 2.74s)...
              skipping IPs
    Total time for private messages: 21.66s
     
    performing avatar diff...
    *snip* 27316 27317 27321 27338 27361 27404 27459 27467 27480 27510
    Total time for avatars: 165.49s
    **Don't forget to run the avatar cleanup scripts**
     
    performing attachment conversion (1000 batches)...
            334 new attachments
            485 new attachments
            385 new attachments
            477 new attachments
            462 new attachments
            408 new attachments
            420 new attachments
            486 new attachments
            544 new attachments
            624 new attachments
            543 new attachments
            634 new attachments
            671 new attachments
            583 new attachments
            650 new attachments
            297 new attachments
            472 new attachments
            582 new attachments
            582 new attachments
            721 new attachments
            421 new attachments
            updating attachment view count
    Total time for attachments: 646.80s
     
    exporting forums...
              nodes (80 records in 0.31s)...
              forums (68 records in 0.13s)...
              link forums (1 records in 0.13s)...
              forum read (0 records in 0.21s)...
              watched forums (1,486 records in 0.16s)...
    Total time for forums: 0.94s
     
    exporting micellaneous stuff...
              deletion log (8,649 records in 0.23s)...
              moderator log (50,342 records in 0.91s)...
              daily stats (13,179 records in 4.49s)...
     
    Total time for micellaneous stuff: 5.62s
              [waiting on process for users & user authenticate] DONE.
              [waiting on process for conversation master] DONE.
              [waiting on process for threads] DONE.
              [waiting on process for posts] DONE.
     
        Grand Total For Everything: 14 minutes, 59 seconds (4,724,703 records)
     
    
     
  7. klyde

    klyde Member

    Would you like to test with a 16M of post forum? :)

    I am importing (testing) my forum with the actual importing system available and it takes more than 4 hours to import 50% of private messages........
     
    CyclingTribe likes this.
  8. Slavik

    Slavik XenForo Moderator Staff Member

    PMed
     
  9. Alfa1

    Alfa1 Well-Known Member

    Its great that we will see post edit history in 1.2
    I hope this importer will also support import of post edit history.

    Now hopefully we will see thread tags in 1.2 as well, so these can also be imported.
     
  10. Slavik

    Slavik XenForo Moderator Staff Member

    Talking about what this importer will support and wont is premature. I doubt I will look to add any features in past what it already supports, unless a big board wanted to pay for that particular feature and then id integrate it for everyone.
     
  11. Slavik

    Slavik XenForo Moderator Staff Member

    A test import of AV Forums (~18M+ Posts), a few bugs to hammer out, but initial results are promising.

    Code:
    exporting posts...
              [forking process for posts]
              post IPs (18,111,496 records in 609.45s)...
              likes (0 records in 0.22s)...
              post edit history (1,617,112 records in 1,937.58s)...
              reported posts (0 records in 199.25s)...
              reported post comments (0 records in 165.70s)...
              reported profile posts (0 records in 0.37s)...
              reported profile post comments (0 records in 0.08s)...
              reported private messages (0 records in 32.06s)...
              reported private message comments (0 records in 0.74s)...
    Total time for posts: 2,945.54s
     
    exporting users...
              [forking process for users & user authenticate]
              user external auth (0 records in 0.17s)...
              user follow (21,831 records in 0.23s)...
              user ignored (1,071 records in 2.73s)...
              user notes (15,532 records in 1.51s)...
              user options (321,850 records in 8.48s)...
              user profile (0 records in 0.05s)...
              user privacy (321,850 records in 0.92s)...
              user warnings (24,460 records in 270.18s)...
              user bans (0 records in 0.67s)...
              warning action triggers (0 records in 0.45s)...
              skipping user fields (0 records in 0.07s)...
              user registration IPs (321,850 records in 17.25s)...
              user account confirmation IPs (0 records in 2.46s)...
              user change log (0 records in 0.05s)...
              user change log IPs (0 records in 1.70s)...
              profile posts (32,141 records in 0.39s)...
              profile post IPs (32,141 records in 1.68s)...
              user upgrades active (0 records in 0.06s)...
              user upgrades expired (0 records in 0.04s)...
              user upgrades log (0 records in 0.07s)...
    Total time for users: 309.16s
    *** Don't forget to rebuild user cache ***
     
    exporting threads...
              [forking process for threads]
              watched threads (2,107,481 records in 2.42s)...
              thread user post (7,122,379 records in 11.33s)...
              polls (514,585 records in 18.27s)...
              thread read (311,621 records in 1.52s)...
    Total time for threads: 33.58s
    *** Don't forget to rebuild poll cache ***
     
    exporting private messages...
              conversation master (950,553 records in 25.32s)...
              conversation message (950,553 records in 103.26s)...
              conversation recipient (1,496,769 records in 20.87s)...
              conversation user (1,084,521 records in 19.94s)...
              skipping private conversation IPs
    Total time for private messages: 169.40s
     
    exporting forums...
              nodes (368 records in 0.31s)...
              forums (0 records in 0.04s)...
              link forums (19 records in 0.04s)...
              forum read (43,712 records in 0.11s)...
              watched forums (4,044 records in 0.07s)...
    Total time for forums: 0.56s
     
    exporting micellaneous stuff...
              deletion log (264,618 records in 2.67s)...
              moderator log (186,761 records in 23.39s)...
              daily stats (24,852 records in 0.88s)...
    Total time for micellaneous stuff: 26.94s
    Grand Total For Everything: 58 minutes, 08 seconds (35,562,499 records)
    ***Avatars and attachments skipped***
     
    
     
  12. klyde

    klyde Member

    Today i tried to import my board with the actual importer for vb4.x and this is the result:

    Imported 6,162,485 items. (24 hours 3 minutes 41.78 seconds) for Private Messages....
    :cry::cry::cry:
     
  13. Slavik

    Slavik XenForo Moderator Staff Member

    Well comparing to this importer, it would take approximately 15 minutes to do that ;)
     
  14. TheComputerGuy

    TheComputerGuy Active Member

    AV Forums is a 3.8 forum..so that makes me hopeful!
     
    Thomas likes this.
  15. Slavik

    Slavik XenForo Moderator Staff Member

    This is now working with VB4 attachments in both the filesystem and the database :)
     

    Attached Files:

    Thomas, klyde and TheComputerGuy like this.
  16. Slavik

    Slavik XenForo Moderator Staff Member

  17. Slavik

    Slavik XenForo Moderator Staff Member

    Aside from some cleanup changes, I am now confident this will work for most vBulletin 4 Boards as long as the technical requirements are met.

    I will be posting a video up shortly demonstrating a full transfer.
     
    Alfa1 likes this.
  18. Walter

    Walter Well-Known Member

    Any chance for a 3.8 importer like this?
     
  19. Slavik

    Slavik XenForo Moderator Staff Member

    This will do everything in 3.8 aside from attachments. (and maybe avatars).

    If I were to update it to do vb 3.8 I would need someone to provide me a full working copy of their site to experiment on.
     
  20. TheComputerGuy

    TheComputerGuy Active Member

    I've offered my full site for testing purposes (backup site) as this is what I'm holding off for converting my site.
    If I need to purchase before hand again, let me know.

    Let me know.
     
Thread Status:
Not open for further replies.

Share This Page