S.A.F.A.R.I (Slavik's Amazingly Fast And Responsive Importer) [vB4] [Paid] [Deleted]

Status
Not open for further replies.
Slavik

Slavik

XenForo moderator
Staff member
Slavik submitted a new resource:

S.A.F.A.R.I (Slavik's Amazingly Fast And Responsive Importer) [vB4] (version 1.0) - A Generic VB4 Importer Based on Shawn Hogans A.R.F.I

(Currently unavailable pending final tests, however rather than clutter up Shawns thread, I will post this ready)

This is a modified version of the A.R.F.I importer by Shawn at DigitalPoint

This importer removes the DigitalPoint only aspects and updates code reliant on Shawns unique setup to allow for normal vBulletin 4 installs to migrate to XenForo in a much quicker time...
Click to expand...

Read more about this resource...
 
Test import of P8ntballer-Forums. 1.2M posts. Took several hours using web importer, excluding attachments, 5 minutes using CLI :)

As expected the longest times is on attachments. Large forums with a lot of attachments are simply going to have to suck that up as part of the process. However, the good news is, this can be staggered, so we can run the attachments part before live import, so the actual import may only need to add a few hundred attachments on the day with the bulk done beforehand (y)


Code: 
exporting posts...
          [forking process for posts]
          post IPs (1,183,726 records in 28.58s)...
          likes (0 records in 0.04s)...
          post edit history (71,037 records in 19.88s)...
          reported posts (0 records in 1.46s)...
          reported post comments (0 records in 0.95s)...
          reported profile posts (0 records in 0.17s)...
          reported profile post comments (0 records in 0.05s)...
          reported private messages (0 records in 0.03s)...
          reported private message comments (0 records in 0.04s)...
Total time for posts: 51.24s
 
exporting users...
          [forking process for users & user authenticate]
          user external auth (0 records in 0.09s)...
          user follow (22,295 records in 0.17s)...
          user ignored (166 records in 0.13s)...
          user notes (2 records in 0.04s)...
          user options (27,341 records in 0.13s)...
          user profile (27,341 records in 1.33s)...
          user privacy (27,341 records in 0.15s)...
          user warnings (1,725 records in 0.36s)...
          user bans (2,943 records in 0.25s)...
          warning action triggers (2,943 records in 0.13s)...
          skipping user fields (0 records in 0.02s)...
          user registration IPs (27,341 records in 0.33s)...
          user account confirmation IPs (1,646 records in 0.23s)...
          user change log (11,489 records in 0.22s)...
          user change log IPs (3,056 records in 0.16s)...
          profile posts (12,667 records in 0.20s)...
          profile post IPs (12,658 records in 0.18s)...
          user upgrades active (291 records in 0.08s)...
          user upgrades expired (487 records in 0.04s)...
          user upgrades log (883 records in 0.08s)...
Total time for users: 4.32s
*** Don't forget to rebuild user cache ***
 
exporting threads...
          [forking process for threads]
          watched threads (295,118 records in 0.55s)...
          thread user post (611,466 records in 1.88s)...
          polls (19,589 records in 0.74s)...
          thread read (0 records in 0.08s)...
Total time for threads: 3.26s
*** Don't forget to rebuild poll cache ***
 
exporting private messages...
          [forking process for conversation master]
          conversation message (182,233 records in 15.61s)...
          conversation recipient (288,037 records in 3.26s)...
          conversation user (199,111 records in 2.74s)...
          skipping IPs
Total time for private messages: 21.66s
 
performing avatar diff...
*snip* 27316 27317 27321 27338 27361 27404 27459 27467 27480 27510
Total time for avatars: 165.49s
**Don't forget to run the avatar cleanup scripts**
 
performing attachment conversion (1000 batches)...
        334 new attachments
        485 new attachments
        385 new attachments
        477 new attachments
        462 new attachments
        408 new attachments
        420 new attachments
        486 new attachments
        544 new attachments
        624 new attachments
        543 new attachments
        634 new attachments
        671 new attachments
        583 new attachments
        650 new attachments
        297 new attachments
        472 new attachments
        582 new attachments
        582 new attachments
        721 new attachments
        421 new attachments
        updating attachment view count
Total time for attachments: 646.80s
 
exporting forums...
          nodes (80 records in 0.31s)...
          forums (68 records in 0.13s)...
          link forums (1 records in 0.13s)...
          forum read (0 records in 0.21s)...
          watched forums (1,486 records in 0.16s)...
Total time for forums: 0.94s
 
exporting micellaneous stuff...
          deletion log (8,649 records in 0.23s)...
          moderator log (50,342 records in 0.91s)...
          daily stats (13,179 records in 4.49s)...
 
Total time for micellaneous stuff: 5.62s
          [waiting on process for users & user authenticate] DONE.
          [waiting on process for conversation master] DONE.
          [waiting on process for threads] DONE.
          [waiting on process for posts] DONE.
 
    Grand Total For Everything: 14 minutes, 59 seconds (4,724,703 records)
 
Would you like to test with a 16M of post forum? :)

I am importing (testing) my forum with the actual importing system available and it takes more than 4 hours to import 50% of private messages........
 
klyde said:
Would you like to test with a 16M of post forum? :)

I am importing (testing) my forum with the actual importing system available and it takes more than 4 hours to import 50% of private messages........
Click to expand...

PMed
 
Its great that we will see post edit history in 1.2
I hope this importer will also support import of post edit history.

Now hopefully we will see thread tags in 1.2 as well, so these can also be imported.
 
Alfa1 said:
Its great that we will see post edit history in 1.2
I hope this importer will also support import of post edit history.

Now hopefully we will see thread tags in 1.2 as well, so these can also be imported.
Click to expand...

Talking about what this importer will support and wont is premature. I doubt I will look to add any features in past what it already supports, unless a big board wanted to pay for that particular feature and then id integrate it for everyone.
 
A test import of AV Forums (~18M+ Posts), a few bugs to hammer out, but initial results are promising.

Code: 
exporting posts...
          [forking process for posts]
          post IPs (18,111,496 records in 609.45s)...
          likes (0 records in 0.22s)...
          post edit history (1,617,112 records in 1,937.58s)...
          reported posts (0 records in 199.25s)...
          reported post comments (0 records in 165.70s)...
          reported profile posts (0 records in 0.37s)...
          reported profile post comments (0 records in 0.08s)...
          reported private messages (0 records in 32.06s)...
          reported private message comments (0 records in 0.74s)...
Total time for posts: 2,945.54s
 
exporting users...
          [forking process for users & user authenticate]
          user external auth (0 records in 0.17s)...
          user follow (21,831 records in 0.23s)...
          user ignored (1,071 records in 2.73s)...
          user notes (15,532 records in 1.51s)...
          user options (321,850 records in 8.48s)...
          user profile (0 records in 0.05s)...
          user privacy (321,850 records in 0.92s)...
          user warnings (24,460 records in 270.18s)...
          user bans (0 records in 0.67s)...
          warning action triggers (0 records in 0.45s)...
          skipping user fields (0 records in 0.07s)...
          user registration IPs (321,850 records in 17.25s)...
          user account confirmation IPs (0 records in 2.46s)...
          user change log (0 records in 0.05s)...
          user change log IPs (0 records in 1.70s)...
          profile posts (32,141 records in 0.39s)...
          profile post IPs (32,141 records in 1.68s)...
          user upgrades active (0 records in 0.06s)...
          user upgrades expired (0 records in 0.04s)...
          user upgrades log (0 records in 0.07s)...
Total time for users: 309.16s
*** Don't forget to rebuild user cache ***
 
exporting threads...
          [forking process for threads]
          watched threads (2,107,481 records in 2.42s)...
          thread user post (7,122,379 records in 11.33s)...
          polls (514,585 records in 18.27s)...
          thread read (311,621 records in 1.52s)...
Total time for threads: 33.58s
*** Don't forget to rebuild poll cache ***
 
exporting private messages...
          conversation master (950,553 records in 25.32s)...
          conversation message (950,553 records in 103.26s)...
          conversation recipient (1,496,769 records in 20.87s)...
          conversation user (1,084,521 records in 19.94s)...
          skipping private conversation IPs
Total time for private messages: 169.40s
 
exporting forums...
          nodes (368 records in 0.31s)...
          forums (0 records in 0.04s)...
          link forums (19 records in 0.04s)...
          forum read (43,712 records in 0.11s)...
          watched forums (4,044 records in 0.07s)...
Total time for forums: 0.56s
 
exporting micellaneous stuff...
          deletion log (264,618 records in 2.67s)...
          moderator log (186,761 records in 23.39s)...
          daily stats (24,852 records in 0.88s)...
Total time for micellaneous stuff: 26.94s
Grand Total For Everything: 58 minutes, 08 seconds (35,562,499 records)
***Avatars and attachments skipped***
 
Today i tried to import my board with the actual importer for vb4.x and this is the result:

Imported 6,162,485 items. (24 hours 3 minutes 41.78 seconds) for Private Messages....​
:cry::cry::cry:​
 
klyde said:
Today i tried to import my board with the actual importer for vb4.x and this is the result:

Imported 6,162,485 items. (24 hours 3 minutes 41.78 seconds) for Private Messages....​
:cry::cry::cry:​
Click to expand...

Well comparing to this importer, it would take approximately 15 minutes to do that ;)
 
Aside from some cleanup changes, I am now confident this will work for most vBulletin 4 Boards as long as the technical requirements are met.

I will be posting a video up shortly demonstrating a full transfer.
 
Walter said:
Any chance for a 3.8 importer like this?
Click to expand...

This will do everything in 3.8 aside from attachments. (and maybe avatars).

If I were to update it to do vb 3.8 I would need someone to provide me a full working copy of their site to experiment on.
 
I've offered my full site for testing purposes (backup site) as this is what I'm holding off for converting my site.
If I need to purchase before hand again, let me know.

Let me know.
 
Status
Not open for further replies.
Back
Top Bottom