A.R.F.I (A Really Fast [vB4] Importer) [Paid] 1.0.0

[digitalpoint]

digitalpoint submitted a new resource:

A.R.F.I (A Really Fast [vB4] Importer) (version 1.0.0) - The custom importer we built to migrate digitalpoint.com

This is the CLI-based importer that we custom built to retain all primary keys and import EVERYTHING from vBulletin 4 to XenForo 1.1.3. It will absolutely NOT work out of the box for any site other than ours, and will require programming work on your end to customize it for your site.

That being said, we were able to do a complete import with over 18M posts (75M+ total records) in less than an hour.

The "$25 cost" is actually the maximum cost for 1 month of premium membership on our site....

Read more about this resource...

 

[Adam Howard]

 

[Allan]

screens ?

 

[digitalpoint]

It's a CLI based tool... the output is posted in the resource description to show you what it does.

If you want screenshots of what it can do with vBulletin 4 data, take a look at this site. https://forums.digitalpoint.com/

 

[Chris D]

Great work, digitalpoint. This will be well worth the money for big boards. Definitely wish I had got this when we did our import last year.

Well done.

 

[Slavik]

A heads up, getting this working will take some effort, however having had a play.

exporting posts...
          postsERROR 1054 (42S22) at line 1: Unknown column 'message_state' in 'field list'
rm: cannot remove `/tmp/xf_post_*': No such file or directory
wc: /tmp/xf_post.txt: No such file or directory
(0 records in 0.09s)...
          post IPs (1,183,726 records in 16.59s)...
          likes (0 records in 0.04s)...
          post edit history (71,037 records in 16.16s)...
          reported posts (0 records in 0.93s)...
          reported post comments (0 records in 0.91s)...
          reported profile posts (0 records in 0.04s)...
          reported profile post comments (0 records in 0.05s)...
          reported private messages (0 records in 0.04s)...
          reported private message comments (0 records in 0.04s)...
Total time for posts: 34.93s

Thats pretty obscene.

I'm going to go through this a section at a time and get it working for stock vBulletin boards (hopefully) and will pass on the changes to Shawn if he doesn't mind.

 

[bousaid]

paid !!

 

[Slavik]

paid !!

Theres a free vb4 importer available

 

[kkm323]

I am glad to see digital point making the move to xenforo

 

[Chris D]

paid !!

As per his post the cost is for a month's paid membership at his site. I would guess that this allows you to download all of the other awesome stuff he has available.

All seems pretty reasonable to me.

Why shouldn't it be paid? A lot of hard work and effort has gone into it and the other premium content he has. And as Slavik says there's a free importer available which would fit most people's needs.

 

[Digital Doctor]

$25 is essentially free. If your forum is big enough to need the importer ... you won't think $25 is a barrier.

 

[digitalpoint]

To be honest the one fee to download anything (and lots of other site benefits) is more an arbitrary barrier just so I don't have to spend time trying to support people who don't actually *need* the stuff... Sphinx Search for vB4, Master/Slave DB support for XF, the really fast importer, etc... All stuff that small sites will never need, so trying to support a small site trying to use it for no reason other than *to* use it... Is a bit of a waste of everyone's time. That's the main reasoning about it.

As far as if Slavik (or anyone else) forks the importer and makes a more generic one, I'm all for that... They can even sell it for their own profit if they want as long as they post it in our digital goods marketplace.

The importer literally has hundreds of hours of work into it even in it's super raw form.

 

[SneakyDave]

xF needs their entire import system CLI based, in my opinion. Great work!

 

[Slavik]

As far as if Slavik (or anyone else) forks the importer and makes a more generic one, I'm all for that... They can even sell it for their own profit if they want as long as they post it in our digital goods marketplace.

I'll probably never meet the requirements to be able to post it there

However, in terms of making it generic (and also more user friendly (however I found a catch which basically invalidated one of my changes, but meh, cba to edit it back)) ive gotten the posts and users done thus far.

exporting posts...
          post IPs (1,183,726 records in 16.62s)...
          likes (0 records in 0.04s)...
          post edit history (71,037 records in 16.14s)...
          reported posts (0 records in 0.88s)...
          reported post comments (0 records in 0.90s)...
          reported profile posts (0 records in 0.04s)...
          reported profile post comments (0 records in 0.04s)...
          reported private messages (0 records in 0.04s)...
          reported private message comments (0 records in 0.04s)...
Total time for posts: 117.22s

exporting users...
          user & user authenticate (27,341 records in 6.85s)...
          user external auth (0 records in 0.34s)...
          user follow (22,295 records in 0.37s)...
          user ignored (166 records in 0.33s)...
          user notes (2 records in 0.31s)...
          user options (27,341 records in 0.43s)...
          user profile (27,341 records in 1.34s)...
          user privacy (27,341 records in 0.45s)...
          user warnings (1,725 records in 0.39s)...
          user bans (2,943 records in 0.49s)...
          warning action triggers (2,943 records in 0.42s)...
          skipping user fields (6,424 records in 0.16s)...
          user registration IPs (27,341 records in 0.72s)...
          user account confirmation IPs (1,646 records in 0.64s)...
          user change log (11,489 records in 0.68s)...
          user change log IPs (3,056 records in 0.47s)...
          profile posts (12,667 records in 0.63s)...
          profile post IPs (12,658 records in 0.50s)...
          user upgrades active (291 records in 0.33s)...
          user upgrades expired (487 records in 0.33s)...
          user upgrades log (883 records in 0.39s)...
Total time for users: 16.58s
*** Don't forget to rebuild user cache ***

 

[digitalpoint]

Did you comment out the exporting of xf_post table? It's fairly important.

 

[Slavik]

Did you comment out the exporting of xf_post table? It's fairly important.

No... Why do you say that?

I guess it may be because theres no postswc: /tmp/xf_post.txt output? But your importer breaks them down into xf_post_aa xf_post_ab etc... so there is actually no xf_post.txt by the time you go to count it because it's been removed I think.

 

[digitalpoint]

Oh yeah, forgot I added the splitting late, and since I always did an "Everything" export the post output was never shown anyway since it was off in it's own forked thread.

You could move this right below that exec (before the splitting) if you wanted the output:

if ($this->pid['post'] !== 0)
{
echo " (" . number_format(intval(shell_exec('wc -l /tmp/xf_post.txt'))) . ' records in ' .  number_format(microtime(true) - $start, 2) . "s)...\r\n";
}

 

[Slavik]

Oh yeah, forgot I added the splitting late, and since I always did an "Everything" export the post output was never shown anyway since it was off in it's own forked thread.

You could move this right below that exec (before the splitting) if you wanted the output:

Never thought of that, I just added a loop 5 times in to check if each individual file exists and then print the individual outputs...

exporting posts...
          posts
showing first 5M output records...
 (1,000,000 records in 84.00s)...
 (183,726 records in 84.13s)...
          post IPs (1,183,726 records in 17.17s)...
          likes (0 records in 0.04s)...
          post edit history (71,037 records in 17.23s)...
          reported posts (0 records in 0.93s)...
          reported post comments (0 records in 0.90s)...
          reported profile posts (0 records in 0.04s)...
          reported profile post comments (0 records in 0.05s)...
          reported private messages (0 records in 0.04s)...
          reported private message comments (0 records in 0.04s)...
Total time for posts: 120.59s

 

[digitalpoint]

84 seconds for over a million posts... that's not bad, right? lol

 

[Slavik]

84 seconds for over a million posts... that's not bad, right? lol

Sucks, make it go faster