Migrating a large forum to XenForo (test migration)

[Deebs]

Wow ... nice work, well done.

Good luck with the move.

Thanks, it's been a long hard wait but now the time is near where I migrate from an old legacy system to a state of the art system. Reminds me of my day job. User feedback has been positive and so many of them have just said "Just do it already". I think I will be offline for around 4 hours the most but have given myself 5 hours to do it in, I also have a nice holding page so be sure to check back here on Friday for a link to it

 

[Andy.N]

Good luck, Deebs
Seeing you have plenty of contingency plan in place, I'm sure it will be a smooth ride and your members will love it.

 

[Deebs]

Good luck, Deebs
Seeing you have plenty of contingency plan in place, I'm sure it will be a smooth ride and your members will love it.

I certainly do. Lots of testing and giving my members exposure to XenForo (since April or so this year). They all know it is coming and have been instrumental in the tuning of the migration.

 

[MGSteve]

Where did you get the Multi-threaded version from?

edit - ignore me, I hadn't spotted that you needed to install the CLI importer as an addon. I need a coffee!

 

[MGSteve]

Well, the CLI importer does make a hell of a difference, this is a grab from aTop for those who understand these things! lol

ATOP - vorlon            2011/10/19  00:38:55                3 seconds elapsed
PRC | sys  1.87s | user  9.91s | #proc    328 | #zombie    0 | #exit      2 |
CPU | sys    62% | user    330% | irq      1% | idle  1150% | wait    62% |
cpu | sys      5% | user    26% | irq      0% | idle    69% | cpu014 w  0% |
cpu | sys      5% | user    25% | irq      0% | idle    69% | cpu012 w  0% |
cpu | sys      3% | user    25% | irq      0% | idle    67% | cpu002 w  5% |
cpu | sys      4% | user    24% | irq      0% | idle    72% | cpu010 w  0% |
cpu | sys      3% | user    23% | irq      0% | idle    73% | cpu000 w  0% |
cpu | sys      5% | user    21% | irq      0% | idle    74% | cpu008 w  0% |
cpu | sys      5% | user    21% | irq      0% | idle    74% | cpu015 w  0% |
cpu | sys      4% | user    21% | irq      0% | idle    61% | cpu011 w 13% |
cpu | sys      5% | user    20% | irq      0% | idle    75% | cpu013 w  0% |
cpu | sys      4% | user    20% | irq      0% | idle    76% | cpu009 w  0% |
cpu | sys      3% | user    19% | irq      0% | idle    78% | cpu004 w  0% |
cpu | sys      4% | user    17% | irq      0% | idle    79% | cpu006 w  0% |
cpu | sys      3% | user    17% | irq      1% | idle    38% | cpu003 w 42% |
cpu | sys      2% | user    18% | irq      0% | idle    80% | cpu001 w  0% |
cpu | sys      3% | user    17% | irq      0% | idle    80% | cpu005 w  0% |
cpu | sys      3% | user    16% | irq      0% | idle    81% | cpu007 w  0% |
CPL | avg1  9.18 | avg5    6.34 | avg15  3.56 | csw  247109 | intr    4682 |
MEM | tot  23.5G | free    4.3G | cache  9.3G | buff  705.9M | slab  777.5M |
SWP | tot    1.0G | free    1.0G |              | vmcom  10.7G | vmlim  12.8G |
DSK |  cciss/c0d1 | busy    60% | read      1 | write  1131 | avio    1 ms |
DSK |  cciss/c0d0 | busy      2% | read      0 | write    12 | avio    4 ms |
NET | transport  | tcpi    1069 | tcpo    955 | udpi      0 | udpo      0 |
NET | network    | ipi    1076 | ipo      955 | ipfrw      0 | deliv  1074 |
NET | eth0    1% | pcki    462 | pcko    587 | si  115 Kbps | so 1738 Kbps |
NET | lo    ---- | pcki    674 | pcko    674 | si 1209 Kbps | so 1209 Kbps |

  PID  SYSCPU  USRCPU  VGROW  RGROW  RDDSK  WRDSK  ST EXC S  CPU CMD    1/3
24880  1.43s  4.08s 16384K  8772K    4K 27032K  --  - S 184% mysqld
32347  0.02s  0.28s    0K    16K    0K    0K  --  - S  10% php
32351  0.03s  0.27s    0K    0K    0K    0K  --  - S  10% php
32361  0.02s  0.27s    0K    8K    0K    0K  --  - S  10% php
32357  0.03s  0.26s    0K    20K    0K    0K  --  - S  10% php
32349  0.03s  0.25s    0K    64K    0K    0K  --  - S  9% php
32355  0.02s  0.25s    0K    0K    0K    0K  --  - S  9% php
32371  0.02s  0.25s    0K    0K    0K    0K  --  - S  9% php
32365  0.02s  0.25s    0K    28K    0K    0K  --  - S  9% php
32375  0.03s  0.24s    0K    60K    0K    0K  --  - S  9% php
32377  0.02s  0.24s    0K    16K    0K    0K  --  - S  9% php
32367  0.01s  0.25s    0K    28K    0K    0K  --  - S  9% php
32363  0.02s  0.24s    0K    12K    0K    0K  --  - S  9% php
32359  0.03s  0.22s    0K    0K    0K    0K  --  - S  8% php
32369  0.02s  0.23s    0K    0K    0K    0K  --  - S  8% php
32353  0.01s  0.24s    0K    16K    0K    0K  --  - S  8% php
32373  0.01s  0.24s    0K    20K    0K    0K  --  - S  8% php
32591  0.01s  0.20s    0K    64K    0K    0K  --  - S  7% httpd

The main bit is the amount of data written by mySQL - 27032K - that's the amount of data written to the database in the last 3 seconds, that's getting on for 10mb a sec. its currently doing about 50,000 threads a minute which is quite a bit faster than the browser. Impressive

 

[CTXMedia]

Dang ... wish it worked with IPB

 

[MGSteve]

Dang ... wish it worked with IPB

Well, bearing in mind this is in alpha at the moment, I'm sure the process isn't too dissimilar with IPB, perhaps an enterprising developer will adapt it to work with IPB.

If you have any anti virus on the server as well, I'd disable it - I've just done that with mcaffee that came with the server and the disk write rates have just shot up by 20%!

 

[MGSteve]

its currently doing about 50,000 threads a minute which is quite a bit faster than the browser. Impressive

hmm, a tad optimistic there, its actually doing around 3-4,000 a minute, I may have an a zero violation in the original post

 

[Deebs]

FreddysHouse is currently migrating to XenForo

http://forums.freddyshouse.com (love the page )

 

[principia]

FreddysHouse is currently migrating to XenForo

http://forums.freddyshouse.com (love the page )

good luck and have fun with xf

 

[Deebs]

Thanks I am under pressure due to the countdown....

 

[Andy.N]

FreddysHouse is currently migrating to XenForo

http://forums.freddyshouse.com (love the page )

Is that your site?
Love the image. VB house is broken. XF house is spanking new.

 

[Deebs]

Yes, it is my site.

 

[Andy.N]

Thanks I am under pressure due to the countdown....

What steps are you in the process? How many percentage done so far?

 

[Deebs]

Currently doing the posts

 

[MGSteve]

Thanks I am under pressure due to the countdown....

Yeah, nice way to put pressure on yourself! lol

 

[MGSteve]

"12 processes across 4 CPUs" - I thought you were supposed to do one per core?

 

[Deebs]

Nah, 1 per core didnt max out the CPU.

 

[MGSteve]

I did notice that, I'll have to do another test run through, and add the results to my cli vs cli & ramdrive stats

 

[MGSteve]

Well, the CLI importer does make a hell of a difference, this is a grab from aTop for those who understand these things! lol

Interestingly the cpu usage when using a ram drive to buffer the output from the threads/post converter was double what it was in InnoDB mode.