converting a huge community to xenforo

[shock]

my friend and me(i was the CEO) are curious about how much it will take to convert a community with this stats to xen foro:
users: 747,653.
messages: 83,613,000.
threads: 9,054,146.
forums: 938.(including subforums)
Average users online: 23,000.

it may be possible that the owner of the forum will consider switching the forum software, because VB is realy heavy, slow, and causes alot of problems with the search... (when users search alot the forum crashes)

and we wanna know if xen foro 1.1 can handle this stats with minimum troubles.

 

[roldan_18]

I recommend use your localhost/PC to convert or import your huge board.

 

[DeltaHF]

I recommend use your localhost/PC to convert or import your huge board.

I wouldn't do that unless you've got an extremely fast local internet connection. Downloading and then re-uploading the converted database on most residential internet connections would add many hours, if not days, to the conversion process.

One of the best ways to speed up the conversion process of a large forum is to do it with a solid-state hard drive. In my testing, this dropped the conversion process of a 5.6-million vBulletin 3.8 forum from 3 days to 8 hours.

 

[roldan_18]

I wouldn't do that unless you've got an extremely fast local internet connection. Downloading and then re-uploading the converted database on most residential internet connections would add many hours, if not days, to the conversion process.

One of the best ways to speed up the conversion process of a large forum is to do it with a solid-state hard drive. In my testing, this dropped the conversion process of a 5.6-million vBulletin 3.8 forum from 3 days to 8 hours.

How much is your forum post,thread and members?

 

[DeltaHF]

201k topics, 5.6 million posts, and 196k users. The database is about 8GB in size. The live site isn't on XenForo yet, though, as I'm still evaluating the software.

 

[roldan_18]

Ow.,, I don't even try to handle that kind of Forum

 

[CTXMedia]

You need to install the addon, then select the appropriate importer (it has alpha in the title). It works fine as I helped test it. So far it only works under Linux.

Currently doing a test import using this importer and 1.1 b2 for a 4 million post forum.

Are you able to provide basic instructions on how to do this as I'd like to try a test import of 1.7 million posts from my IPBoard site.

Thanks,
Shaun

 

[MGSteve]

Using the importer I have imported 50,000 threads in around 30 minutes (cannot tell you the number of posts included in that as InnoDB does not return accurate row counts)

Run the query "SELECT count(*) FROM xf_user" to get the row count for the user table, just change xf_user to get the counts for the other tables.

 

[MGSteve]

btw, this is the stats from our latest import into 1.1, these were done on our live server via the browser, so it was a good test run.

What I need to find out is how to speed up the re-indexing, that took hours before.



The Threads & posts probably can be reduced by 2 hours or so as the browser crashed whilst it was importing them and I had to restart it. Still, was pleasantly surprised by the speed, all said.

I wish it would state the post count on the importer - it was around 4.1m posts.

 

[SheepCow]

Run the query "SELECT count(*) FROM xf_user" to get the row count for the user table, just change xf_user to get the counts for the other tables.

That won't work quickly/accurately with InnoDB

 

[MGSteve]

That won't work quickly/accurately with InnoDB

It is accurate, but can be slow on large datasets. There are theads on the 'net about faster ways using indexes, but I haven't encountered speed issues to require me to look into it yet.

 

[SheepCow]

It is accurate, but can be slow on large datasets. There are theads on the 'net about faster ways using indexes, but I haven't encountered speed issues to require me to look into it yet.

Sorry what I meant was "SELECT COUNT(*) FROM table" for InnoDB is slow, "SHOW TABLE STATUS LIKE 'table'" is fast but only a guess

 

[MGSteve]

Sorry what I meant was "SELECT COUNT(*) FROM table" for InnoDB is slow, "SHOW TABLE STATUS LIKE 'table'" is fast but only a guess

Yup, spot on. Although its not slow on xf_user on my db with 100K users. I'll give the post table a quick try...

Update: on a 4.6m post table it took about 10-15 secs.

 

[Deebs]

Are you able to provide basic instructions on how to do this as I'd like to try a test import of 1.7 million posts from my IPBoard site.

Thanks,
Shaun

Of course, my reply was incomplete, it "should" work under windows and other platforms but has specific linux code in there to allow the proper affinity settings for spreading the load over multiple cpu/cores for Linux. So to test follow these steps:

1. Import the addon.
2. Start the import as per normal but select the 3.7/3.8 Alpha importer.
3. When it comes to the thread (aka posts) import it will present a dialog box where you can customise some things. From my extensive testing I have found that a single PHP process cannot max out a core so I generally select 3 times my core count in the options (cannot remember the options for the life of me, sorry). I think I normally select (on a dual dual core test rig), 2 cpus, 2 cores. This will configure the system to create 4 PHP processes.
4. The Importer will then display a line of text which includes the path to your PHP process. Run it under a SCREEN. That way if your client crashes the import will still work.
5. Start the import with screen <line of text from above>.
6. Once it is finished you carry the export on as normal.

I did a test on my production server the other day, 4 million posts, 4 hours or so to import. On my new server the same test took 2 hours 22 minutes This was using 4 PHP processes.

Any problems give me a shout.

 

[Deebs]

btw, this is the stats from our latest import into 1.1, these were done on our live server via the browser, so it was a good test run.

What I need to find out is how to speed up the re-indexing, that took hours before.

View attachment 19838

The Threads & posts probably can be reduced by 2 hours or so as the browser crashed whilst it was importing them and I had to restart it. Still, was pleasantly surprised by the speed, all said.

I wish it would state the post count on the importer - it was around 4.1m posts.

I didn't use the browser based reindex for the search table, just the disable keys, import, and then enable keys. Took 15 mins to index 4 million posts.

 

[MGSteve]

I presume you need to disable keys on xf_search_index before you rebuild the search index?

 

[CTXMedia]

Thanks for the instructions Deebs - I'm guessing, from my test just now, that the CLI doesn't support IPB 3.1.4 yet?

 

[Deebs]

I presume you need to disable keys on xf_search_index before you rebuild the search index?

Yup.

Thanks for the instructions Deebs - I'm guessing, from my test just now, that the CLI doesn't support IPB 3.1.4 yet?

I think so far it is only for vBulletin (from memory).

Would just like to add that it works like a charm, everything is imported correctly from what I can see apart from thread prefixes.

 

[Kier]

FWIW, the CLI importer should not have actually been released - it still needs a lot of testing before I'd recommend that rely upon it.

 

[CTXMedia]

FWIW, the CLI importer should not have actually been released - it still needs a lot of testing before I'd recommend that rely upon it.

Right you are boss ...

Just out of interest, is there any way to speed up imports - such as disabling/removing certain indexes during the import phase and building them afterwards?
Cheers,
Shaun

 

[MRaburn]

I have a 4 million post forum I am ready to move. Once 1.1 comes out of BETA and this command line script is solid Ill be on the move!