XF 1.5 Import Very Slow?

[Tibit]

On latest Xenforo, importing from latest 3.4 series of IPB. Imported about 22k users, 2m posts, and 90k threads. So far it's been almost 20 hours and is on 91% of the posts/threads import. Should it take this long?

I have used import tools from other forum platforms with similar amounts, never takes more than a few (3-4) hours at most. Importing from a Xenforo installation of a similar size to IPB 3.4 was quicker than even that.

I know it's not my servers resources as as even when I boosted the rate limit to 9999 threads (from 100) and 99999 posts (from 800) it's still using less than 10% CPU, all other resources are under similar usage, nothing near maxed out. However, even with the rate limit increase to that (I even double checked by closing the import page, then re-continuing it) it doesn't seem to go that fast, it's only increasing 0.03% - 0.06% every 10-30 seconds. Is it not able to handle that much at a time or is something else wrong?

Anyway to speed this up in the future? It recommends you close the board for the day when doing this, keeping it down for a whole day is really bad, especially for larger boards.

Does completing the import take awhile either? If so, anyway to speed that up? (if it updates the index, cache, etc.) Or can I at least reopen the site while it re-indexes the new content?

 

[Paul B]

The bottleneck could be the MySQL read/writes.
Other than tuning the server, there's not much else which can be done to try and speed it up.

Or can I at least reopen the site while it re-indexes the new content?

Yes, you can open the site while the caches are being rebuilt, but it may cause some weirdness with missing content (searches), incorrect counts, etc.
My advice would be to keep it closed until it is complete - the rebuilds shouldn't take that long.

 

[Tibit]

Yes, you can open the site while the caches are being rebuilt, but it may cause some weirdness with missing content (searches), incorrect counts, etc.
My advice would be to keep it closed until it is complete - the rebuilds shouldn't take that long.

Is the rebuild when clicking "complete import" the only rebuild I need to do? Or do I need to rebuild the cache or any further action after that as completed? It's going through rebuilding the users right now.

When it starts rebuilding the threads and posts, anyway to speed that up?

The bottleneck could be the MySQL read/writes.
Other than tuning the server, there's not much else which can be done to try and speed it up.

Even those aren't close to maxed out. Only thing I'd care to speed up now are any processes that remain after the users are rebuilt once clicking "complete import."

 

[Paul B]

You will need to rebuild the search index as the final step.

With 2 million posts, you may want to consider the Enhanced Search add-on as MySQL typically has a limit of around 1 million.

 

[Tibit]

With 2 million posts, you may want to consider the Enhanced Search add-on as MySQL typically has a limit of around 1 million.

I'll install that late at night (EST), would that require closing the board for any length of time greater than 20-30 minutes for a forum of 3m posts, 110k threads, 30k users?

You will need to rebuild the search index as the final step.

Any idea how long it'll take to complete that process? Only through rebuilding 9k users in the last 7-9 minutes.

 

[Paul B]

It's impossible to give an estimate - it varies depending on site/server.

 

[Tibit]

It's impossible to give an estimate - it varies depending on site/server.

Based off what I've said so far does it take about the same amount of time as it did to import the content? Or does this part move faster than importing content?

And is there anyway to speed up the process? (any code based rate limits like the import tool had that can be increased?)

 

[andybond]

I dont know the exact layout of your boards , but could I make a suggestion ?

Create a new category but dont let it be visable in the node list.

Keep your board open .

Import the erm ,import into the new catergory. The content should not be visible as its parent is not shown in the node list.

Wait.

Once complete , move as needed , or allow the viability in the node list.