Import & Rebuild Caches Speed

[Paul M]

Just for information really, as Ive been hammering the vb4 importer by testing it with my Live vb tables.

On the whole, it seems a bit slow. For small or medium sites (a few thousand users, a few hundred thousand posts, its probably not much of an issue. Maybe a few hours at best.

However, even taking into account a few delays not directly related to the import, its taken over a day for me to import about 75,000 users and 1.7 million posts, plus PMs and other stuff.

The Cache Rebuild is also going really slow - at its current rate, its going to take almost 4 hours just to do the users - and then its got forums, threads and whatever else it does.

Based on this you are looking at a minimum of a few days to migrate a large site. Am I doing something wrong, or is this typical ?

 

[Rjs37]

The importing has speeded up dramatically in Beta 2 (from my experiences) though Threads/Posts still took over 2 hours (1.2 million posts), and Users (32,000) an hour.

I've only just managed to complete my import for the first time yesterday (due to character issues and such) so I'm not sure how speedy the Caching was in Beta 1 but it is indeed incredibly slow. It won't currently get further than 1000 for me (reported in bugs) but even from that 1000 I can see that the caching would take a long time.

 

[Paul M]

It finished the users in just over 3 hours, doing threads now - going to be another hour or more (112,000).

 

[Mike]

An hour to rebuild that many threads isn't really that much - that's over 30 threads per second. I will look into the user rebuild speed issues to see what's happening there.

 

[Paul M]

Just timed it, and its doing about 20 a second, so not too bad I guess. When I timeed the users it was about 6 a second, it did seem to speed up slightly towards the end.

 

[puertoblack2003]

Just for information really, as Ive been hammering the vb4 importer by testing it with my Live vb tables.

On the whole, it seems a bit slow. For small or medium sites (a few thousand users, a few hundred thousand posts, its probably not much of an issue. Maybe a few hours at best.

However, even taking into account a few delays not directly related to the import, its taken over a day for me to import about 75,000 users and 1.7 million posts, plus PMs and other stuff.

The Cache Rebuild is also going really slow - at its current rate, its going to take almost 4 hours just to do the users - and then its got forums, threads and whatever else it does.

Based on this you are looking at a minimum of a few days to migrate a large site. Am I doing something wrong, or is this typical ?

i went through it myself on a test server.Not happy with the amount of time it takes to migrate everything over to xen.It took me 1 day and a 1/2.With almost 89,000 members 76,000 threads and 311,000 post.I thought it was a server config issue, so i test few and still the same.At least I'm not the only one that went through it.. ;0)