Import speed drops dramatically on Attachments

[Baron]

I'm migrating my fairly large forum to xenforo from a vb 3.8 database and everything goes fine until I get to the attachments import, then it's painfully slow. There are about 175,000 attachments in the database and the importer is processing those at about 20% per DAY! At this rate, it will take 5 days to import, which doesn't seem right. Perhaps my DB settings aren't optimized for handling these types of transactions?

My server is an Apple Xserve with 2 x 3.0 GHz Dual-Core Intel Xeon processors and 8GB of Ram

Attached is my MySQL variable output....

Any insight is greatly appreciated!

 

[Slavik]

That is a lot of attachments!

Potentially just a bottleneck on I/O resources.

How large is your vb3 database?

 

[Jake Bunce]

Are the vB3 attachments stored in the database or file system? I would hope they are in the file system.

Increasing the batch size might help:

http://xenforo.com/community/thread...port-be-can-we-speed-it-up.35324/#post-401122

There is also the command line importer which would be ideal, but it's not officially supported and people have reported problems with it.

And as Slavik said, I/O could be a problem here. Importing 175k attachments is going to involve a lot of I/O. If you can improve your disk performance or even install a SSD you could benefit greatly.

 

[Baron]

Are the vB3 attachments stored in the database or file system? I would hope they are in the file system.

They are in the database

This import is a test run so do you think it would be a better strategy to first have vb 3.8 convert the attachments to the filesystem and then do the xf import?

 

[Slavik]

They are in the database

This import is a test run so do you think it would be a better strategy to first have vb 3.8 convert the attachments to the filesystem and then do the xf import?

It would be worth a shot.

 

[Jake Bunce]

They are in the database

The problem is that in vB3 the actual file data was still stored in the attachment table instead of in a separate table = crappy performance when you have lots of attachments.

do you think it would be a better strategy to first have vb 3.8 convert the attachments to the filesystem and then do the xf import?

 

[Baron]

So just as a followup, I used VB's attachment manager to dump the attachments out of the database and into the filesystem. When I retried the attachment import into XF, my import time went from 5 days to an hour and a half!

 

[Slavik]

So just as a followup, I used VB's attachment manager to dump the attachments out of the database and into the filesystem. When I retried the attachment import into XF, my import time went from 5 days to an hour and a half!

Good to hear