Import speed drops dramatically on Attachments

Baron

Member
I'm migrating my fairly large forum to xenforo from a vb 3.8 database and everything goes fine until I get to the attachments import, then it's painfully slow. There are about 175,000 attachments in the database and the importer is processing those at about 20% per DAY! At this rate, it will take 5 days to import, which doesn't seem right. Perhaps my DB settings aren't optimized for handling these types of transactions?

My server is an Apple Xserve with 2 x 3.0 GHz Dual-Core Intel Xeon processors and 8GB of Ram

Attached is my MySQL variable output....

Any insight is greatly appreciated!
 

Attachments

Slavik

XenForo moderator
Staff member
That is a lot of attachments!

Potentially just a bottleneck on I/O resources.

How large is your vb3 database?
 

Jake Bunce

XenForo moderator
Staff member
Are the vB3 attachments stored in the database or file system? I would hope they are in the file system.

Increasing the batch size might help:

http://xenforo.com/community/threads/how-fast-should-a-vb3-import-be-can-we-speed-it-up.35324/#post-401122

There is also the command line importer which would be ideal, but it's not officially supported and people have reported problems with it.

And as Slavik said, I/O could be a problem here. Importing 175k attachments is going to involve a lot of I/O. If you can improve your disk performance or even install a SSD you could benefit greatly.
 

Baron

Member
Are the vB3 attachments stored in the database or file system? I would hope they are in the file system.
They are in the database :eek:

This import is a test run so do you think it would be a better strategy to first have vb 3.8 convert the attachments to the filesystem and then do the xf import?
 

Slavik

XenForo moderator
Staff member
They are in the database :eek:

This import is a test run so do you think it would be a better strategy to first have vb 3.8 convert the attachments to the filesystem and then do the xf import?
It would be worth a shot.
 

Jake Bunce

XenForo moderator
Staff member
They are in the database :eek:
The problem is that in vB3 the actual file data was still stored in the attachment table instead of in a separate table = crappy performance when you have lots of attachments.

do you think it would be a better strategy to first have vb 3.8 convert the attachments to the filesystem and then do the xf import?
(y)
 

Baron

Member
So just as a followup, I used VB's attachment manager to dump the attachments out of the database and into the filesystem. When I retried the attachment import into XF, my import time went from 5 days to an hour and a half! :D
 

Slavik

XenForo moderator
Staff member
So just as a followup, I used VB's attachment manager to dump the attachments out of the database and into the filesystem. When I retried the attachment import into XF, my import time went from 5 days to an hour and a half! :D
Good to hear :)
 
Top