XF 1.5 Attachments import failed

[dougdirac]

I'm doing a test migration from vB 4.2.3 to XF 1.5.

Everything seems to go well with no error messages. However, the vast majority of the attachments don't get imported.

In my vB attachments directory there are number subdirectories from "1" to say "39448". Each with a few numbered files with extensions .thumb and .attach. I should have thousands if not 10s of thousands of attachments on my 1M post forum.

When I selected import attachments, process only took about 15 minutes during which I could watch the doneness percentage climb steadily from 0 to 100%.

In my new XenForo attachments directory there is only one subdirectory and it's labeled "0". In that directory there are about 250 small jpg's. The images are all thumbnail sized.

So why didn't it process my attachments? Any suggestions?

 

[Mike]

In my vB attachments directory there are number subdirectories from "1" to say "39448".

Hmm, if I interpreted that correctly, I don't think that's the format our code expects. I think the files should be in directories like 3/9/4/4/8.

If that's the case, the easiest thing may actually be to move the attachments back into the DB and let XF process them from there.

 

[dougdirac]

Hmm, if I interpreted that correctly, I don't think that's the format our code expects. I think the files should be in directories like 3/9/4/4/8.

If that's the case, the easiest thing may actually be to move the attachments back into the DB and let XF process them from there.

Move 50 gigs to the database?

What's the scheme behind the directory structure for attachments in XenForo (and vB since I don't understand that either)?

Thanks!

 

[Mike]

Move 50 gigs to the database?

I made that suggestion more on the basis that you mentioned only having thousands of attachments, so I didn't expect it to be nearly that significant. (We have significantly more attachments here and much less usage, though I wouldn't want to shift ours into the DB.)

What's the scheme behind the directory structure for attachments in XenForo (and vB since I don't understand that either)?

The structure in XF isn't overly relevant here because the issue seems like it's down to the files not being found from the source DB. But XF puts files into directories based on IDs in blocks of 1000 (eg, ID 7565 goes into directory 7).

The source expects files like this:

$attachFileOrig = "$options[path]/" . implode('/', str_split($attachment['filedata_userid'])) . "/$attachment[filedataid].attach";

So the user ID divided by digit as the folder name and then the ID of the data (/path/to/attachments/1/2/3/4/5/6789.attach).

It sounds like your files aren't using that structure, though I'm not sure why. There may have been a different (deprecated?) format that was used for a period of time and only supported as a legacy version? Unfortunately, our code wouldn't support reading from that format. That's roughly why I suggested moving them back into the DB (you could move them back to the FS again), as we don't have support for the file structure that you appear to be using. (I may have seen this once(?) before, but most imports seem to bring the files over without issue so I suspect it's rare at best.)

 

[dougdirac]

Yeah, it's weird. I see evidence of that nested structure on the low numbers. But it seems mainly that the directory is the user ID itself, rather than coded into subdirectories.

Playing around with a test site that is from a backup from over a year ago I see this:

Statistics
Unique / Total Attachments 56,847 / 52,598
Attachment File-size Sum 30.95 GB
Disk Space Used 28.35 GB
Storage Type Attachments are currently being stored in the filesystem at "[path]"/attachments
Average Attachment File-size 570.8 KB

So I'm testing out the move to the database, but it's not promising. It only took a few minutes to run and I get this result before finalizing:

Confirm Attachment Removal
We have completed moving attachments to the filesystem. Presently, the attachments still remain in the database and are being read from there. If you are sure that the process completed without incident, go ahead and select yes below to remove the attachments from the database. This step cannot be reversed and may take a few minutes depending on the size of your database. Please be patient.

Attachments in Database: 52598
Total Attachments Processed: 52598

The above two values should be the same!

Number of Attachments that could not be created or are 0 bytes: 52376
The above value should be 0! Having 0 byte attachments is not uncommon so a few is nothing to worry about but if the number equals the number of attachments in the database then DO NOT FINALIZE this step or you will lose all of your attachments.
Finalize
Yes No

Not a good sign, but I'll click yes and see what happens.

 

[dougdirac]

I'm not sure why the above says, "moved to the file system" since the option I selected and the only one available said move to the database.

 

[dougdirac]

Well, as expected I suppose, that process killed my attachments. I mean I still see the files in the file system, but the attached images in threads are all gone. So how to proceed?

 

[Slavik]

Well, as expected I suppose, that process killed my attachments. I mean I still see the files in the file system, but the attached images in threads are all gone. So how to proceed?

As per the converter warning, you have probably just wiped out all bar 200 attachments from your database.

You would need to restore from a backup.

 

[dougdirac]

As per the converter warning, you have probably just wiped out all bar 200 attachments from your database.

You would need to restore from a backup.

Like I had said, that was just a sacreficial test site based on an old back up.

I meant how do I proceed to preserve my attachments while migrating to XenForo.

 

[Slavik]

Moving them all to the database isn't a viable option really, not with that many.

Your best bet would be to create a script that references the database for the user ID then sorts the relevant attachment into the corresponding correct folder structure. Thats assuming you have a mix and match of the old and new structures.

If they are all using the incorrect format, you could probably alter the import script instead.

 

[dougdirac]

Your best bet would be to create a script that references the database for the user ID then sorts the relevant attachment into the corresponding correct folder structure. Thats assuming you have a mix and match of the old and new structures.

Yeah, that's what I was thinking. Poking around it seems I have working attachments with both schemes.

For example, I look at the files in the directory [path]/attachments/9703/ and look up user ID 9703, I find those are his images. At the same time, if I look in [path]/attachments/1/3/2/7/ and look up user ID 1327, those are also the correct attachments.

(No idea how that happened and not sure of the history. I know we used to be on vB3 years back and the admin at the time was inexperienced and probably did some gallery type add-ons. I took over after we had moved to vB4 which was kinda broken and got us updated to vB 4.2 which was much more stable.)

So if I write a script to scan the directories with numbers larger than 9, and move files from /1234/ into /1/2/3/4/ , will that be enough for the XF importer to properly link the attachments?

Thanks!

 

[dougdirac]

For example, I look at the files in the directory [path]/attachments/9703/ and look up user ID 9703, I find those are his images. At the same time, if I look in [path]/attachments/1/3/2/7/ and look up user ID 1327, those are also the correct attachments.

Correction. In my live site, with only one or two spot checks, it seems only files in directories /1234/ work and those in /1/2/3/4/ are broken. On my test site, after that exercise of trying to put attachments into the database, the /1234/ attachments are no longer linked but the /1/2/3/4/ work again. Would be great if I could recover both sets imported properly. Though the priority is those in the /1234/ directories since seems there are only 220 or so of the other kind.

 

[dougdirac]

Did a test by moving the files in /143/ into /1/4/3/ which already had some files in there, and both types seemed to have been properly converted by the XF imported. So looks like this will work. Thanks.

 

[dougdirac]

Just to follow up on this. I made a script that automatically created the missing nested subdirectories and moved/resorted all the files into them in the way the XF importer expected. So it worked, so I'm happy.

 

[elsparkodiablo]

I apologize for the thread necromancy here, but we seem to be running into this issue with our imports. Is this script available?

 

[Mike]

I'm not aware of it being released. Moving your attachments back to the database in the source system should resolve the issue.

(Based on another thread, I believe you're using a non-official importer. It may be best posting in that thread as the approaches are totally different.)

 

[elsparkodiablo]

Yeah we are using the non-official importer. Unfortunately we have dozens of gigs of attachments so moving them to the db is a non-starter. We'll have to create a script I guess unless someone has the code handy to do so

 

[hzg7]

Yeah we are using the non-official importer. Unfortunately we have dozens of gigs of attachments so moving them to the db is a non-starter. We'll have to create a script I guess unless someone has the code handy to do so

I know this is over a year late but I had the same legacy attachment structure issue, found this thread and was bummed that a resolution wasn't posted... so I decided to brush back up on bash and give it a try. I have successfully converted my attachment structure and imported them into xenforo.

Here's the gist: https://gist.github.com/hzg/98ebc770ebd48d64a15bd93e95c9a8d6
This will recursively convert the structure /1234/ to /1/2/3/4/ and so on. It makes a copy and leaves the original destination alone, so be sure you have enough disk space.

Hope this helps some of you get out of the grips of vb.

 

[dougdirac]

Yeah we are using the non-official importer. Unfortunately we have dozens of gigs of attachments so moving them to the db is a non-starter. We'll have to create a script I guess unless someone has the code handy to do so

Sorry, I never noticed this request. It was a while ago, so I'd have to dig up what I did. Seems a moot issue now. I think I did it in command line php. If someone really wants, I could try to find it.