1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

XF 1.5 Attachments import failed

Discussion in 'Installation, Upgrade, and Import Support' started by dougdirac, Sep 23, 2015.

  1. dougdirac

    dougdirac Member

    I'm doing a test migration from vB 4.2.3 to XF 1.5.

    Everything seems to go well with no error messages. However, the vast majority of the attachments don't get imported.

    In my vB attachments directory there are number subdirectories from "1" to say "39448". Each with a few numbered files with extensions .thumb and .attach. I should have thousands if not 10s of thousands of attachments on my 1M post forum.

    When I selected import attachments, process only took about 15 minutes during which I could watch the doneness percentage climb steadily from 0 to 100%.

    In my new XenForo attachments directory there is only one subdirectory and it's labeled "0". In that directory there are about 250 small jpg's. The images are all thumbnail sized.

    So why didn't it process my attachments? Any suggestions?
  2. Mike

    Mike XenForo Developer Staff Member

    Hmm, if I interpreted that correctly, I don't think that's the format our code expects. I think the files should be in directories like 3/9/4/4/8.

    If that's the case, the easiest thing may actually be to move the attachments back into the DB and let XF process them from there.
  3. dougdirac

    dougdirac Member

    Move 50 gigs to the database?

    What's the scheme behind the directory structure for attachments in XenForo (and vB since I don't understand that either)?

  4. Mike

    Mike XenForo Developer Staff Member

    I made that suggestion more on the basis that you mentioned only having thousands of attachments, so I didn't expect it to be nearly that significant. (We have significantly more attachments here and much less usage, though I wouldn't want to shift ours into the DB.)

    The structure in XF isn't overly relevant here because the issue seems like it's down to the files not being found from the source DB. But XF puts files into directories based on IDs in blocks of 1000 (eg, ID 7565 goes into directory 7).

    The source expects files like this:
    $attachFileOrig = "$options[path]/" . implode('/', str_split($attachment['filedata_userid'])) . "/$attachment[filedataid].attach";
    So the user ID divided by digit as the folder name and then the ID of the data (/path/to/attachments/1/2/3/4/5/6789.attach).

    It sounds like your files aren't using that structure, though I'm not sure why. There may have been a different (deprecated?) format that was used for a period of time and only supported as a legacy version? Unfortunately, our code wouldn't support reading from that format. That's roughly why I suggested moving them back into the DB (you could move them back to the FS again), as we don't have support for the file structure that you appear to be using. (I may have seen this once(?) before, but most imports seem to bring the files over without issue so I suspect it's rare at best.)
  5. dougdirac

    dougdirac Member

    Yeah, it's weird. I see evidence of that nested structure on the low numbers. But it seems mainly that the directory is the user ID itself, rather than coded into subdirectories.

    Playing around with a test site that is from a backup from over a year ago I see this:

    So I'm testing out the move to the database, but it's not promising. It only took a few minutes to run and I get this result before finalizing:

    Not a good sign, but I'll click yes and see what happens.
  6. dougdirac

    dougdirac Member

    I'm not sure why the above says, "moved to the file system" since the option I selected and the only one available said move to the database.
  7. dougdirac

    dougdirac Member

    Well, as expected I suppose, that process killed my attachments. I mean I still see the files in the file system, but the attached images in threads are all gone. So how to proceed?
  8. Slavik

    Slavik XenForo Moderator Staff Member

    As per the converter warning, you have probably just wiped out all bar 200 attachments from your database.

    You would need to restore from a backup.
  9. dougdirac

    dougdirac Member

    Like I had said, that was just a sacreficial test site based on an old back up.

    I meant how do I proceed to preserve my attachments while migrating to XenForo.
  10. Slavik

    Slavik XenForo Moderator Staff Member

    Moving them all to the database isn't a viable option really, not with that many.

    Your best bet would be to create a script that references the database for the user ID then sorts the relevant attachment into the corresponding correct folder structure. Thats assuming you have a mix and match of the old and new structures.

    If they are all using the incorrect format, you could probably alter the import script instead.
  11. dougdirac

    dougdirac Member

    Yeah, that's what I was thinking. Poking around it seems I have working attachments with both schemes.

    For example, I look at the files in the directory [path]/attachments/9703/ and look up user ID 9703, I find those are his images. At the same time, if I look in [path]/attachments/1/3/2/7/ and look up user ID 1327, those are also the correct attachments.

    (No idea how that happened and not sure of the history. I know we used to be on vB3 years back and the admin at the time was inexperienced and probably did some gallery type add-ons. I took over after we had moved to vB4 which was kinda broken and got us updated to vB 4.2 which was much more stable.)

    So if I write a script to scan the directories with numbers larger than 9, and move files from /1234/ into /1/2/3/4/ , will that be enough for the XF importer to properly link the attachments?

    Last edited: Sep 24, 2015
  12. dougdirac

    dougdirac Member

    Correction. In my live site, with only one or two spot checks, it seems only files in directories /1234/ work and those in /1/2/3/4/ are broken. On my test site, after that exercise of trying to put attachments into the database, the /1234/ attachments are no longer linked but the /1/2/3/4/ work again. Would be great if I could recover both sets imported properly. Though the priority is those in the /1234/ directories since seems there are only 220 or so of the other kind.
  13. dougdirac

    dougdirac Member

    Did a test by moving the files in /143/ into /1/4/3/ which already had some files in there, and both types seemed to have been properly converted by the XF imported. So looks like this will work. Thanks.
  14. dougdirac

    dougdirac Member

    Just to follow up on this. I made a script that automatically created the missing nested subdirectories and moved/resorted all the files into them in the way the XF importer expected. So it worked, so I'm happy.
    Mike likes this.
  15. I apologize for the thread necromancy here, but we seem to be running into this issue with our imports. Is this script available?
  16. Mike

    Mike XenForo Developer Staff Member

    I'm not aware of it being released. Moving your attachments back to the database in the source system should resolve the issue.

    (Based on another thread, I believe you're using a non-official importer. It may be best posting in that thread as the approaches are totally different.)
  17. Yeah we are using the non-official importer. Unfortunately we have dozens of gigs of attachments so moving them to the db is a non-starter. We'll have to create a script I guess unless someone has the code handy to do so

Share This Page