Attachment images not appearing in post body after database restore

[tourmeister]

I had an issue where I had to use a VERY recent backup to restore my database. By recent, I mean within an hour prior to the problem. Fortunately, this was in the wee hours of the night when there was little activity on the site, so there was no real loss of data by overwriting the active database with the backup. However, when looking through a long thread that has MANY image attachments within the posts, the images no longer appear. All that is seen is the file name of the images. What is strange is that this only happens in the first half or so of the posts in the thread and then after a certain point, in fact in the middle of a particular post, the images appear as they should for every remaining post in the thread. Here is that post,

There comes a time...

Thanks for sharing. Great pics. If you're like me, you stare and stare and stare and take countless pics that just never quite capture the majesty and beauty of it all.

www.twtex.com

Every post before that looks like the first part of that post with just the filenames and no image. Every post after that one shows the images as they should. Given that the bulk of the posts in this particular thread that include images are posts made by me, I could manually go into each post and re-attach the images, but that would be a LOT of work. Also, it is affecting posts by different users where I would not be able to go in and re-attach the images. It is possible there are other threads that are affected, but I have not looked for them as of right now. I have no idea what might be causing this or how to fix it.

I am hoping it is something simple...

 

[Mr Lucky]

The posts seem to have a correct URL path so maybe something happened re: the attachment folders.

This one https://www.twtex.com/forums/attachments/img_7808-jpeg.360980/

should be in the folder: /internal_data/attachments/360

(I think that's how it works based on attachment id

 

[Kirby]

This one https://www.twtex.com/forums/attachments/img_7808-jpeg.360980/

should be in the folder: /internal_data/attachments/360

The folder names are data_id / 1000, not attachment_id / 1000.

So while attachment_id and data_id might be identical (this is usually the case), they don't have to be.

 

[Mr Lucky]

Also have you checked the attachment browser in the Admin panel > content? - you can filter by date.

The folder names are data_id / 1000, not attachment_id / 1000.

So while attachment_id and data_id might be identical (this is usually the case), they don't have to be.

Just found this so Im a bit confused:

Attachment folders, what is the structure?

I see that attachments are stored in the following folder: /internal_data/attachments/0/ Coming from vB4 I'm used to the userid being the folder path. How is xF different? Are attachments placed in the /0/ folder until a certain amount and then a new folder is created?

xenforo.com

 

[Kirby]

Just found this so Im a bit confused:

Attachment folders, what is the structure?

I see that attachments are stored in the following folder: /internal_data/attachments/0/ Coming from vB4 I'm used to the userid being the folder path. How is xF different? Are attachments placed in the /0/ folder until a certain amount and then a new folder is created?

xenforo.com

To be fair, the post by @Chris D is about 10 years old.
But you are right, it is confusing.

IMHO it should read "Attachments with data_id 0-999 are stored in the 0 directory, ..." to avoid ambiguity.

 

[tourmeister]

So I went to the Admin CP and used the username and date filter to show all attachments. This is what it looks like for the missing images,



If I click on any of them, I just get an "Oops, we ran into a problem" type of message.



Here is what I see when I edit one of the posts with the missing images,



When I go the Filemanager on my cPanel for the server, I found the diretcory,

/public_html/forums/internal_data/attachments/

and then there are the numbered subdirectories. Other than perhaps looking at the date the files were last modified, how could I correlate one of these to an actual attachment in a post?



Should the six digit number in the the ATTACH tags match the first six digits in a filename in one of those directories?

 

[tourmeister]

The posts seem to have a correct URL path so maybe something happened re: the attachment folders.

This one https://www.twtex.com/forums/attachments/img_7808-jpeg.360980/

should be in the folder: /internal_data/attachments/360

(I think that's how it works based on attachment id

The largest folder number is 347. I have been able to find attachments that were created AFTER those that are missing in folder 347 and prior folders. So I would have expected the missing images to also be in folder 347 or prior, but I can find any of them. When a thread containing attachments is deleted, do the images in those folders get deleted as well?

 

[Mr Lucky]

and prior folders. So I would have expected the missing images to also be in folder 347 or prior, but I can find any of them.

So it sounds like those folders 348 and higher have gone missing.

As the image URL I looked at showed image id 360980 (or data_id) then it should be in folder 360 - because the first numbers of the id reveal the directory number (that go up in 1000s)

ie 360980 is going to be in folder 360.

how could I correlate one of these to an actual attachment in a post?

I'm not sure, maybe by the date and time is the only way I can think of. Or search in database xf_post using the id

But

Should the six digit number in the the ATTACH tags match the first six digits in a filename in one of those directories?

No, the filename is not relevant, it is the id (attachment id or data id ) that is given it by xF that is relevant

When a thread containing attachments is deleted, do the images in those folders get deleted as well?

Yes. If a post is hard deleted, bye bye attachment

 

[Kirby]

When a thread containing attachments is deleted, do the images in those folders get deleted as well?

Not immediately

The files are removed by cron Hourly clean up which by default is executed 10 minutes past every full hour.

So if a posts were deleted after your DB backup and the cron was triggered before the restore the files would have been deleted.

 

[Kirby]

As the image URL I looked at showed image id 360980 (or data_id) then it should be in folder 360

360980 is the attachment_id, this is not necessarily the same value as data_id (which is used to calculate the directory)

 

[tourmeister]

Not immediately

The files are removed by cron Hourly clean up which by default is executed 10 minutes past every full hour.

So if a posts were deleted after your DB backup and the cron was triggered before the restore the files would have been deleted.

I believe this is what happened. The thread in question has several thousand images in it from a 50 day motorcycle trip I did with my two oldest kids recently. As soon as I realized what I had done, I immediately closed down the forum to prevent new data being entered into the database and used a backup to restore. I think the hourly cleanup cron took out just under 1000 images, about half the trip, before I was able to get the database restored from the backup. No other threads were affected because this was the only one deleted prior to the backup. So.... it seems I will be doing a good bit of re-uploading of attachments in the coming days/weeks. The good thing is that all my report narrative is still intact so I don't have to try to recreate that! Also, 99.9% of the images in that thread are from me, so I don't have to worry about attachments from other users being affected.

Thanks for the feedback.

 

[Kirby]

Do you have a filesystem backup that was taken before the thread was deleted?

If so I'd try to restore the missing files from there.

If not: I'd try to automate the "re-upload" by creating a ZIP/TAR of all images, upload this to the server, unpack to a temporary directory and copy / move files in place based on original filename.

 

[tourmeister]

I've already started to re-upload images. I am thinking the new files might cause conflicts with the old files? Either way, the old files have LONG and bizarre names, so I am not sure how I would recreate those without a backup of the original files.

 

[Kirby]

I am thinking the new files might cause conflicts with the old files?

No. You should just remove the "ghost attachments".

Either way, the old files have LONG and bizarre names

That's great so they are most likely unique

The database table xf_attachment_data has the original filenames and the data_id