RM 1.1 Import Files from DownloadsII

[Sadik B]

This is more of a discussion thread and not a support question. My Oracle Forums has about 12000 files in DownloadsII. I have been considering moving this site to XF for over a year now.

Have any of you successfully imported from Downloads II several thousand files? Is there an importer I may have overlooked?

If not, I am considering coding an importer for my needs. Would there be any commercial interest in it? Else I could just import by directly running queries on the database and not bother with an importer. Thoughts?

 

[0ptima]

With multiple table entries per RM entry and serialized fields, I beleive it would be easier to create an importer.

It should be relatively easy to create a Downloads II importer by combining my VB Garage importer with this Threads to Resource converter I had developed.

The VB Garage importer is based on the XF importer framework which can process hundreds of thousands of records with ease. It creates custom fields (single line text and dropdown), populates the custom fields and created threads with attachments. Although the custom fields are for the Showcase addon, it should be easy to adapt it for the RM.

Once the thread and RM custom fields have been created, the code from the Threads to Resource converter could be used to converter the thread to a RM entry.

Once the conversion is complete you will need to rebuild your RM entries in order to see the custom fields.

 

[Sadik B]

Thank you for your suggestion.

I see the benefit of using the Threads to Resource converter. However I see one potential problem.

My Documents section using DownloadsII has over 12K files, some of which are also images. (Charts mostly showing business processes). Correct me if I am wrong but your threads to resource converter imports the first non-image attachment from the thread.

And of course then there is the issue of altering the VB Garage importer for the DownloadsII schema.

Hmmm, I am not sure if it would just be simpler creating an import script for DownloadsII directly into resources. I was checking out the actionSave when uploading the resource, and the main challenge I see is in the attachment data.

I am going to have a crack at this for my own site though at this point and not bother to create a general importer. Let's see how it goes!

 

[0ptima]

My Documents section using DownloadsII has over 12K files, some of which are also images. (Charts mostly showing business processes). Correct me if I am wrong but your threads to resource converter imports the first non-image attachment from the thread.

All images and the first non image post will be converted.

And of course then there is the issue of altering the VB Garage importer for the DownloadsII schema.

Just replace the select statements with the one you would use to select the data from DownloadsII.

Hmmm, I am not sure if it would just be simpler creating an import script for DownloadsII directly into resources. I was checking out the actionSave when uploading the resource, and the main challenge I see is in the attachment data.

The garage script could be used as a base as it has everything you need to connect to VB and XF. For the attachments, take a look at the _addAttachments() method.

 

[0ptima]

I created a working DownloadsII importer based on the two addons. For each download entry, a new thread will be created with the download as an attachment and all images associated to the download will also be attached to the same thread. The thread and all attachments will then be converted to a resource.

To run the importer, both addons will need to be installed and the VBPGConverter.php file will need to be replaced with the one attached in the zip file.

Select the DownloadsII converter


Then enter in the config (There will still be references to VB Garage and Showcase, but just ad your DownloadsII and RM info)



Test it out on a test system.

 

[Sadik B]

That is absolutely amazingly awesome!

I am trying out right now on my dev install!

Will let you know how it goes...

 

[Sadik B]

Hey Many Thanks Optima.

It worked great. Faced a few issues during import but nothing which I couldn't fix. Also I added a step to import the categories and and later passed on the category id to the transferring method so that my category Ids were retained.

Did a little dirty trick where I disabled the _discussionPreSave prefix_id checking (commented it out in the Discussion Thread DataWriter) and then passed the file id as the prefix_id of the thread which your addon created. Then when transferring to resource used the prefix_id for resource_id, and then updated thread prefix_id back to 0! (All this because I am too lazy to extend the thread datawriter and add a new column! )

Well, all in all it worked wonderfully well. Can't thank you enough my friend, you certainly saved me at least 2 days of work!

For anyone else interested in this, I have attached my modified VBPGConverter.php file if anyone wants to see... A lot of stuff is custom to my forum so use that file only if you know what you are doing.

 

[0ptima]

Thanks for the update.

I noticed that if the download entry is an image, the image will show in the resource and the resource will also have an empty download button.


The following should fix this issue

 

[Alpha1]

Interesting. I wonder if it would work to do this:
Import:
Andrew D's Links and Downloads Manager (LDM) -> DownloadsII -> XF threads -> XFRM

And then redirect LDM urls to XFRM urls.

 

[Sadik B]

The is_fileless tweak worked for the image downloads.

Discovered two issues, investigating for a fix...

1.

When the filenames in $dl['url'] are too long, apparently it's not being imported. For instance I have a file in downloads2 Oracle-Apps-Forms-Personalization-Demonstrated-ExampleShipping-and-Transaction-FormUsing-Zooming.pdf

I have a total of about 1700 files which didn't get imported, possibly all due to their file names being too long. Investigating a fix..

2.

Apparently there is something funny going on with xenforo's deleting of attachments. Since I have been running the importer a few times today, suddenly my server's disk space was 100% full, this in spite of the fact that I deleted all the Resource Categories and assumed the resources and attachments got deleted and also deleted the associated forum node and assumed the thread attachments got deleted, but this is clearly not the case.

Of course also ran the hourly and daily cleanup crons but yet the attachments are sitting there in the filesystem...

Again, investigating what's going on there...

Interesting. I wonder if it would work to do this:
Import:
Andrew D's Links and Downloads Manager (LDM) -> DownloadsII -> XF threads -> XFRM

And then redirect LDM urls to XFRM urls.

Yes, if you could have data inside downloads2, this will import just fine.

Once I have this perfected, I will attach the final importer here.

Regards

 

[Alpha1]

How do you plan to handle url redirection? My forum posts have at least tens of thousands of links to LDM entries and files.

 

[Sadik B]

How do you plan to handle url redirection? My forum posts have at least tens of thousands of links to LDM entries and files.

I tweaked Optima's importer to retain the ID of the file. So my downloads2 urls and resource manager url have the same id in them. It will just be a matter writing the proper htaccess redirects.

 

[0ptima]

When the filenames in $dl['url'] are too long, apparently it's not being imported. For instance I have a file in downloads2 Oracle-Apps-Forms-Personalization-Demonstrated-ExampleShipping-and-Transaction-FormUsing-Zooming.pdf

I have a total of about 1700 files which didn't get imported, possibly all due to their file names being too long. Investigating a fix..

Can you upload one of these files to a regular XF thread? Mayne the file is corrupt or cant be read due to file permissions incorrectly set on that file

Apparently there is something funny going on with xenforo's deleting of attachments. Since I have been running the importer a few times today, suddenly my server's disk space was 100% full, this in spite of the fact that I deleted all the Resource Categories and assumed the resources and attachments got deleted and also deleted the associated forum node and assumed the thread attachments got deleted, but this is clearly not the case.

Of course also ran the hourly and daily cleanup crons but yet the attachments are sitting there in the filesystem...

Try on a clean install. I deleted all XF threads and RM cats and RM resources. I ran your version and and then deleted the category from the Admin CP. After running the hourly clean up cron, the file system was empty.

 

[0ptima]

Interesting. I wonder if it would work to do this:
Import:
Andrew D's Links and Downloads Manager (LDM) -> DownloadsII -> XF threads -> XFRM

And then redirect LDM urls to XFRM urls.

Are the links in a table called local_linkslink ? If so, I converted them to threads and then to resources with the url in a RM custom field. The only thing is that it does not handle redirects.

 

[Sadik B]

Try on a clean install. I deleted all XF threads and RM cats and RM resources. I ran your version and and then deleted the category from the Admin CP. After running the hourly clean up cron, the file system was empty.

Ok one mystery solved.

The Hourly cleanup cron job calls the Model method deleteUnusedAttachmentData which has a hardcoded limit of 1000 when fetching attachment data rows to delete. Since I ran the importer 3 times (and I have 12K+ files) I was having about 36K records in my data table which were orphaned when I deleted the resource categories to try again.

The hourly cleanup only deleted 1000 rows leaving my hard disk completely filled!

Took some digging that! I can understand in normal circumstances it is really impossible that a board would have 1K+ files to delete every hour.

Still trying to determine why large file names didn't import....

 

[Sadik B]

Found it. It turns out over the years we pruned some users and some of the users who had uploaded files had their accounts deleted. I just removed the userIdMap check to import the files anyway.

Now writing a new step to import download stats for the files.

 

[0ptima]

Found it. It turns out over the years we pruned some users and some of the users who had uploaded files had their accounts deleted. I just removed the userIdMap check to import the files anyway.

Which user are you associating those downloads with?

 

[Sadik B]

Which user are you associating those downloads with?

Well none. Those were zero posters account inactive for several years which I had pruned. Just did not want to lose the documents.



The Author links remain unlinked, on the resource page which is fine by me.

 

[Alpha1]

Is this import through CLI?

 

[Sadik B]

No, regular browser based.