[TH] Import Tools [Deleted]

[Code Monkey]

I think I figured it out. The default is xf_import_log but the script asks you if you want to dump it into archived_import_log

From the manual

Open the 301config.php file for the import scripts and find the line that references the import log table, and edit it according to the instructions given in that file to point to your archived import log table:

define('IMPORT_LOG_TABLE', 'import_log_old_forum');>

I need to change that to archived_import log. I did a new import of the old forums onto what is now my dev site. I will just dump that table into the live site and change the setting and I should be good to go.

Or better yet I just dumped the data into xf_import_log.

 

[Alpha1]

Does this work on xf 1.2?

 

[Jon W]

Does this work on xf 1.2?

I'm not aware of any issues.

 

[Marcel]

I have a couple of questions. I've used this a couple of times now, and I'm a little confused.

Install forums, install plugin, then go straight to the Importer.
Select "Retain imported content ID's" as we'd like to do that.
Enter database values, then go through most of the steps of the importer
(Users, threads, posts, etc etc etc).
Finalise import, saving the log to the suggested archived_import_log.

Then when I need to 'update' the import with new threads and posts, I retrieve 'archived_import_log', don't select "retain imported content ID's" (as this would throw up a 'duplicate PRIMARY' key error).
And start the import.

I then skip the ones I've already done that I know haven't changed since last time, then when "Threads and posts" become available, redo that again (and it *should* skip over the threads and posts already imported).

Then finalise import again.

Now what if I wanted to use the importer / steps multiple times?
Each time I need to retrieve the log, but each time I need to save to a new log too.
So when I get to the third time using the importer, I retrieve the 2nd import log, but this doesn't contain what was imported the first time?

Does all that make sense?

 

[Jon W]

Then when I need to 'update' the import with new threads and posts, I retrieve 'archived_import_log', don't select "retain imported content ID's" (as this would throw up a 'duplicate PRIMARY' key error).
And start the import.

This is incorrect.

If you have imported the log correctly, it should be fine to "retain imported content id's" because it will skip over the ones it has already imported.

Now what if I wanted to use the importer / steps multiple times?
Each time I need to retrieve the log, but each time I need to save to a new log too.
So when I get to the third time using the importer, I retrieve the 2nd import log, but this doesn't contain what was imported the first time?

Does all that make sense?

This is incorrect.

When you run the importer for the second time and then save the log, the log will contain the data from the 1st + 2nd import combined (because you imported the 1st log at the beginning). Run it a 3rd time after importing the 2nd log, and the 3rd log will contain the data from the 1st + 2nd + 3rd import combined. And so on.

It might help you to understand better if I explain what is in the log. The log is a list of IDs from the site you are importing from and the corresponding IDs of the site you are importing to. So it is ignoring all IDs that have already been imported and each import is just adding new lines to the log for the new IDs that it has imported.

Hope this clears things up a bit.

 

[Marcel]

That most certainly does

So any subsequent imports I *should* be selecting the "retain imported content ID's", and if the archived_import_log table is 'loaded' correctly then it should just continue where it left off.
The same for the 2nd and 3rd imports too. I think when I first tried it the other night, I did choose a second import log table, and when browsing through PHPMyadmin I noticed the second table was much much smaller than the first (as the second import was much smaller). So I assumed that it only contained info from the second import.

Thanks for the reply, I'm going to see if I can make more headway with this import. a good few million posts means that each step when it rebuilds the cache takes quite a while

Thanks again.

 

[Jon W]

I think when I first tried it the other night, I did choose a second import log table, and when browsing through PHPMyadmin I noticed the second table was much much smaller than the first (as the second import was much smaller). So I assumed that it only contained info from the second import.

I can't think of any reason why this should be the case. Do let me know if you notice any problems as there may be a bug. How I've described it above is how this add-on is supposed to work.

 

[Marcel]

OK Thanks. I'm just going through another cycle now.
I'm not sure what I've done but although I've imported the forum structure etc and users, whenever I go back to do a second import cycle, it goes through the motions of importing threads and posts (but only take 45 mins), updates the caches....and nothing's there aftwards (empty forums).
If I select retain imported content ID's it warns me of a duplicate ID because I've imported the forum structure and users already.

Basically what I'm trying to acheive is get as much as I can imported, then set up and get ready on our XenForo installation (plugins, permissions, styles etc).....then at the last minute, close the old VB, import as little as I can get away with (hopefully just new users, new threads and new posts since the last 'import' cycle), and then reopen with some shiny new forums, which is why Im trying my best to get this importer to work for me

 

[Jon W]

Sounds like you are not actually importing the log at the right point if you are getting errors.

You need to import the log AFTER you have started the new import.

 

[Mendo]

I got those same issues too. Nothing I did seemed to work if I was importing more than once. I even made sure to save the tables every step of the way and name my saves different and tried same names/overwriting and it still didn't work after that first time. I ended up having to meld table data together manually from different imports into a frankenforum lol

Just to clarify I'm talking about the whole retain ids, duplicate ids error. And I was doing the same idea where old forum was imported but stayed live, tweaked the xenforo the way I wanted it, then closed old forum and tried to reimport.​

 

[Jon W]

I've created a video of an import in two stages from phpBB3 with 'retain imported content IDs' selected. See that I don't get any error messages:
http://xenforo.com/community/resour...ges-with-import-tools-by-waindigo-video.2289/

 

[Marcel]

Oooh thanks, will take a look at that.

Before I do though, I don't know what I've done, but I got it working.
I did a partial import, finalised it and returned to it today.
I clicked Import External Data, configured the database values, and only when it came up with a list of steps (with the first two being available...just like a fresh new import) did I then just select "Import archive log" from the menu.
I then went back to Import External Data and it asked me to continue, which I did.
I think it still showed the first two options (again, like a new import would), but I skipped those and when I got down to threads and posts, when I clicked Import Threads and Posts....it actually started copying these over. (Previously it wouldn't...it would just go through the whole process taking about an hour..get to the end, and there would be nothing in the posts table).

So, in a nutshell...I seem to have got something working. Although I think it's hung, it's been at 57.74% for about an hour now

Thanks for the help Waindigo

 

[Jon W]

That is the correct way to do it. You will see that is exactly how I did it in my video.

Not sure why it is stuck at 57.74% or how to un-stick it unfortunately. Try disabling the Import Tools add-on.

 

[Jon W]

You could try this link:
/admin.php?import/skip-step&step=threads

 

[Jon W]

Just noticed something interesting...

Selecting "retain imported content IDs" will always delete the nodes with IDs 1 and 2. This means that if you select that option, it is important that you do not skip the "Import Forums" step.

I'm going to see if I can come up with a way to workaround this.

 

[Jon W]

Waindigo updated Import Tools by Waindigo with a new update entry:

Version 1.0.1 released

New features:

• Improved support for XenForo 1.2.

Bug fixes:

• Disabled (by default) the ability to skip the 'Import Forums' step when 'Retain Imported Content IDs' is selected, due to this option causing nodes with IDs 1 and 2 being deleted if this configuration is selected (therefore almost always requiring re-import). An option is now available to override this behaviour should you still need to skip this step.

Read the rest of this update entry...

 

[ehd]

Hi, thanks a lot for your tool. Yes, I read all the threads, but it not 100% clear to me. And trying is hard, because this would take over a day....I have hundreds of thousands of attachments...So if I do the import and then wait a week or so, and many more attachments were added after .
The importer skips all the old attachments and just adds the new ones? So it doesnt take 24 hours but just a little time?

 

[Jon W]

Hi, thanks a lot for your tool. Yes, I read all the threads, but it not 100% clear to me. And trying is hard, because this would take over a day....I have hundreds of thousands of attachments...So if I do the import and then wait a week or so, and many more attachments were added after .
The importer skips all the old attachments and just adds the new ones? So it doesnt take 24 hours but just a little time?

Correct.

 

[Marcel]

Thanks for the update. I wiped the DB and restarted the import again. Although the import of posts craps out with 503 errors (That's one for another thread, another issue ), it seems to be working OK now, as youve shown in your video.

I'm not at the point of Completing the Import and then going back in yet, but it's installed and working it seems (And I've enabled the new option as we don't have any forums with an ID of 1 or 2, so it's a moot issue).

Thanks again

 

[Jon W]

Great news. Hope you manage to sort out your other issues.