Duplicate Threads?

[Mopquill]

Okay, so, I'd installed XenForo, and imported my vBulletin 3.8 settings just fine. However, due to a very long story, one of my staff accidentally wiped the database. I hadn't made a backup yet (D'oh!), but it'd only been a few days, so, I figured I'd just re-import vBulletin, as it's been "off" and unused since the first import (i.e. the exact same import I did the first time).

Now though, when I import, everything proceeds fine as far as the import itself goes. If I check the forums root then, it says there are 86,000-something messages. Once I proceed with the import completion (Click the "Complete Import" button), it goes and rebuilds caches. Then, it tells me to do two things: 1. check the usergroup permissions, and 2. rebuild the search cache. But, at this point, the forums say there are 168,000-something messages. If I view any forum, I can see two of each thread all the way down.

I've tried dropping every single table in my database and starting the process over- three times. This keeps happening. Anyone have any ideas? I've completely re-uploaded the XenForo files already, and wiped the database, and even used a different (new) database/user.

The Error log in the AdminCP is empty. I'm thinking there might be some cache file somewhere ruining something, so, I figure I might need to delete the /forums/ directory, wipe the database, re-upload it, and re-install. Anyhow, this seems to be indicative of some bug or glitch or something. I don't want to have to go through my board and delete some 3,000 threads (and have all the orphaned thread/post IDs to go with that), so, I'm wondering if anyone has come across this, and has any potential solutions. I'm willing to try anything that sounds intelligent.

At the very least, thanks for reading.

EDIT: Not sure if this is relevant, but, I'm adding it in:
The staff member was trying to install their own xenforo forum under a different linux user on a different website, and somehow got MY forum's database information in their config.php . I've had them check and double-check, but, they have no idea how it happened. I'm sort of willing to blame bumbling, but, it sort of seems like something else is happening here.

Considering that wiping the database and files should leave everything exactly the same as the first install and it's -not-, that would indicate to me that something it can read somewhere changed. I'm not sure what it changed, though.

 

[Mopquill]

Thanks, Dean. I've seen a lot of this silent note-passing. While it's great for communication, it seems to often leave people *feel like* they're being ignored. I've kept posting hoping something like this was going on, so, thanks for letting me know.

I've been trying to make sure my posts are thorough.

cedivad, what did you change? I'm not exactly comfortable with running arbitrary code on my databases, you know? Plus, I'd be worried additional problems might crop up from it, but I understand PHP logic pretty well. If you tell me what you changed, I can hopefully understand it and make a decision about its use. Either way, thank you for sharing, you've been extremely considerate and helpful. =]

 

[Peggy]

Sorry Mopquill, Mike and Kier took some much needed time off this weekend. We've made sure that they see your thread, and will help when they do.

 

[Mopquill]

Thanks very much. =]

 

[Dean]

They are really busy.

I will try to do an import, Ive been getting things arranged for that (complicated). Unknown if that will help or not.

 

[Mopquill]

I know, that is why I'm trying to be patient. =]

The problem doesn't seem to be on an import, it seems to be on a second import, as if it has to do with it storing extra files somewhere or something.

My first import worked fine, but my administrator tried installing his own XenForo, and the config.php file it gave him had my exact database information. For all I know, the import script is broken in more ways than one.

Anyhow, he installed the new stuff over mine, and I wiped his stuff, and now I'm trying to get mine working- however, now when I import, it duplicates a number of imported objects: threads, posts, messages, and likely some other stuff I don't even know about.

Hope this helps things a bit.

 

[Dean]

I have not read this thread in detail yet, have you tried to wipe everything, re-upload, and start with a brand new database?

(that would have been my approach)

Specifically, if you have it installed on yourdomain.com/testxf
-delete *everything* in that directory
-re-upload the virgin files from your hard disk
-delete the database you had for xf
-create 2 new databases, add a user to both databases, enable all mysql permissions
-get a fresh backup from your site, restore that into one of the databases
-go to www.yourdomain.com/testxf
-it should then ask for your database - give the details for the clean database - not the backup database you just restored.
-install.

Then re-import.

 

[Mopquill]

I definitely appreciate the in-depth instructions, but, I've done all of that already. Multiple times. It's like the import script is storing variables somewhere that isn't the database or the /forums/ directory. I have no idea what's up. Although it's possible that the files I have on my hard drive have been slightly changed or something.

I can try again via zip extraction in a bit, but the process takes me around half an hour. I'll try again and let you know.

 

[Dean]

*scratching head*

yes it takes quite a while to re-upload all the individual files, I do know there are shortcuts to doing that, but I believe in doing things exactly as the customer does them.

 

[Mopquill]

Sorry, I meant the re-import process takes about half an hour. I upload the zip as given to me by XenForo and just use unzip file.zip on it.

I think that's a good belief you have, though!

 

[Andy.N]

Had same problem.
I got a good import the first time, when i updated the backup and reimport into xf, i got duplicates.
Reinstalled XF and reimported got duplicates as well.

So i will just wait for beta 2 to do a reinstall.

 

[Dean]

I cannot imagine how this could be happening, and why doing everything over does not fix it. But if 4 different people have the exact same problem... it is quite probably real.

I will post a bug report with a link to this thread. I know Mike reads those, not sure if he reads this forum yet.

 

[Mopquill]

I felt it a bit arrogant to call it a bug, as it might be isolated to me, or something stupid I'm doing. Although at this point, it'd certainly seem to be a bug.

I'm currently trying everything from scratch at once: New (empty) database for XenForo, New Database with vBulletin data freshly imported from a backup XenForo had not touched, new files, freshly extracted from zip on server, zip was freshly downloaded a few minutes ago.

I'll let you know how it turns out, but, I'm not very hopeful.

 

[Mopquill]

After doing everything I stated above, it's still happening. It's like it knows that I've imported before, even though what it imported from was a newly-created database with tables imported from a backup of vB before I had XenForo, and even though my /forums/ directory was freshly extracted from a freshly downloaded zip, and even though its database was also newly-created. It's like it's storing a variable somewhere besides MySQL or my /forums/ directory, and then duplicating based on that?

But the thing is, I didn't run any of this as root. It should only have access to that one database, and the directory it's in. And I guess /tmp/ . Although I've tried clearing that before (though not this time), and it doesn't seem to want to do it.

I'm truly at a loss here.

If you like, I can provide you with the vB database I'm trying to import via PM, and if you can get it working, you can just give me the working Xen database. If not, you'll be able to witness my problem. It seems productive either way, but it's your time, so it's obviously your choice. Either way, thanks for your help thus far.

 

[cedivad]

cedivad, what did you change? I'm not exactly comfortable with running arbitrary code on my databases, you know? Plus, I'd be worried additional problems might crop up from it, but I understand PHP logic pretty well. If you tell me what you changed, I can hopefully understand it and make a decision about its use. Either way, thank you for sharing, you've been extremely considerate and helpful. =]

I've only added one query per thread to check if it was already imported before importing it.
Open the file and search for '// EDIT'.

 

[Mopquill]

I've only added one query per thread to check if it was already imported before importing it.
Open the file and search for '// EDIT'.

I see no such comment (there are five instances of the word edit (any case), none of which are yours), however, I believe I found your code:

            $cnt = $this->_db->fetchOne('
                    SELECT COUNT(*)
                    FROM xf_thread
                    WHERE title = ? AND post_date = ?
                    ', array($thread['title'], $thread['dateline']));

            if($cnt > 0) continue;

This actually looks good. I'll try it and let you know.

 

[cedivad]

I see no such comment

I was wrong, another time.
However that's the code and it should work.

 

[Mopquill]

I'm currently trying, I'll let you know what happens. =]

 

[Mopquill]

No go, unfortunately. It seems to have significantly reduced the number of duplicated threads (160,000 - 180,000 to around 104,000), but the number is supposed to be 86,000.

It might be worth mentioning two things.

One, the total number of posts I have seems to differ with each import.
Two, I seem to have the correct number of messages (or close to it) before I hit Complete Import.

I appreciate you trying, though. =]

 

[cedivad]

I confirm that with my little code edit, my import works perfectly.
I don't know why it won't work for your, but i can guess: maybe since that the username field is in latin1 in the vbulletin database (while it's in utf8 in the xf database) with some usernames this is a problem.

For example not with 'name' but with 'nàme' latin1('nàme') is != utf8('nàme').

 

[Mopquill]

Hmm. I did not try your script at the same time as clearing everything. Are there are files or caches I need to delete, or do you not know?

I could try doing everything from scratch again WITH your script, though I'm still rather pessimistic about any of this working. XD

I have seriously tried importing this like... 15 times now. I'm a bit reluctant to try something that will not likely work. XD