bbPress 0.9 and 1.0: import issues?

[Wildcat Media]

A good friend of mine and fellow moderator just bought a license for XF, and we are going to work on converting two existing forums into one XF. We plan on using vB as the go-between, for obvious reasons.

As I understand, ImpEx can indeed import more than one forum instance into vB. In other words, he has some archived, earlier posts in bbPress 1.0, with the current forum running bbPress 0.9. We should be able to import the earlier forum, check vB to make sure it all imported, then do a second import to pull in the current forum.

Has anyone done an import from bbPress as of yet? What kind of issues should we watch for? Do items all import (attachments, avatars, etc.) and do users have to reset their passwords on first access? Any altered BBCode we need to run a cleanup script on?

Any advice appreciated! If there isn't any, we'll forge ahead, take notes and report back.

 

[Jake Bunce]

With that solution the attachments wouldn't be within xenForo's permission system. And you would have to develop and maintain your own addon for the BB code. Personally I wouldn't want to have to maintain a bridge like that moving forward. And it would have to be a permanent solution since you can't do an incremental import at a later time.

A proper import of those attachments would be better.

 

[Wildcat Media]

I think we're lucky this time, in that this forum will not be publicly viewable. After Keith mentioned the idea, my thought was to try to do attachments like this:

* If the attachment was an image, convert it to an IMG BBCode and append it at the bottom of the post;

* If the attachment were another type of file, insert it as a URL at the bottom of the post.

This could run as a home-brew script, grabbing the appropriate info and appending it to the post. Having seen the code for a typical import module, it's probably a toss-up as to which one would be easier to work with.

 

[Jerry]

This would eliminate the need to actually import the information through Impex. The attachments would still be on the server, Xenforo would display them, but they wouldn't be handled by Xenforo's attachment system... it would just be bbCode. Any new posts made in Xenforo would use Xenforo's regular attachment system, of course.

It would be considerably easier to parse the attachments BB code out after an import and get the data into xF . Having attachments lying around like that is a nightmare to manage.

 

[Wildcat Media]

It would be considerably easier to parse the attachments BB code out after an import and get the data into xF . Having attachments lying around like that is a nightmare to manage.

Hmmm. Are you saying that it may be easier to find a way to import attachments into XF directly from the post content, as opposed to bringing them in through an Impex import to vB?

I spent awhile looking at the code to create an additional import module and unfortunately, it is not making much sense. It's not the code's fault though--it just takes me awhile to get used to another developer's coding style, and how they've laid things out.

I'm just trying to figure out where I would point references to the bbPress attachment table. (I grabbed 010.php from the SMF2 importer, since I have worked with SMF2 before and am at least a little familiar with how its tables are laid out.) I found the code where it derives the raw filename, and that is easily modified. And I can see where it is prepared for insertion into vB. I guess I'm just at a loss to see where it actually grabs data from the source tables, and where I may need to manipulate it to get it ready for vB. I'm pretty sure I saw a place in 000.php (?) where you list valid tables, so the bbPress attachment table would need to be inserted there.

Maybe dinner and more caffeine will clear things up for me.

 

[Jake Bunce]

Hmmm. Are you saying that it may be easier to find a way to import attachments into XF directly from the post content, as opposed to bringing them in through an Impex import to vB?

It would be easier to make a module for ImpEx to import your attachments from bbpress to vB. The xenForo importer already takes care of attachments once they are in vB.

All other solutions will be more difficult. Your xenForo BB code idea is an incomplete solution that will be difficult to manage moving forward. And you can't easily import attachments directly from bbpress to xenForo because then you have to deal with a double postid mapping. Going through ImpEx is better.

If you are having trouble writing code for ImpEx then you will need to contract with a developer. xenFans offers an import service. Maybe they can manage your bbpress import with attachments.

 

[Wildcat Media]

Attachments aren't that important, but we'd like to save them if we could...

 

[Wildcat Media]

And save them we did! I finally had a clear moment to get into ImpEx and figure it out properly, rather than just dabbling at it. I grabbed the attachment import module from SMF2 and renamed it to 007.php (Bond, James Bond Edition™ apparently). In 000.php, I needed two things--I had to put the name of the bbPress attachment table in the approved tables list...

	var $_valid_tables = array ('forums', 'posts', 'tagged', 'tags', 'topicmeta', 'topics', 'usermeta', 'users', 'attachments');

...and also had to add a function referenced from 007 (where I also had to edit the query made to the attachments database, and changed the function name here and in 007.php):

	function get_bbpress_attach($Db_object, $databasetype, $tableprefix, $start_at, $per_page)
	{
		$return_array = array();

		// Check that there is not a empty value
		if(empty($per_page)) { return $return_array; }

		if ($databasetype == 'mysql')
		{
			$dataset = $Db_object->query("SELECT * FROM {$tableprefix}attachments WHERE id > {$start_at} ORDER BY id LIMIT {$per_page}");

			while ($row = $Db_object->fetch_array($dataset))
			{
				$return_array['data']["$row[id]"] = $row;
			}
			$return_array['lastid'] = ($start_at + $per_page);
			$return_array['count'] = count($return_array['data']);
		}
		return $return_array;
	}

In 007, I had to edit a line below the file's header (in the comment section):

/**
* bbpress_007
*

These lines had to be changed to reference 007 and 006 as appropriate:

class bbpress_007 extends bbpress_000
{
    var $_dependent = '006';

    function bbpress_007(&$displayobject)

And don't forget: that function added in 000 needs to be edited in 007 (replacing "bbpress" for "smf"):

		$data_array = $this->get_bbpress_attach($Db_source, $s_db_type, $s_tb_prefix, $start_at, $per_page);

This bbPress add-on stored attachments rather simply: they added the attachment ID to the beginning of the filename. So for an attachment with ID of 1884, with a file name of mypicture.jpg, the stored attachment filename is simply 1884.mypicture.jpg .

I did take a shortcut though--since there were only about 1500 attachments, I moved them all into one directory. Normally, attachments are stored 1000 per directory, so attachment 1884 would have been stored in ./attach/1/1884.mypicture.jpg, attachment 2355 would go in ./attach/2/2335.anothefile.jpg and so on. In one directory, it was easier to work with.

This is all needed for $raw_filename, which now reads:

$raw_filename = $data['id'] . '.' . $data['filename'];

Apart from that, all you really need to do is match up database fields in these areas, where the second parameter is the vB field name, and the matching field from bbPress is in the $data array (which is where you make your changes):

            $try->set_value('nonmandatory', 'extension',        $data['ext']);

I guessed at this one, though, as I did not find a matching field in bbPress, but knew this should be set to "1" in vB...

            $try->set_value('nonmandatory', 'visible',            '1' );

That is pretty much all I ended up doing to get this working. Once I fixed a couple of minor errors (such as, not entering the dependent prior module 006, and found a stray ")" at the end of one of the lines), it ran the first time.

 

[Keith Hanlon]

Nicely done, Rudy! Our users will be very excited once they start using Xenforo. Thank you!

 

[Wildcat Media]

Thanks Keith! It's also nice that in your forum, and the two others I'm using XF at (our A&M board is converting also...we raised enough to get another license), we have long-time members who are understanding about these changes and trust us to choose something that will make their visits easier and more enjoyable. I've been on the "dark side" before (over at "you know who's" forum), having done upgrades and having had dozens of nitpicky complaints about every little issue under the sun.

 

[Wildcat Media]

I hit a hurdle with quoted posts. I had to run the post table through two different regular expressions on two separate passes, AND do a search/replace to clean up what was left over! Quotes are (or were) not native to bbPress, and it looks like Keith had a couple of different generations of plugins in there over the years. The format basically was like this (with varying whitespace):

<blockquote> <cite>Username - 2 hours ago <a href="....backlink to original post">&raquo;</a></cite> Quoted text here. </blockquote>

One variation had "wrote" rather than the hyphen after the username. Other quotes were "broken" inadvertently from some of the forum members while replying to the post, and others for some reason or another looked almost the same but didn't convert, which is how I came to tweak the regex to clean those up.

I did all this before going over to XF from VB. We just decided to go with a simple [ quote ] tag with username, rather than try to link back to the original post. (To be honest in all the years I've used forums, I've never clicked a quote link to go back to the original post.)

I also flagged some posts incorrectly. The "visible" field in the post and thread tables are "0" for moderated, "1" for visible and "2" for soft-deleted. It was easy for me to flip all the zeros to twos, and that got us the soft-deleted posts.

There were also a lot of stray extended characters. One was the trademark symbol (which we use a lot of in his forum ), and others appeared to be open/closing quotes, and "fancy" apostrophes. I may also have found the last few HTML entities that got left in the posts.

Overall though, it looks good! I finished the import to XF this morning. One thing, though--I thought that a fresh import overwrote everything from a previous import, but it did not do this. I ended up with my test import still remaining, and the final import tacked on. I wondered why it asked me to merge users. No big deal though--the user merges worked perfectly, and since we had cleaned up and renamed a few things in vB, I could easily tell which category was the correct one. I deleted the others, rebuilt the caches, and it's all online and looking fine now!

Our final step is to copy the database over to Keith's server, along with the attachments, and we'll be all finished. After that, some minor tweaking in XF (user and forum permissions, some remaining post/thread cleanup, etc.) are all we need to do.

The only issue I have is that the quote plugin used nested quotes. Once you get past five quotes, you end up with the post being in a vertical stack. The right side of the quote block has a wide margin. I need to find out where in the CSS this is set, and then figure out what I can add in extras.css to override it. I would like to nudge it a bit wider if I could.

Now that I'm more familiar with the cleanup scripts and now Impex works, I'm ready to tackle the next one. My next project is a combination of two forums: a WebBBS forum from 1997-2002, and phpBB2 from 2002 to present. So, if I can't import WebBBS into vB, I know I can import it into phpBB2, as I found a script that I had used on two other forums in the past. I'll leave that for a separate thread.

Finally, a handy regex tool I used for preg_replace:

http://regex.larsolavtorvik.com/

You can pretty much work out the basics here before testing in your own script. Very handy! I've always been weak in regular expressions, but I picked up a lot more this time around. (I actually used them more in Apache working with mod_rewrite, while writing scripts that had friendly URLs.)

 

[Paul B]

One thing, though--I thought that a fresh import overwrote everything from a previous import, but it did not do this. I ended up with my test import still remaining, and the final import tacked on. I wondered why it asked me to merge users.

See here: http://xenforo.com/help/importing/

Repeating the import process will result in duplicated content, including attachments and avatars. Users however will not be duplicated, due to the constraint related to user names.

 

[Wildcat Media]

I thought I recalled in the earlier versions that it started imports over by clearing out the previously imported data. Unless I missed something back then...

Well, through a comedy of errors, we lost Keith for about 20 minutes, and a scrollbar... That and we had moved smilies around and they didn't appear, only to find them working about 15 minutes later. (Page cache somewhere?)

One remaining issue is that the time on the older posts appears to be off. I can probably issue a query to add or subtract from the timestamp as needed, but I don't know how bbPress stored the timestamp--whether it was based on local time in California, or UTC time. I checked a timestamp from XF on my own server, and it works out to be UTC time.

 

[Jake Bunce]

I thought I recalled in the earlier versions that it started imports over by clearing out the previously imported data. Unless I missed something back then...

That is true of ImpEx if you don't close the session between imports. xenForo doesn't let you rerun import modules in the same session like that though.

One remaining issue is that the time on the older posts appears to be off. I can probably issue a query to add or subtract from the timestamp as needed, but I don't know how bbPress stored the timestamp--whether it was based on local time in California, or UTC time. I checked a timestamp from XF on my own server, and it works out to be UTC time.

Perhaps stating the obvious, but did you check the timezone setting of your user account in xenForo?

 

[Wildcat Media]

That is true of ImpEx if you don't close the session between imports. xenForo doesn't let you rerun import modules in the same session like that though.

Perhaps stating the obvious, but did you check the timezone setting of your user account in xenForo?

Yes, we actually did check all that--after the import (as mentioned in some other thread here), all the users' time zones were set to Atlantic time. We adjusted our profiles and corrected the board's default time zone, but we've figured out that the last post made on the old forum was around 4:31pm local time, and it shows up after our XF import as 8:31pm. I just wonder if I have to use a query to correct all of those timestamps...

 

[tictocchoc]

Sorry to deter an old topic but I am in the process of moving my board from bbpress 1.1 to XF 1.3. That is for a small board (2,000 posts) but there are attachments.

Can the guys at gConverter.com do the job? Did some of you used their services in the past?

 

[Brent W]

I have used gConverter before. They did a good job.

 

[Digital Doctor]

gConverter

http://xenforo.com/community/threads/why-there-is-no-converter-for-mybb-to-xenforo.24816/
I've seen a number of positive reports.
Even better would be using Jake Bunce or SchmitzIT.
Jake would be more expensive but would be able to fine tune the conversion if needed.
Jake thinks in binary and does regex with his cheerios just to get the juices flowing.
SchmitzIT has done lots of these conversions as well with a mountain of positive reports ...
http://xenforo.com/community/threads/schmitzit-xenforo-services.26449/

 

[tictocchoc]

It was a bbpress 1.1 with attachments to XF and the guys at gConverter did a good job for me. If the first try was not successful, they didn't give up and the second time, everything was fixed.