Writing a new importer. Tips please

[Andrew van der Stock]

Hi there,

I've searched, but there are no obvious tutorials or posts that can help me.

I've previously bought XenForo 1.0 just after it came out, but didn't have the time necessary back then to complete an importer for the ultimate son of XMB -> UltimaBB -> GaiaBB. So I've just re-bought XenForo 1.2.4.

On my main forum, I have about 15,600 users, nearly 900,000 posts, 300,000 PMs, just shy of 90,000 threads and around 83,000 attachments, and growing daily. So I need a very solid and relatively fast importer. The good news is that as an ex-developer of XMB, UltimaBB, and GaiaBB, I do know my way around PHP and MySQL, but not so much around Zend or XenForo.

Can someone who has written a comprehensive importer from scratch give me some tips on the order of importing. My guess is:

• Ranks (as these are used by the membership table)
• Members
• Create a password adapter (XMB, UltimaBB, and GaiaBB use just MD5, which was due to be fixed, but in my infinite spare time)
• Forums
• Forum permissions (We have around 70 forums, including a shared password forum, restricted suspended posts and moderator's paradise forums), so getting a "safe" out of the box experience would be best rather than leaving it to post-installation)
• Threads
• Attachments (ours are in the database, which will need to be migrated to the file system by the looks of things)
• Posts (to allow fixups to the attachments)
• Polls (these are stored in exactly the same way as phpBB in UltimaBB and GaiaBB)
• PM attachments (ditto to the above)
• PMs (this is kludge city, and will probably require the most massaging)

Optionals:

• Favorites / Subscriptions (we have 18,000 favorites/subscriptions)
• Avatars and Photos (at least these are already on disk)
• Smilies (we have a large number of smilies used by our members)
• Address Book (it's not a heavily used feature and relies on usernames rather than userIDs)

I've installed XenForo, and my GaiaBB test forum on my devbox, which is a fraction of the size of my main forum. I've got them ready to restore back to scratch as I bet I will need to do this a bunch of times. Once the importer is mostly ready, I'll do a few trial imports of the real deal, as the main forum is a long term survivor of XMB 1.8 -> 1.9 -> UltimaBB -> XMB 1.9.11 -> UltimaBB -> GaiaBB.

Out of all the existing importers, which is the cleanest one I can borrow code from?

I could assist in the creation of an XMB importer once I've got UltimaBB/GaiaBB done (they're very close, depending on the schemaver, the main differences in settings and how polls work).

thanks
Andrew

 

[Andrew van der Stock]

I forgot settings. UltimaBB has a zillion features, and so about 200 settings values, many of which would be nice to carry over to XenForo if they exist. I do like the chance to de-clutter functionality if possible.

 

[Andrew van der Stock]

So I've decided to make sure that we get an XMB convertor out of this as well, by using the vB importer as an example of a class that is then overridden by particular classes that operate ever so slightly differently (a la config/settings and poll voting). So I've created the following:

class XenForo_Importer_Xmb extends XenForo_Importer_Abstract

Most of the importer business logic will be here, as there's not a lot of change between XMB and GaiaBB for the things we're importing.

class XenForo_Importer_Xmb19 extends XenForo_Importer_Xmb

This will have the old style settings importer, the older poll options and results decoder (it's horrible!), and not bother importing PM attachments. There are fewer forum permission options (sample: 1|1|1) to decode and apply.

And finally

class XenForo_Importer_GaiaBB extends XenForo_Importer_Xmb

This will contain the GaiaBB specific stuff, like the newer key / value settings importer, improved forum permissions (sample: 1|1|1|1|1), and the phpBB style vote importer.

I think the phpBB importer will be more help in writing these importers than the vB importer, as XMB/UltimaBB/GaiaBB seems to be nothing at all like vB under the hood. Not that XMB/GaiaBB is much like phpBB.

I think early adopter UltimaBB adopters, if any remaining at all, should have reasonable luck with the XMB importer if I think I'm getting the gist of how the XenForo importer steps work, as early UltimaBB (up to about 2005 or 2006 or so) had XMB permissions, settings, and polls. Late model UltimaBB forums should upgrade just fine with the GaiaBB importer as I'm delving into settings I know exist in most if not all versions of UltimaBB. I've got one or two dusty backups from ultimabb.com that should let me know if this is the case or not, so I'm not going to write a separate UltimaBB importer for the time being as I'm not sure if there's any of active forums left. Plus, I'm lazy and I don't have a lot of time.

Observations: I don't see namespaces anywhere, but Zend Studio is littering my code with it. I'll continue to trim namespace support out. However, I do think PHP 5.2.x days are done.

Question: There's a vB3/4 Big Board CLI script importer for $150 that uses PCNTL (process control) and command line PHP to do its thing. Is there any documentation on how this might work within the 1.2.4 code base, just in case I need to adopt a similar strategy for my big forum? As my 5 GB database backup takes nearly an hour and a half to back up on my VPS, I can't imagine it will convert quickly via the web interface.

 

[Andrew van der Stock]

Hmm, I get the parent class in the list of potential imports.



If I leave out getName() from the parent class, it throws an exception. Is there a way to avoid this class showing up? It's not going to be a good XMB convertor on its own.

Edit: I've simply marked it as "zz don't use this" for the time being. The other alternative I've thought about is to simply make XenForo_Importer_Xmb the actual XMB 1.9.11 importer, and override necessary steps in XenForo_Importer_GaiaBB with GaiaBB specific implementations.

 

[Andrew van der Stock]

I've rolled XMB19 and Xmb.php into one thus eliminating the issue with having a file just lying around.

And then I found 1.9.11.13 (the last ever patch for XMB) uses UltimaBB/GaiaBB's/phpBB's voting system. Interesting. Means we might be able to do this thing using a single file, which will be nice.

 

[Andrew van der Stock]

Today, I installed a bunch of other forums for which importers exist to learn how their importers work versus the schema and files on disk.

Today's question - I need to figure out how to create a view without writing an add on. Or should I write an add on, like the punBB / fluxBB importers?

Trail of breadcrumbs for those who want to write an importer for XenForo, so you don't need to re-discover the same thing over and over again.

It turns out my fading memory was jogged - myBB is a long lost descendant of XMB pre history (1.5, 1.6 or 1.8 or thereabouts), and although myBB looks somewhat similar to XMB even today, it's very different under the hood in terms of code. That said, our schemas are nearly identical for the things I want to keep, so it might be a good choice as donor importer.

CREATE TABLE `mybb_forums` (
  `fid` smallint(5) unsigned NOT NULL AUTO_INCREMENT,
  `name` varchar(120) NOT NULL DEFAULT '',
  `description` text NOT NULL,
  `linkto` varchar(180) NOT NULL DEFAULT '',
  `type` char(1) NOT NULL DEFAULT '',
  `pid` smallint(5) unsigned NOT NULL DEFAULT '0',
  `parentlist` text NOT NULL,
  `disporder` smallint(5) unsigned NOT NULL DEFAULT '0',
  `active` int(1) NOT NULL DEFAULT '0',
  `open` int(1) NOT NULL DEFAULT '0',
  `threads` int(10) unsigned NOT NULL DEFAULT '0',
  `posts` int(10) unsigned NOT NULL DEFAULT '0',
  `lastpost` int(10) unsigned NOT NULL DEFAULT '0',
  `lastposter` varchar(120) NOT NULL DEFAULT '',
  `lastposteruid` int(10) unsigned NOT NULL DEFAULT '0',
  `lastposttid` int(10) NOT NULL DEFAULT '0',
  `lastpostsubject` varchar(120) NOT NULL DEFAULT '',
  `allowhtml` int(1) NOT NULL DEFAULT '0', # We got rid of this around c.2006 as it's XSS baby
  `allowmycode` int(1) NOT NULL DEFAULT '0', # BBCode, in other words
  `allowsmilies` int(1) NOT NULL DEFAULT '0',
  `allowimgcode` int(1) NOT NULL DEFAULT '0',
  `allowvideocode` int(1) NOT NULL DEFAULT '0',
  `allowpicons` int(1) NOT NULL DEFAULT '0', # Post icons
  `allowtratings` int(1) NOT NULL DEFAULT '0',
  `status` int(4) NOT NULL DEFAULT '1',
  `usepostcounts` int(1) NOT NULL DEFAULT '0',
  `password` varchar(50) NOT NULL DEFAULT '',
...

And then GaiaBB:

CREATE TABLE `gbb_forums` (
  `type` varchar(15) NOT NULL DEFAULT '',
  `fid` smallint(6) NOT NULL AUTO_INCREMENT,
  `name` varchar(50) NOT NULL DEFAULT '',
  `status` set('on','off') NOT NULL DEFAULT 'on',
  `lastpost` int(7) NOT NULL DEFAULT '0',
  `moderator` varchar(100) NOT NULL DEFAULT '',
  `displayorder` smallint(6) NOT NULL DEFAULT '0',
  `private` varchar(30) DEFAULT '1', # We still have old style per-forum permissions
  `description` text,
  `allowsmilies` set('yes','no') NOT NULL DEFAULT 'yes',
  `allowbbcode` set('yes','no') NOT NULL DEFAULT 'yes',
  `userlist` text NOT NULL, # Optionally, allow a list of users to access a forum as we don't have groups
  `theme` smallint(3) NOT NULL DEFAULT '0', # We support per forum theming. Crazy but true.
  `posts` int(100) NOT NULL DEFAULT '0',
  `threads` int(100) NOT NULL DEFAULT '0',
  `fup` smallint(6) NOT NULL DEFAULT '0',  # Known as parentforum in myBB, but identical otherwise
  `postperm` varchar(7) NOT NULL DEFAULT '', # We still have old style per-forum permissions
  `allowimgcode` set('yes','no') NOT NULL DEFAULT 'yes',
  `attachstatus` set('on','off') NOT NULL DEFAULT 'on', # Permissions to allow attachments within the forum
  `pollstatus` set('on','off') NOT NULL DEFAULT 'on', # Permissions to allow polls within the forum
  `password` varchar(32) NOT NULL DEFAULT '',
  `guestposting` set('on','off') NOT NULL DEFAULT 'off', # GaiaBB no longer supports guest posting, but it's still in our schema
  `minchars` smallint(5) NOT NULL DEFAULT '0', # Mininum number of characters for a post within a forum. 
  `attachnum` tinyint(2) NOT NULL DEFAULT '3', # Max attachments per post in this forum
  [ ... ] # More features options removed
  `postcount` set('on','off') NOT NULL DEFAULT 'on', # "usepostcounts" in myBB

So as you can see, GaiaBB is a moderately close relative of myBB, which is excellent as I now have a reasonable donor for my importer.

I've started fleshing out the configure() and validateConfiguration() methods to inspect XMB's / GaiaBB's config.php file. GaiaBB did _everything_ within the settings table. The config.php file is literally a pointer to a database. Luckily, they share the same naming standards, so it should be no trouble to share this in the parent class. Looking through XMB 1.9.11.13, they still use old xmb_settings table, so I will need to have two importers, one which overrides the xmb_settings table in favor of a key/pair solution. They use much the same nomenclature once read in though.

 

[tyteen4a03]

Hmm, I get the parent class in the list of potential imports.

View attachment 63874

If I leave out getName() from the parent class, it throws an exception. Is there a way to avoid this class showing up? It's not going to be a good XMB convertor on its own.

Edit: I've simply marked it as "zz don't use this" for the time being. The other alternative I've thought about is to simply make XenForo_Importer_Xmb the actual XMB 1.9.11 importer, and override necessary steps in XenForo_Importer_GaiaBB with GaiaBB specific implementations.

That is how you are supposed to do it. See X_I_vBulletin, X_I_vBulletin4 and X_I_vBulletin36.

Hi there,

I've searched, but there are no obvious tutorials or posts that can help me.

I've previously bought XenForo 1.0 just after it came out, but didn't have the time necessary back then to complete an importer for the ultimate son of XMB -> UltimaBB -> GaiaBB. So I've just re-bought XenForo 1.2.4.

On my main forum, I have about 15,600 users, nearly 900,000 posts, 300,000 PMs, just shy of 90,000 threads and around 83,000 attachments, and growing daily. So I need a very solid and relatively fast importer. The good news is that as an ex-developer of XMB, UltimaBB, and GaiaBB, I do know my way around PHP and MySQL, but not so much around Zend or XenForo.

Can someone who has written a comprehensive importer from scratch give me some tips on the order of importing. My guess is:

• Ranks (as these are used by the membership table)
• Members
• Create a password adapter (XMB, UltimaBB, and GaiaBB use just MD5, which was due to be fixed, but in my infinite spare time)
• Forums
• Forum permissions (We have around 70 forums, including a shared password forum, restricted suspended posts and moderator's paradise forums), so getting a "safe" out of the box experience would be best rather than leaving it to post-installation)
• Threads
• Attachments (ours are in the database, which will need to be migrated to the file system by the looks of things)
• Posts (to allow fixups to the attachments)
• Polls (these are stored in exactly the same way as phpBB in UltimaBB and GaiaBB)
• PM attachments (ditto to the above)
• PMs (this is kludge city, and will probably require the most massaging)

Optionals:

• Favorites / Subscriptions (we have 18,000 favorites/subscriptions)
• Avatars and Photos (at least these are already on disk)
• Smilies (we have a large number of smilies used by our members)
• Address Book (it's not a heavily used feature and relies on usernames rather than userIDs)

I've installed XenForo, and my GaiaBB test forum on my devbox, which is a fraction of the size of my main forum. I've got them ready to restore back to scratch as I bet I will need to do this a bunch of times. Once the importer is mostly ready, I'll do a few trial imports of the real deal, as the main forum is a long term survivor of XMB 1.8 -> 1.9 -> UltimaBB -> XMB 1.9.11 -> UltimaBB -> GaiaBB.

Out of all the existing importers, which is the cleanest one I can borrow code from?

I could assist in the creation of an XMB importer once I've got UltimaBB/GaiaBB done (they're very close, depending on the schemaver, the main differences in settings and how polls work).

thanks
Andrew

Look at the import order of other XF importers - you see that user group is always the first. The thing is that forum tables and data usually depend on other tables/data and you want to recalculate as little as possible for the import, so you want to lay out a "battle plan" to see which should be attacked first.

For example, the forum table in your GaiaDB example depends on users data (userlist, moderators field), and the user table depend on user groups. So, to import these correctly you would have to first import the user group section, then the user section, then finally the forum section.

One import step can unlock multiple other steps (by importing user groups and users you can pretty much do everything else not posts related) and sometimes you need to take out a part of the step to fit XenForo's style (e.g Moderators is a step of its own because moderators are stored in a separate table, therefore to avoid code clutter you put this in a separate step).

To see how XenForo's bundled importer handle dependency, see their getSteps() function. It's just a puzzle of which comes first.

 

[Andrew van der Stock]

We don't have groups, per se. We have "status", which is one of "Banned", "Member", "Moderator", "Super Moderator", "Admin" and "Super Admin". I'm happy enough to translate these to the XenForo default groups.

On top of that we have ranks, which is sort of equivalent to XenForo Trophies, but solely granted for post count. I will have to work on that a bit as our members worked hard for their rank status.

 

[tyteen4a03]

We don't have groups, per se. We have "status", which is one of "Banned", "Member", "Moderator", "Super Moderator", "Admin" and "Super Admin". I'm happy enough to translate these to the XenForo default groups.

On top of that we have ranks, which is sort of equivalent to XenForo Trophies, but solely granted for post count. I will have to work on that a bit as our members worked hard for their rank status.

Obviously don't import super admins directly (they're $config-only for a reason).

As for ranks, from your use case I think it would be better to import them into my addon. Importers are not supposed to "interpret" features and their use cases (or at least keep it to a minimum) - these are left for the admins to fix.

 

[Ntown]

Sorry to bump an ancient thread but this is the only thread I can find even close to what I need.

Question: Is there an importer for XMB to xf?

 

[Paul B]

There is not.

It would require custom developing one, or doing a two stage import to software first for which there is an importer.