XF 2.2 Migration script from phorum and info about the users table

euge

New member
Hello,
I am in the process of migrating an old Phorum installation to XR (see https://xenforo.com/community/threads/migration-from-phorum-custom-email-validation.202583/ for details).

I am migrating users, I have two doubts and maybe someone can help.

1) I see that passwords are stored in a binary field in the xf_user_authenticate-data table. I guess this is the only way and there isn't any other fields where I can just store the hash of the password in plain text, right?

2) what is the "secret key" field (users table ) used for? Can I leave it blank for my imported users?

Thanks

Best,

E.
 
Last edited:
Are you intending to write an importer using our importer framework or are you just going to knock up a script to handle it?

The former is recommended, although there will obviously be a learning curve. You should be able to download the XenForo Importers add-on from your customer area which contains a number of existing importers that you can use as a reference.

The reason I suggest that approach is because while there is a learning curve it does do a lot of heavy lifting. For example the secret key would be automatically generated plus there are helpers for inserting authentication data.

Worth considering.

To answer your exact questions:

1. Yeah you will need to store them in the same format as demonstrated by the initial user account you created.

The scheme_class should be XF:Core12 (which is our handler which supports bcrypt) and data should be a PHP serialized array in this format:

PHP:
['hash' => '$2y$10$m7ys9MgXLAIO8BnDVeBpc.h7qG5E9sVl9o/cy9J4qjpULf06JRS2K']

2. Theoretically it can be used for multiple things that require a secret or unique value. One such thing is calculating an unsubscribe hash we can add to outgoing emails so that users can one-click unsubscribe from applicable users without logging in.

The secret key itself is just generated using this in our code:

PHP:
$user->secret_key = \XF::generateRandomString(32);

That function pretty much just uses PHP's random_bytes function with the specified length. We then base64_encode it and remove/convert certain problematic characters:

PHP:
$string = strtr(base64_encode($random), [
   '=' => '',
   "\r" => '',
   "\n" => '',
   '+' => '-',
   '/' => '_'
]);

How you generate that secret_key doesn't particularly matter as long as it is unique so we'd recommend generating something, ideally about 32 characters, that is unique per user.
 
Thanks for the detailed info!

My idea was to just write a script from scratch; actually I was dreaming about doing everything in SQL (except from avatar files) but that's probably just a dream.

Now that I see that your importers are so well structured I might follow your advise and create a new importer starting from an existing one.

What's the most effective way, should I just open one of the importers and try to understand it step-by-step or there is documentation available?
Do you have any idea about which importer I should start from, considering that the db schema of Phorum is much simpler than the XF's one?
 
To be honest even if we start developing a new importer we would tend to refer to one of the existing ones we've written. The code is generally easy to understand but the importer system is not documented in a step-by-step/tutorial style. My go-to importers for reference are usually MyBb, PhpBb or SMF. These three are all slightly on the simpler side though I'm not sure which one is closest to Phorum.
 
Good news.
I have found this script:
which is far from being perfect but after a few modifications I was able to convert most of the content from Phorum to phpBB and then, using your importer, to XF.
So I am on board and I hope to be able to deploy in a few days.

There are at least a couple of things that I need to fix, could you please give me some advise?

1) All the posts have been imported as "awaiting moderation" (I guess because the Phorum->phpBB script didn't work as expected). I have used the XF batch thread update tool and approved all the threads successfully, however only the threads has been approved and not the messages composing each thread, is there something I can do?

2) During the XF installation I created an admin user. The import procedure also imported the admin user I had in phorum and I set him as administrator as well. So I have now two administrators. However, the first one, from the board front-end, can see all the posts (even if they are not moderated) while the second one can only see his posts. I thought they had exactly the same permissions but maybe I am missing something.

Thanks!

Eugenio
 
1) All the posts have been imported as "awaiting moderation" (I guess because the Phorum->phpBB script didn't work as expected). I have used the XF batch thread update tool and approved all the threads successfully, however only the threads has been approved and not the messages composing each thread, is there something I can do?
The relevant code in our PhpBb importer is:

PHP:
'discussion_state' => $thread['topic_approved'] ? 'visible' : 'moderated'

And:

PHP:
'message_state' => $post['post_approved'] ? 'visible' : 'moderated',

For threads and posts respectively.

in other words, for discussion_state and message_state to be set as moderated it would suggest the Phorum to PhpBb process set them all as topic_approved = 0 and post_approved = 0.

My recommendation would be to handle this before or during the import rather than trying to fix it after the fact in XF. This would involve either; modifying the Phorum to PhpBb script accordingly or once the import to PhpBb is finished, making sure the topic_approved and post_approved fields in the respective tables are set to 1 before starting the PhpBB to XF import.

We don't have a batch update posts tool and there are associated records and counts which can be affected by the discussion_state/message_state fields in XF so it's just easier to get it right before it goes into XF.

2) During the XF installation I created an admin user. The import procedure also imported the admin user I had in phorum and I set him as administrator as well. So I have now two administrators. However, the first one, from the board front-end, can see all the posts (even if they are not moderated) while the second one can only see his posts. I thought they had exactly the same permissions but maybe I am missing something.
As you may be doing a further import from scratch as per my recommendation above, it may be worth making sure the XF admin user has the same email address and username as the original Phorum admin user, and during the configuration of the PhpBB to XF import, make sure you select the option to merge users. This will merge that Phorum admin user into the XF user seamlessly and hopefully resolve any permission issues you experienced there.

But on permissions generally, being an administrator does not necessarily give you additional power to do anything on the front end. Being an administrator grants you access to the admin control panel, and it enables you to set permissions for which areas of the admin control panel can be accessed.

In fact, being an administrator doesn't even make you a moderator, which may be different to what you're used to. Administrators and moderators are essentially separate roles.

As a general rule of thumb...

  • All standard users in XF should have their primary user group as "Registered" (regardless of what other roles they have). Permissions set on this user group should reflect the baseline permissions that all users should have as a minimum.
  • If they are a moderator they can have "Moderating" set as their secondary group (applied in addition to Registered as the primary group). Permissions set on this user group should reflect the baseline permissions that all moderators should have as a minimum.
  • If they are an administrator they can have "Administrative" set as their secondary secondary group. You can apply additional permissions here if needed but if they are a moderator as well it should only be permissions that differ from that of the Moderating user group.

If you run into any permission problems, the above should help. If the import process results in bringing over separate administrator/moderator user groups it would be better to delete those and assign the default XF ones to the relevant users rather than trying to fix the permissions that have been brought over (importing permissions and trying to make them equivalent can be messy).

If in doubt, make sure everyone is in "Registered" as their primary group, remove from all secondary user groups, and set the permissions up from scratch.
 
Thanks Chris for the detailed explanation.
About permissions: I have now set the permissions of the "second" admin (former phorum admin) exactly as the XF admin and everything works as expected on that side.

About doing again the import: the phorum->phpBB import takes a lot of time and I have already spent a considerable amount of time settings things on the XF installation. Doing everything from scratch would mean wasting hours of work. Is there anything you can advise me to do on the DB to get all the posts approved?
My solution would be to manually approve one of the post and compare the dumps of the db (before and after) to see what has changed but if you can point out the tables to modify it would be much easier.

Finally, I have noticed that for the first messages of my forum, the author in XF is often "anonymous". During the first years, registration was not compulsory but the author of the message was, the result in my XF installation, for those posts, is: id_user 0 and username "anonymous".
It is not super important but it would be nice to have the original poster's name instead of anonymous.
To mimic what I think is happening if you allow guest posting, I would leave the id_user 0 and I would modify the username according to the one I have in my phorum DB (the post IDs on phorum and XF are the same so I can easily do it with a single SQL UPDATE), am I right?

Thanks
 
About doing again the import: the phorum->phpBB import takes a lot of time and I have already spent a considerable amount of time settings things on the XF installation. Doing everything from scratch would mean wasting hours of work.
My advice will remain to be to start again to ensure the data is imported correctly.

But if you were to fix the issue in-place then after running batch update threads as you have done to affect the threads, you would run these two queries:

SQL:
UPDATE xf_post
SET message_state = 'visible'
WHERE message_state = 'moderated'

SQL:
TRUNCATE xf_approval_queue

Once you're happy with the import (make this the last thing you do) I recommend running the following rebuilds under Tools > Rebuild caches:
  • Rebuild search index
  • Rebuild threads
  • Rebuild forums
  • Rebuild user caches
  • Rebuild sitemap
If Phorum has any sort of likes/reactions system that has been imported into PhpBB and therefore now into XF you will also need to run:
  • Rebuild reactions counted status
  • Rebuild reaction score

Finally, I have noticed that for the first messages of my forum, the author in XF is often "anonymous". During the first years, registration was not compulsory but the author of the message was, the result in my XF installation, for those posts, is: id_user 0 and username "anonymous".
It is not super important but it would be nice to have the original poster's name instead of anonymous.
To mimic what I think is happening if you allow guest posting, I would leave the id_user 0 and I would modify the username according to the one I have in my phorum DB (the post IDs on phorum and XF are the same so I can easily do it with a single SQL UPDATE), am I right?
That would seem right to me. Just make sure to constrain your queries so they only work WHERE user_id = 0 AND username = 'anonymous' just to avoid any mishaps with overwriting correct data. Although I'm not sure why they'd be listed as anonymous. I presume the PhpBB importer was pulling the username from Phorum for the wrong field, or it defaults to "anonymous" if the user ID is 0.

Seems like you're nearly there. Good luck for the last few steps.
 
I have just done all the steps and everything worked as expected, there wasn't any likes/reactions system in Phorum but I rebuilt the last two caches as well to update users' points (at the moment only based on the number of messages posted).

Thanks a lot for your valuable support!!
 
Top Bottom