XF 1.5 SMF to XenForo > PM to Conversations

Mr. Jinx

Well-known member
I'm about to convert my SMF board to XenForo. After a lot of testing and tweaking we are almost ready for the big step.
There is just one thing that bugs me. The PM's in SMF are converted, but they are not visible as conversations.

For example, if I would sent a PM in SMF to a user, and that user replies, this is shown as a conversation/discussion in SMF.
The two PM's belong together.

After the conversion, these PM's are separated into 2 conversations in XenForo. The importer seems to treat every PM as a separate conversation.

Is this a limitation of the converter, or would this be to difficult to accomplish?
 

Mr. Jinx

Well-known member
I asked support about it. This is just not implemented.
Currently,the conversations import from SMF is kinda basic and limited.

It only imports the PM's a user has received. PM's a user has send (outbox) are ignored, as well as PM's from deleted users.

They have fixed most of the things right now, and are now working on a solution to keep the conversation history. (y)
That would really be a seamless conversion from SMF to XF.
 

marquisite

Well-known member
Yes the imported conversations with only the received messages is confusing - it is like eavesdropping on a telephone call where you only hear one side of the conversation.

It would appear as though the SMF Importer just needs to take into consideration the value in the column 'smf_personal_messages.id_pm_head' and merge any messages with an identical number into the one private conversation.

Did support give you any indication of when SMF conversation improvements would be released, even in beta phase, @Mr. Jinx ? I'm almost at the point of rolling out Xenforo publicly but this imported conversation issue is a bit of a bummer. :(
 

Mr. Jinx

Well-known member
Yes, two important things about this:
  • Send PM's and PM's from/to deleted users are currently not imported. XF Support provided fixes for this (after a lot of trial and error) and pushed the fixes to the development team which will review it for a next release (maybe). No timeline or promesses. I'll share those fixes when I'm done. I'm currently in a migration :)
  • Keeping the conversation 'groups' will not be implemented because this works different in XF. There was no easy migration possible.
    Example: In SMF, user1 can start a conversation with user2. Then at some point, user1 can add user3 to the conversation. At that point, user3 cannot see the previous conversation between user1 & user2. In fact, user3 could reply to only user2 within the same conversation. In XF, all users (1, 2 and 3) will be able to see the whole conversation. If you would convert this situation, some users might see conversations they could not see in the past. This might be dangerous. Therefore I just accepted the fact we will loose the grouped conversations.

    ps: I got a very neat modified script from another user that made it possible to keep the grouped conversations, but it still has the problem described above. If you accept that risk, I can share that as well.
 

marquisite

Well-known member
Interesting info and valid concerns with privacy (who knows what goes on in PM ;)?). Perhaps some checking could be done that would only group messages with the same id_pm_head number that involve two (2) user ids, and keep any three (3) or more ids separated for privacy reasons?

Nevertheless even without the grouping at least having sent & received messages separately would be much better than the current one-sided situation. Let me know how you go :). I'm guessing the next Xenforo patch will be some time away from release and won't allow for an April migration of my forum.
 

Mr. Jinx

Well-known member
These are the fixes I received from support regarding the import of PM's.
It should fix the missing import of send PM's and PM's from/to deleted users (guests).
This worked fine for me.

XF Version 1.5.13

In library/xenforo/Importer/SMF.php

Find:
PHP:
            if (!$toUsers)
            {
                continue;
            }
Replace with:
PHP:
            if (!$toUsers AND !$pmText['id_member_from'])
            {
                continue;
            }


Find:
PHP:
            $newFromUserId = $this->_mapLookUp($mapUserIds, $pmText['id_member_from']);
            if (!$newFromUserId)
            {
                continue;
            }
Replace with:
PHP:
            $newFromUserId = $this->_mapLookUp($mapUserIds, $pmText['id_member_from']);
            if (!$newFromUserId)
            {
                $newFromUserId = 0;
                //continue;
            }


Find:
PHP:
            $unreadState = $sDb->fetchPairs('
                SELECT id_member, is_read
                FROM ' . $prefix . 'pm_recipients
                WHERE id_pm = ' . $sDb->quote($pmText['id_pm']) . '
                    AND deleted = 0
                GROUP BY id_member
            ');
Add after:
PHP:
            $deletedRecipients = $sDb->fetchPairs('
                SELECT id_member, deleted
                FROM ' . $prefix . 'pm_recipients
                WHERE id_pm = ' . $sDb->quote($pmText['id_pm']) . '
                    AND deleted = 1
                    GROUP BY id_member
                UNION
                SELECT id_member_from, deleted_by_sender
                FROM ' . $prefix . 'personal_messages
                    WHERE id_pm = ' . $sDb->quote($pmText['id_pm']) . '
                    AND deleted_by_sender = 1
                    GROUP BY id_member_from
            ');


Find:
PHP:
                else
                {
                    $lastReadDate = $pmText['msgtime'];
                    $deleted = true;
                    $isUnread = 0;
                }
Replace with:
PHP:
                else
                {
                    $lastReadDate = $pmText['msgtime'];
                    // $deleted = true;
                    $deleted = isset($deletedRecipients[$userId]) ? true : false;
                    $isUnread = 0;
                }
 

marquisite

Well-known member
These are the fixes I received from support regarding the import of PM's.
It should fix the missing import of send PM's and PM's from/to deleted users (guests).
This worked fine for me.

XF Version 1.5.13
Thanks!

I was thinking some more about the access issues with grouping conversations (where recipients can view PMs within the conversation that they weren't included in). I'm not sure about you but I generally use PMs only on a 1-to-1 basis, which means a maximum of 2 participants in a conversation. If the SMF -> Xenforo importer could group PMs into conversations only in cases where there are 2 participants, then I think this would prevent privacy issues with group PM conversations (3+ participants) while allowing some PM conversations to be grouped, limited to those that are 1-to-1 (with max 2 participants).
 

Overscan

Active member
ps: I got a very neat modified script from another user that made it possible to keep the grouped conversations, but it still has the problem described above. If you accept that risk, I can share that as well.
I'd be interested in a copy of this please! I don't have any three way conversations to speak of in my database, so I'm happy to take this risk.
 

Mr. Jinx

Well-known member
I forgot about this, aaarrggh what a nightmare it was. Importing the SMF PM's to XF conversations.
Tested it over and over again.
In the end, I converted everything as standalone messages. Nobody ever complained!

I'll dig up that script for you and post it over here.
 

Mr. Jinx

Well-known member
So, this is the modification to convert SMF pm's to real XF conversations.
Make sure to do some serious testing before using this for real.
All credits go to @GeorgWin for this!

edit: see below
 
Last edited:

GeorgWin

Member
Corrected version, I used it myself:

Open ../library/XenForo/Importer/SMF.php and replace the function stepPrivateMessages:
Code:
    public function stepPrivateMessages($start, array $options)
    {
        $options = array_merge(array(
            'limit' => 100,
            'pmDateStart' => 0,
            'pmLimit' => 800,
            'max' => false
        ), $options);

        $sDb = $this->_sourceDb;
        $prefix = $this->_prefix;

        /* @var $model XenForo_Model_Import */
        $model = $this->_importModel;

        if ($options['max'] === false)
        {
            $options['max'] = $sDb->fetchOne('
                SELECT MAX(id_pm_head)
                FROM ' . $prefix . 'personal_messages
                WHERE id_pm_head = id_pm AND id_member_from != 0
            ');
        }       
    
        $conversations = $sDb->fetchAll($sDb->limit(
            '
            SELECT *
            FROM ' . $prefix . 'personal_messages
            WHERE id_pm_head >= ' . $sDb->quote($start) . '
                AND id_pm_head = id_pm AND id_member_from != 0
            ORDER BY id_pm_head
            ', $options['limit']
        ));       
        
        if (!$conversations)
        {
            return true;
        }

        $next = 0;
        $total = 0;
        $totalPosts = 0;

        XenForo_Db::beginTransaction();   
        
        foreach ($conversations AS $conversation)
        {
            
            /* Количество ответов в каждой переписке */
            $countPost = $sDb->fetchOne('
                SELECT COUNT(*)
                FROM ' . $prefix . 'personal_messages
                WHERE id_pm_head = ' . $conversation['id_pm_head'] . '
            ');   

            $reply_count = $countPost - 1;

            /* Выборка данных из первой строки в диалоге */
            $pmDataFirstSql = $sDb->fetchAll('
                SELECT *
                FROM ' . $prefix . 'personal_messages
                WHERE id_pm_head = ' . $sDb->quote($conversation['id_pm_head']) . '
                ORDER BY msgtime ASC LIMIT 1'
            );               

            $pmdata = array();
            foreach ($pmDataFirstSql AS $data)
            {
                # Первое сообщение в переписке: first_post
                # Дата первого сообщения: start_date
                $pmdata = array(
                    'first_post' => $data['id_pm'],
                    'start_date' => $data['msgtime']
                );
            }

            /* Выборка данных из последней строки в диалоге */
            $pmDataLastSql = $sDb->fetchAll('
                SELECT *
                FROM ' . $prefix . 'personal_messages
                WHERE id_pm_head = ' . $sDb->quote($conversation['id_pm_head']) . '
                ORDER BY msgtime DESC LIMIT 1'
            );

            foreach ($pmDataLastSql AS $data)
            {
            # ID замыкающего диалог пользователя (последний): last_message_user_id
            # Ник замыкающего диалог пользователя (последний): last_message_username
            # Последнее сообщение в переписке: last_post
            # Дата последнего сообщения в диалоге: last_read_date
                $pmdata += array(
                    'last_message_user_id' => $data['id_member_from'],
                    'last_message_username' => $data['from_name'],
                    'last_post' => $data['id_pm'],
                    'last_read_date' => $data['msgtime']
                );
            }       

            /* Участники диалога */
            $toUsers = $sDb->fetchPairs('
                SELECT recip.id_member, member.member_name
                FROM ' . $prefix . 'pm_recipients AS recip
                INNER JOIN ' . $prefix . 'members AS member ON
                    (recip.id_member = member.id_member)
                WHERE recip.id_pm = ?
            ', $conversation['id_pm_head']);
        
            if (!$toUsers)
            {
                continue;
            }

            $users = array(
                $conversation['id_member_from'] => $conversation['from_name']
            ) + $toUsers;

            $mapUserIds = $model->getImportContentMap('user', array_keys($users));

            $newFromUserId = $this->_mapLookUp($mapUserIds, $conversation['id_member_from']);

            if (!$newFromUserId)
            {
                continue;
            }

            $fromUserName = $this->_convertToUtf8($conversation['from_name'], true);

            /* Количество получателей в переписке */
            $recipient_count = count($users);

            /* Захват сообщений */

            $pmDateStart = $options['pmDateStart'];

            $next = $conversation['id_pm_head'] + 1; // uses >=, will be moved back down if need to continue
            $options['pmDateStart'] = 0;

            $maxPosts = $options['pmLimit'] - $totalPosts;

            $postsPm = $sDb->fetchAll($sDb->limit(
                '
                    SELECT *
                    FROM ' . $prefix . 'personal_messages
                    WHERE id_pm_head = ' . $sDb->quote($conversation['id_pm_head']) . '
                        AND msgtime > ' . $sDb->quote($pmDateStart) . '
                    ORDER BY id_pm
                ', $maxPosts
            ));

            if (!$postsPm)
            {
                if ($pmDateStart)
                {
                    // continuing conversation but it has no more posts
                    $total++;
                }
                continue;
            }

            /* Импорт конференции */
            if ($pmDateStart)
            {
                // continuing conversation we already imported
                $conversationId = $model->mapConversationId($conversation['id_pm_head']);   
            }
            else
            {           
                /* Формирование конференции */
                $import = array(
                    'recipient_count' => $recipient_count,
                    'reply_count' => $reply_count,
                    'start_date' => $pmdata['start_date'], #
                    'last_message_id' => $pmdata['last_post'], #
                    'last_message_date' => $pmdata['last_read_date'], #
                    'last_message_user_id' => $pmdata['last_message_user_id'],
                    'last_message_username' => utf8_substr($pmdata['last_message_username'], 0, 50), #
                    'title' => $this->_convertToUtf8($conversation['subject'], true),
                    'user_id' => $newFromUserId,
                    'username' => $fromUserName,
                    'open_invite' => 0,
                    'conversation_open' => 1
                );               

                /* Делает выборку всех последних ЛС (включая удалённые), - прочтённые непрочтённые */           
                $unreadState = $sDb->fetchPairs('
                    SELECT rec.id_member, rec.is_read
                    FROM ' . $prefix . 'pm_recipients AS rec
                    LEFT JOIN ' . $prefix . 'personal_messages as pm ON (pm.id_pm = rec.id_pm)
                    WHERE pm.id_pm_head = ' . $sDb->quote($conversation['id_pm_head']) . '
                    ORDER BY pm.id_pm DESC LIMIT 1
                ');
                    
                $recipients = array();
                foreach ($users AS $userId => $username)
                {
                    $newUserId = $this->_mapLookUp($mapUserIds, $userId);

                    if (!$newUserId)
                    {
                        continue;
                    }

                    /* Вернём все удалённые обратно и правильно отметим непрочитаные перенесенные переписки (если надо, пользователи удалят снова, а то каша получается из переписок) */
                    if (isset($unreadState[$userId]))
                    {   
                
                        $lastReadDate = $unreadState[$userId] ? $pmdata['last_read_date'] :  0;
                        $deleted = false;
                        $isUnread = $unreadState[$userId] == 0 ? 1 : 0;
                    }
                    else
                    {
                        
                        $lastReadDate = $pmdata['last_read_date'];
                        $deleted = false;
                        $isUnread = 0;
                    }   


                    $recipients[$newUserId] = array(
                        'username' => $this->_convertToUtf8($username, true),
                        'last_read_date' => $lastReadDate,
                        # Отмечаем все ЛС как активные
                        'recipient_state' => ($deleted ? 'deleted' : 'active'),
                        'is_unread' => $isUnread
                    );
                }

                $conversationId = $model->importConversation($conversation['id_pm_head'], $recipients, $import);

                if (!$conversationId)
                {
                    continue;
                }               
                
            }

            /* Импорт сообщений конференции */
            if ($conversationId)
            {
                $userIdMap = $model->getUserIdsMapFromArray($postsPm, 'id_member_from');
        
                foreach ($postsPm AS $i => $postPm)
                {   
                    $messages = array(
                        array(
                            'conversation_id' => $conversationId,
                            'message_date' => $postPm['msgtime'],
                            'user_id' => $this->_mapLookUp($userIdMap, $postPm['id_member_from'], 0),
                            'username' => $this->_convertToUtf8($postPm['from_name'], true),
                            'message' => $this->_sanitizeBbCode($postPm['body'])
                        )
                    );

                    $model->importConversationPost($postPm['id_pm'], $messages);

                    $options['pmDateStart'] = $postPm['msgtime'];
                    $totalPosts++;
                }

                if (count($postsPm) < $maxPosts)
                {
                    // done this conversation
                    $total++;
                    $options['pmDateStart'] = 0;
                }
                else
                {
                    // not necessarily done the conversation; need to pick it up next page
                    break;
                }
            }       

            if (count($postsPm) < $maxPosts)
            {
                // done this conversation
                $total++;
                $options['pmDateStart'] = 0;
            }
            else
            {
                // not necessarily done the conversation; need to pick it up next page
                break;
            }
        }

        if ($options['pmDateStart'])
        {
            // not done this thread, need to continue with it
            $next--;
        }

        XenForo_Db::commit();

        $this->_session->incrementStepImportTotal($total);

        return array($next, $options, $this->_getProgressOutput($next - 1, $options['max']));
    }
Open ../library/XenForo/Model/Import.php and replace the function importConversation:
Code:
    /**
     * Maps an old conversation ID to a new/imported thread ID
     *
     * @param integer $id
     * @param integer $default
     *
     * @return integer
     */
    public function mapConversationId($id, $default = null)
    {
        $ids = $this->getImportContentMap('conversation', $id);
        return ($ids ? reset($ids) : $default);
    }   

    /**
     * Imports private messages etc. into a XenForo conversation
     *
     * @param integer Source ID
     * @param array $conversation Data for XenForo_DataWriter_ConversationMaster
     *
     * @return integer Imported conversation ID
     */
    public function importConversation($oldId, array $recipients, array $conversation)
    {
        $db = $this->_getDb();
        XenForo_Db::beginTransaction($db);

        $conversationId = $this->_importData($oldId, 'XenForo_DataWriter_ConversationMaster', 'conversation', 'conversation_id', $conversation);
    
        foreach ($recipients AS $userId => $info)
        {
            $db->insert('xf_conversation_recipient', array(
                'conversation_id' => $conversationId,
                'user_id' => $userId,
                'recipient_state' => $info['recipient_state'],
                'last_read_date' => $info['last_read_date']
            ));

            if ($info['recipient_state'] == 'active')
            {
                if (isset($info['is_unread']))
                {
                    $isUnread = $info['is_unread'];
                }
                else
                {
                    $isUnread = ($info['last_read_date'] >= $lastMessage['message_date'] ? 0 : 1);
                }

                $recipientUser = array(
                    'conversation_id' => $conversationId,
                    'owner_user_id' => $userId,
                    'is_unread' => $isUnread,
                    'reply_count' => $conversation['reply_count'],
                    'last_message_date' => $conversation['last_message_date'],
                    
                     # При выборке каждого нового ЛС в диалоге, эти поля обновляется актуальными данными в функции importConversationPost
                    'last_message_id' => $conversation['last_message_id'],
                    'last_message_user_id' =>  $conversation['last_message_user_id'],
                    'last_message_username' => $conversation['last_message_username'],
                );

                $db->insert('xf_conversation_user', $recipientUser);
            }
        }   

        XenForo_Db::commit($db);       
        return $conversationId;
    }

    /**
     * Imports private messages etc. into a XenForo conversation
     *
     * @param integer Source ID
     * @param array $messages Data for XenForo_DataWriter_ConversationMessage
     *
     * @return integer Imported message ID
     */   
    public function importConversationPost($oldId, array $messages)
    {
        if (!$messages)
        {
            return false;
        }

        $db = $this->_getDb();
        XenForo_Db::beginTransaction($db);
        
        
        $firstMessageID = 0;
        $lastMessage = array();

        foreach ($messages AS $messageId => $message)   
        {
            $conversationId = $message['conversation_id'];
            $messageId = $this->_importData($oldId, 'XenForo_DataWriter_ConversationMessage', 'conversation_message', 'message_id', $message);
            if (!$messageId)
            {
                continue;
            }

            $message['message_id'] = $messageId;
            $lastMessage = $message;
        }

        $conversationUserUpdate = array(
            'last_message_id' => $lastMessage['message_id'],
            'last_message_user_id' => $lastMessage['user_id'],
            'last_message_username' => utf8_substr($lastMessage['username'], 0, 50)
        );

        $db->update('xf_conversation_user', $conversationUserUpdate, 'conversation_id = ' . $db->quote($conversationId));

        $firstMessageID = $this->_getDb()->fetchOne('
            SELECT message_id
            FROM xf_conversation_message
            WHERE conversation_id = ?
            ORDER BY message_id ASC LIMIT 1
        ', $conversationId);
        
        $conversationMasterUpdate = array(
            'first_message_id' => $firstMessageID
        );

        $conversationMasterUpdate += $conversationUserUpdate;

        $db->update('xf_conversation_master', $conversationMasterUpdate, 'conversation_id = ' . $db->quote($conversationId));
    

        XenForo_Db::commit($db);
        return $messageId;
    }
 

Overscan

Active member
Thanks so much for that GeorgWin. It worked perfectly, a massive improvement on the default import. I recommend this be included in the default importer.
 

Overscan

Active member
Ooops - it seems I spoke too soon. A user had a giant (few hundreds of messages) PM conversation with another user - this is gone from the imported forum. Are there limits on the number of pms imported in a conversation? Both users still exist on the forum.
 

Mr. Jinx

Well-known member
Nice, that was probably the most safe way and as I said before, nobody ever complained about loosing the threads (on my forum).
So you finally did the conversion for real (y)
 

Overscan

Active member
Nice, that was probably the most safe way and as I said before, nobody ever complained about loosing the threads (on my forum).
So you finally did the conversion for real (y)
Yep, over the line at last. Can't believe I renewed my license before I got it done.
 
Top