Resource icon
vBulletin Big Board Importer [vBulletin 3 + vBulletin 4] [Paid] 1.5.0
- Thread starter Slavik
- Start date
Thanks for the info.
After some time I now have a working Export/Import. However, I've noticed an issue with custom avatars. For some users the avatar is not importing correctly. If on vB 3.8 they had a .gif custom avatar, then on the Import they have a .jpg avatar and the image is broken. Any ideas?
Jake Bunce
Well-known member
The importer reads the image type from the image data, so it is possible for incorrect file extensions to be discovered through the import.
A broken image presumably means the import failed. Look at the output of the Export script for a specific error message on that avatar during the export process.
Ok, thanks. The export log shows entries like "data/avatars/s/0/140.jpg => No move" for the avatars that are not being imported. There is about 500 out of 32,000 so not that bad I guess. I did note that using the GD vs imagemagick greatly improved the speed of the export, 17,963.15s vs 741.57s. Quality appears the same.
The only remaining issue that I am having is with the attachments. I've run the export and the import and it appears the 360,000 attachments have been imported. However, I noticed that there were no thumbs. So, I try to rebuild the attachment thumbs. However, it just rebuilds the first 178 and then keeps running but not rebuilding the thumbs. In the xf_attachment_data table I can see that the attachments are there, but after 178 there are no width, height, thumbnail width or thumbnail height. In the data/attachments folder I only get the 0 directory, no further directories are created and no errors are thrown. I've tried with both GD and imagemagick. I've checked permissions on the directories. Any idea what is going on?
Edit: I've confirmed that the problem is on the Export. Starting fresh, after the first 178 attachments are processed, the thumbs stop getting created and the xf_attachment_data table shows only 0 for width, height, thumbnail width and thumbnail height.
Edit 2: After much investigation, I realized that the thumbnail_filesize field in vb_attachment was set to 0 for most of the attachments. Not sure how/why as thumbnails are functional in vB. So, in Export.php the if statement "if ($attachment['thumbnail_filesize'] > 0 AND $attachment['extension'] != 'bmp')" was excluding these attachments from getting their dimensions which apparently broke XFs ability to generate a new thumbnail. After removing this condition, it appears to be working. I can't see any use for the thumbnail_filesize condition unless it is just to check if a thumbnail exists?
Last edited:
Jake Bunce
Well-known member
That should be fine. The thumbnail condition is meant to detect images, but after that it detects again by getting the image info. So the image processing might take additional time for non-images, but attachments are incremental so it doesn't really matter.
arn
Well-known member
@Jake Bunce or anyone...
We did our import -- which overall went smoothly, I'll post more about that here -- but since then, we've had users have been complaining that when someone replies to a pre-xenforo conversation, they receive an email with the conversation reply. But when they click on the link, they get a "The requested conversation could not be found" error.
Looking up the conversation id in the database, it appears to be there. So, it seems like a permissions issue. I've rebuilt conversations to see if that would fix it. But it hasn't.
arn
We did our import -- which overall went smoothly, I'll post more about that here -- but since then, we've had users have been complaining that when someone replies to a pre-xenforo conversation, they receive an email with the conversation reply. But when they click on the link, they get a "The requested conversation could not be found" error.
Looking up the conversation id in the database, it appears to be there. So, it seems like a permissions issue. I've rebuilt conversations to see if that would fix it. But it hasn't.
arn
arn
Well-known member
@Jake Bunce I got some insight from Mike over here: https://xenforo.com/community/threads/conversation-not-viewable-by-all-participants.99244/ Have you run into this before?
- User Goups Are not exported / imported!
- After importing, I have no admin / super admin rights more !!
Message: Your account does not have admin privileges.
Can not log in with my AdminUser in the ACP
- The import of 5 million contributions lasts more than 6 hours! Not really fast!
So far, the importer is not better than the XF own Importer!
- After importing, I have no admin / super admin rights more !!
Message: Your account does not have admin privileges.
Can not log in with my AdminUser in the ACP
- The import of 5 million contributions lasts more than 6 hours! Not really fast!
So far, the importer is not better than the XF own Importer!
Last edited:
Jake Bunce
Well-known member
The groups do require manual creation and id mapping using this importer.
Generally with this importer you need to manually specify the admin perms after the first attempt. Then those perms will persist in your XF install for all subsequent attempts. For the main admin account you should specify the userid as a super admin in the library/config.php file. And the is_admin flag might need to be enabled in the xf_user record.
6 hours for 5 million posts is slow. You server might have resource problems. Or maybe your InnoDB configuration needs to be optimized. I had an import recently that was suffering from slow performance. It turns out InnoDB was storing all tables in one file instead of separate files. You should revisit your MySQL config.
Jake Bunce
Well-known member
eva2000
Well-known member
@Jake Bunce or anyone else come across this weird issue. Looks like @Ludachris had same issue at https://xenforo.com/community/threa...-3-vbulletin-4-paid.52796/page-33#post-771356
Background
normal importer xenforo copy query returns empty
on big big board importer xenforo copy
in original vB 4.2.3 database notice the firstpostid and lastpostid are also 0. Also, like Ludachris reported experience, interesting part is accessing old forum OLDFORUM/showthread.php?t=1234 redirects to threadid = 9035 which is the same number as the pollid in thread 1234
The Xenforo big board importer database info for thread ids 1234 and 9035
Background
- Using big board import for vB 4.2 to Xenforo 1.4.6 (at the time latest), seems the Xenforo forum has a few threads which have no posts showing just top and bottom pagination numbers leading to no where.
- Querying the threadids show first_post_id and last_post_id = 0 so I grabbed all the threadids for those with first and last post id = 0 and there's ~851 of them.
- Now, this forum also used the regularly Xenforo forum importer for another entirely separate server test import. And if I query that resulting Xenforo database for those same threadids, they don't exist in the resulting xenforo imported database (regular inbuilt importer).
normal importer xenforo copy query returns empty
Code:
mysql -N -e "SELECT thread_id, first_post_id, last_post_id FROM xf_thread WHERE thread_id = 1234;" USERNAME_xftest
Code:
mysql -N -e "SELECT thread_id, first_post_id, last_post_id FROM xf_thread WHERE thread_id = 1234;" USERNAME_xf
+------+---+---+
| 1234 | 0 | 0 |
+------+---+---+in original vB 4.2.3 database notice the firstpostid and lastpostid are also 0. Also, like Ludachris reported experience, interesting part is accessing old forum OLDFORUM/showthread.php?t=1234 redirects to threadid = 9035 which is the same number as the pollid in thread 1234
Code:
mysql -e "SELECT * FROM thread WHERE threadid =1234\G" USERNAME_vb
*************************** 1. row ***************************
threadid: 1234
title: THREAD TITLE
firstpostid: 0
lastpost: 1154018998
forumid: 222
pollid: 9035
open: 10
replycount: 2
hiddencount: 0
postusername: USERNAME
postuserid: 1918
lastposter: LASTPOSTUSERNAME
dateline: 1154916852
views: 46
iconid: 11
notes:
visible: 1
sticky: 0
votenum: 0
votetotal: 0
attach: 0
similar: 21119, 7475, 6458, 9035, 138164
importthreadid: 0
importforumid: 0
deletedcount: 0
lastpostid: 0
prefixid:
taglist: NULL
wrdate:
vbseo_likes: 0
vbseo_linkbacks_no: 0
lastposterid: 0
keywords:
dbtech_vbanalytics_visits: 0
dbtech_vbanalytics_bounces: 0
dbtech_vbanalytics_pageviews: 0
dbtech_vbanalytics_landings: 0
dbtech_vbanalytics_referrals: 0
dbtech_vbanalytics_conversions: 0
dbtech_dbseo_keywords:
dbtech_dbseo_keywords_custom:
dbtech_dbseo_description:
dbtech_dbseo_description_custom:
postercount: 0
panjo_listing_id: NULLThe Xenforo big board importer database info for thread ids 1234 and 9035
Code:
mysql -e "SELECT * FROM xf_thread WHERE thread_id = 1234\G;" USERNAME_xf
*************************** 1. row ***************************
thread_id: 1234
node_id: 222
title: THREAD TITLE
reply_count: 2
view_count: 58
user_id: 1918
username: USERNAME
post_date: 1154916852
sticky: 0
discussion_state: visible
discussion_open: 10
discussion_type: poll
first_post_id: 0
first_post_likes: 0
last_post_date: 1154018998
last_post_id: 0
last_post_user_id: 0
last_post_username: LASTPOSTUSERNAME
prefix_id: 0
Code:
mysql -e "SELECT * FROM xf_thread WHERE thread_id = 9035\G;" USERNAME_xf
*************************** 1. row ***************************
thread_id: 9035
node_id: 144
title: THREAD TITLE
reply_count: 94
view_count: 21404
user_id: 1918
username: USERNAME
post_date: 1153799730
sticky: 0
discussion_state: visible
discussion_open: 1
discussion_type:
first_post_id: 56284
first_post_likes: 0
last_post_date: 1223163607
last_post_id: 583120
last_post_user_id: 21177
last_post_username: LASTPOSTUSERNAME2
prefix_id: 0
Last edited:
Jake Bunce
Well-known member
@arn
@eva2000
arn is correct. If memory serves, vB uses open = 10 to indicate a thread redirect, and then overloads the use of pollid to specify the destination for the redirect. This importer is not checking for this.
Edit the Export file and add the red code:
That should do it.
@eva2000
arn is correct. If memory serves, vB uses open = 10 to indicate a thread redirect, and then overloads the use of pollid to specify the destination for the redirect. This importer is not checking for this.
Edit the Export file and add the red code:
Rich (BB code):
exec('mysql -h' . $this->slaveDbHost . ' ' . $this->sourceDb . ' ' . $this->sourceDBuser . ' ' . $this->sourceDBpassword . ' -N -q -e "SET NAMES binary;SELECT thread.threadid AS thread_id, thread.forumid AS node_id, REPLACE(REPLACE(REPLACE(REPLACE(thread.title, \'&\', \'&\'), \'<\', \'<\'), \'>\', \'>\'), \'"\', \'\\"\') AS title, thread.replycount AS reply_count, thread.views AS view_count, thread.postuserid AS user_id, thread.postusername AS username, thread.dateline AS post_date, thread.sticky, IF(thread.visible = 0, \'moderated\', IF(thread.visible = 2, \'deleted\', \'visible\')) AS discussion_state, thread.open AS disucssion_open, IF(thread.pollid > 0, \'poll\', \'\') AS discussion_type, thread.firstpostid AS first_post_id, (SELECT COUNT(*) FROM ' . self::$tablePrefix . 'reputation as reputation WHERE reputation.postid = thread.firstpostid AND (reputation > 0 OR (reason = \'\' AND reputation = 0))) AS first_post_likes, thread.lastpost AS last_post_date, thread.lastpostid AS last_post_id, ' . (self::$isVB4 ? 'thread.lastposterid' : "'0'") . ' AS last_post_user_id, thread.lastposter AS last_post_username, \'0\' AS prefix_id FROM ' . self::$tablePrefix . 'thread as thread WHERE thread.open != 10 ORDER BY thread.threadid" | ' . self::$sedCommand . ' -r \'s/\&\#[0-9]*?\;//ig\' ' . self::$extraOutCommand . ' > ' . self::$dataDir . 'xf_thread.txt');That should do it.
eva2000
Well-known member
@Jake Bunce thanks. Anything can be done about existing imports that were finalised with the open = 10 not accounted for ?
Jake Bunce
Well-known member
This query should do it:
Rich (BB code):
DELETE xft.*
FROM `xfdb`.xf_thread xft
WHERE xft.thread_id IN (
SELECT vbt.threadid
FROM `vbdb`.thread vbt
WHERE vbt.open = 10
);Backup first of course.
eva2000
Well-known member
WSM_TechMedia
Member
Hi,
I have a general issue / question on the Big Board Importer script.
The Export.php not only uses the source database but also the target database ($this->db).
What I don't understand is this for example:
This part of the code tries to access a table customprofile in the xenforo database which of course never exists. How is this supposed to even work?
Furthermore: The script assumes that the target database and the source database both have the same table prefix which also seems to be a bug. The user table in vBulletin is "user" in XenForo it's "xf_user".
Has anyone experienced the same issues? Or am I just misunderstanding the script?
Thanks a lot for hints and comments.
-Fabian-
I have a general issue / question on the Big Board Importer script.
The Export.php not only uses the source database but also the target database ($this->db).
What I don't understand is this for example:
PHP:
$users = $this->db->fetchAll('
SELECT user.*, COALESCE(IF(customavatar.dateline, customavatar.dateline, customprofilepic.dateline), 0) AS avatar_date, COALESCE(IF(customavatar.dateline, customavatar.width, customprofilepic.width), 0) AS avatar_width, COALESCE(IF(customavatar.dateline, customavatar.height, customprofilepic.height), 0) AS avatar_height, LOWER(SUBSTRING(PASSWORD(MD5(CONCAT(user.salt, user.userid))), 2)) AS remember_key,
(SELECT COUNT(*) FROM ' . $this->sourceDb . '.' . self::$tablePrefix . 'reputation as reputation INNER JOIN ' . $this->sourceDb . '.' . self::$tablePrefix . 'post as post ON (post.postid = reputation.postid) INNER JOIN ' . $this->sourceDb . '.' . self::$tablePrefix . 'thread as thread ON (thread.threadid = post.threadid) WHERE post.visible = 1 AND thread.visible = 1 AND reputation.userid = user.userid AND (reputation > 0 OR (reason = \'\' AND reputation = 0))) AS like_count
FROM ' . $this->sourceDb . '.' . self::$tablePrefix . 'user as user
LEFT JOIN ' . $this->sourceDb . '.' . self::$tablePrefix . 'customavatar as customavatar ON (customavatar.userid = user.userid)
LEFT JOIN ' . $this->sourceDb . '.' . self::$tablePrefix . 'customprofilepic as customprofilepic ON (customprofilepic.userid = user.userid)
ORDER BY user.userid
LIMIT ' . ($limit * 10000) .', 10000
')
)This part of the code tries to access a table customprofile in the xenforo database which of course never exists. How is this supposed to even work?
Furthermore: The script assumes that the target database and the source database both have the same table prefix which also seems to be a bug. The user table in vBulletin is "user" in XenForo it's "xf_user".
Has anyone experienced the same issues? Or am I just misunderstanding the script?
Thanks a lot for hints and comments.
-Fabian-
Similar threads
- Question
- Question
