On my board the members amuse themselves trying to make the longest useless thread ever, enter some wonderful gems:
"The Longest Thread II": 333,175 replies
"The Longest Thread": 130,808 replies
PHP memory limit side, the latest vb4 importer for XF2 blows up spectacularly on this, as for each thread it loads *all the posts* in a single query. By the time this select has completed, one post is imported (or more depending on dateline) and then the import scripts exits and starts with the next post because it exceeded 8 sec. It was hilarious to see the import running at 1 post per 14 seconds, I knew there was no way this could be right. Importing "The longest thread II" would have taken 53 days LOL...
Below is my patch that works around this. I strongly suggest you implement something to this effect. With this patch 5,778,400 posts imported in 7 hours, 8 minutes which is 229 posts per second, a very acceptable rate considering the mysql to 2 different database servers.
"The Longest Thread II": 333,175 replies
"The Longest Thread": 130,808 replies
PHP memory limit side, the latest vb4 importer for XF2 blows up spectacularly on this, as for each thread it loads *all the posts* in a single query. By the time this select has completed, one post is imported (or more depending on dateline) and then the import scripts exits and starts with the next post because it exceeded 8 sec. It was hilarious to see the import running at 1 post per 14 seconds, I knew there was no way this could be right. Importing "The longest thread II" would have taken 53 days LOL...
Below is my patch that works around this. I strongly suggest you implement something to this effect. With this patch 5,778,400 posts imported in 7 hours, 8 minutes which is 229 posts per second, a very acceptable rate considering the mysql to 2 different database servers.
Diff:
diff --git a/htdocs/xenforo/src/addons/XFI/Import/Importer/vBulletin.php b/htdocs/xenforo/src/addons/XFI/Import/Importer/vBulletin.php
index 54c1da3..3ccc063 100644
--- a/htdocs/xenforo/src/addons/XFI/Import/Importer/vBulletin.php
+++ b/htdocs/xenforo/src/addons/XFI/Import/Importer/vBulletin.php
@@ -18,6 +18,8 @@ class vBulletin extends AbstractForumImporter
*/
protected $userFields;
+ protected const MAX_POSTS = 500;
+
public static function getListInfo()
{
return [
@@ -3073,6 +3075,21 @@ class vBulletin extends AbstractForumImporter
$this->lookup('user', $this->pluck($posts, ['userid', 'edituserid']));
+ $continueSameThread = false;
+ if (count($posts)==vBulletin::MAX_POSTS)
+ {
+ $continueSameThread = true;
+ $lastDateline = $posts[vBulletin::MAX_POSTS-1]["dateline"];
+ while (count($posts) && ($posts[count($posts)-1]["dateline"]==$lastDateline))
+ {
+ // since we limited the retrieved posts, we don't know
+ // if there are further posts in the database with
+ // the same dateline, so drop posts until we find one
+ // with an earlier dateline.
+ array_pop($posts);
+ }
+ }
+
foreach ($posts AS $i => $post)
{
$state->extra['postDateStart'] = $post['dateline'];
@@ -3126,6 +3143,10 @@ class vBulletin extends AbstractForumImporter
}
}
+ if ($continueSameThread) {
+ break; // calls resumeIfNeeded and fetches the rest of the thread
+ }
+
$state = $this->setStateNextThread($state, $oldThreadId);
if ($timer->limitExceeded())
@@ -3151,6 +3172,8 @@ class vBulletin extends AbstractForumImporter
protected function getPosts($threadId, $startDate)
{
+ $MAX_POSTS = vBulletin::MAX_POSTS;
+
return $this->sourceDb->fetchAll($this->prepareImportSql($this->prefix, "
SELECT post.*,
IF(user.username IS NULL, post.username, user.username) AS username,
@@ -3167,7 +3190,7 @@ class vBulletin extends AbstractForumImporter
editlog ON (editlog.postid = post.postid)
WHERE post.threadid = ?
AND post.dateline > ?
- ORDER BY post.dateline
+ ORDER BY post.dateline limit 0,$MAX_POSTS
"), [$threadId, $startDate]);
}
I'll be importing from vB3, also have 100,000k post threads, and if I see the same behaviour I'll now know what might be causing it. Cheers.
)
