XF 2.2 Importing from a VB 5.7.5 Forum - Import seems to stall on Step 14: Threads

[DaveRage3D]

Hi All,

I am moving a VBulletin 5.7.5 forum with 455,000 topics, 7.4 Million Posts, 58,000 users to XenForo 2.2. Both my existing forum and the new XenForo forum are hosted on the same server. The server has a Xeon E5-2630 v4 CPU with 8 cores assigned, and 32GB of RAM. I am running PHP version 8.2, and MySQL 8.0.24.

I am using the web based import tool version 1.5.3. The import seems to run well and takes about 15 minutes to get to Step 14: Threads. At this point the process seems to stall out, server load seems to drop off to around 5% CPU usage, 5GB RAM and Disk IO is barely anything at all (maybe 5MB/s). The web page seems to stay stuck at "Step 14 of 265: Threads - [2] 0.00%". I just refreshed the page and it updated to "Step 14 of 26: Threads - [3] - 0.00%". This was after waiting 5 minutes or so after it got to Step 14.

Some history on this forum: It was originally a vbulletin 3.6 forum for many many years (Since 2004 or so). 2 years ago a friend of mine ran a script that pushed the forum to VB 4 and then VB 5.7.5. Since then the forum was mostly stable, but posting replies, new threads was very slow.

Using the importer I am selecting Vbulletin 5 as my source forum, should I try Vbulletin 3? I have doubts that would work, since the forum is actually running Vbulletin 5 currently.

Here is my Mysql config:



In addition, I checked my "innodb" settings based on some search results on this forum, not sure if these are accurate:

innodb_data_home_dir = /www/server/data
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = /www/server/data
innodb_buffer_pool_size = 24000M
innodb_buffer_pool_instances = 8
innodb_log_file_size = 512M
innodb_log_buffer_size = 128M
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
innodb_lock_wait_timeout = 50
innodb_max_dirty_pages_pct = 90
innodb_flush_neighbors = 0
innodb_read_io_threads = 64
innodb_write_io_threads = 64
innodb_io_capacity = 3000

And my php config:



Server load:



Anyway, I've been at this for 7 hours tonight and its 4am. Going to let the import process continue and see where things are. If anyone has any ideas, I would appreciate it!

 

[Paul B]

I have just replied to your ticket.

 

[DaveRage3D]

Thanks. I replied to you in the ticket. I took your advice and made a thread here for server config help: https://xenforo.com/community/threa...enforo-can-someone-check-mysql-config.219916/

I've solved the issue of the import process seeming to stall at step 14, it turns out my browser was timing out. I switched to Firefox and now its not timing out. That being said, Step 14 of importing threads is taking 35 seconds per thread. I have 455,000 threads. Need to find a way to speed this up.

 

[DaveRage3D]

Alright!!

I got command line access on the webserver. Things are moving now!

 

[DaveRage3D]

I get to step 18 of 26: polls and after that I get this error:


In ParallelProcessManager.php line 172:

Child process exited with code 1. See control panel error error logs for de
tails.

XF\Db\Exception: MySQL statement prepare error [1270]: Illegal mix of collations (utf8mb4_general_ci,COERCIBLE), (utf8mb4_general_ci,COERCIBLE), (latin1_bin,IMPLICIT) for operation 'concat' src/XF/Db/AbstractStatement.php:230

            SELECT
                #CONCAT(u.id, '-node', n.parentid) AS attachmentid,
                IF(u.filename IS NULL, CONCAT(u.id, '.', f.extension), u.filename) AS filename,
                n.userid AS userid,
                n.nodeid AS attachmentid,
                n.oldid AS legacyid,
                f.filedataid,
                n.parentid AS postid,
                n.nodeid,
                f.dateline,
                f.userid AS filedata_userid,
                0 AS counter
            FROM (
                SELECT a.filedataid, a.nodeid,
                    IF (a.filename = '', NULL, a.filename) AS filename,
                    CONCAT('attach', a.nodeid) AS id
                    FROM attach AS a
                    WHERE a.filedataid IN(617, 618, 619, 620, 621, 622, 623, 624, 625, 626, 627, 628, 629, 630, 632, 633, 634, 635, 636, 639, 640, 641, 642, 643, 644, 645, 646, 647, 648, 649, 650, 651, 652, 653, 654, 655, 656, 657, 658, 659, 660, 661, 662, 663, 664, 665, 666, 669, 670, 671, 672, 674, 675, 676, 678, 679, 680, 681, 683, 686, 687, 688, 689, 690, 692, 693, 694, 695, 697, 698, 699, 700, 701, 702, 703, 704, 705, 706, 708, 709, 710, 711, 712, 713, 714, 715, 716, 717, 718, 719, 720, 721, 722, 723, 724, 725, 726, 727, 728, 729, 730, 731, 732, 734, 735, 736, 737, 738, 739, 740, 741, 742, 747, 748, 749, 750, 751, 752, 753, 754, 755, 757, 758, 759, 760, 761, 762, 763, 764, 765, 766, 767, 768, 769, 770, 771, 772, 773, 775, 778, 780, 781, 782, 783, 784, 785, 786, 787, 788, 789, 790, 791, 792, 795, 796, 797, 799, 800, 801, 803, 804, 805, 808, 809, 810, 817, 819, 821, 822, 823, 824, 825, 826, 827, 828, 829, 830, 831, 832, 833, 834, 836, 837, 838, 839, 840, 844, 845, 846, 852, 854, 855, 856, 857, 858, 859, 860, 861, 862, 863, 864, 865, 866, 867, 868, 869, 870, 871, 872, 873, 874, 875, 876, 877, 878, 879, 880, 881, 882, 883, 884, 885, 886, 887, 888, 889, 890, 891, 892, 893, 894, 899, 900, 901, 902, 903, 904, 905, 906, 907, 908, 909, 912, 913, 914, 915, 916, 917, 918, 919, 920, 921, 922, 923, 925, 926, 927, 928, 929, 930, 931, 933, 934, 935, 936, 937, 938, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 952, 953, 954, 955, 956, 957, 958, 959, 960, 961, 962, 963, 964, 965, 966, 968, 969, 970, 971, 972, 973, 974, 975, 976, 977, 978, 979, 980, 981, 983, 984, 985, 986, 987, 988, 989, 990, 991, 992, 993, 994, 995, 996, 997, 998, 999, 1000, 1001, 1002, 1003, 1004, 1005, 1006, 1007, 1008, 1009, 1010, 1011, 1012, 1013, 1014, 1015, 1016, 1017, 1018, 1019, 1020, 1021, 1022, 1023, 1024, 1025, 1026, 1027, 1028, 1029, 1030, 1031, 1032, 1033, 1034, 1035, 1036, 1037, 1038, 1039, 1040, 1041, 1043, 1044, 1045, 1046, 1047, 1048, 1051, 1052, 1053, 1054, 1055, 1056, 1057, 1058, 1059, 1060, 1061, 1062, 1063, 1064, 1065, 1066, 1067, 1068, 1069, 1070, 1071, 1072, 1073, 1074, 1075, 1076, 1077, 1078, 1079, 1080, 1081, 1082, 1083, 1084, 1085, 1086, 1087, 1088, 1089, 1090, 1091, 1092, 1093, 1094, 1095, 1096, 1097, 1098, 1099, 1100, 1101, 1102, 1103, 1104, 1105, 1106, 1107, 1108, 1109, 1110, 1111, 1114, 1115, 1116, 1118, 1119, 1120, 1121, 1122, 1124, 1125, 1126, 1127, 1128, 1129, 1130, 1131, 1132, 1133, 1134, 1135, 1136, 1138, 1139, 1140, 1141, 1143, 1144, 1145, 1147, 1148, 1149, 1150, 1151, 1152, 1153, 1154, 1155, 1156, 1157, 1158, 1159, 1160, 1161, 1162, 1163, 1164, 1165, 1166, 1167, 1168, 1169, 1170, 1171, 1172, 1173, 1174, 1175, 1176, 1177, 1178, 1179, 1180, 1181, 1182, 1183, 1184, 1185, 1186, 1187, 1188, 1189, 1190, 1191, 1192, 1193, 1194, 1195, 1196, 1197, 1198, 1199, 1200, 1201, 1202, 1203, 1204, 1206, 1207, 1208, 1211, 1212, 1213, 1214, 1215, 1216, 1217, 1218, 1219, 1220, 1221, 1222, 1223, 1224, 1225, 1226, 1227, 1228, 1229, 1230)
                UNION
                SELECT p.filedataid, p.nodeid, NULL AS filename,
                    CONCAT('photo', p.filedataid) AS id
                    FROM photo AS p
                    WHERE p.filedataid IN(617, 618, 619, 620, 621, 622, 623, 624, 625, 626, 627, 628, 629, 630, 632, 633, 634, 635, 636, 639, 640, 641, 642, 643, 644, 645, 646, 647, 648, 649, 650, 651, 652, 653, 654, 655, 656, 657, 658, 659, 660, 661, 662, 663, 664, 665, 666, 669, 670, 671, 672, 674, 675, 676, 678, 679, 680, 681, 683, 686, 687, 688, 689, 690, 692, 693, 694, 695, 697, 698, 699, 700, 701, 702, 703, 704, 705, 706, 708, 709, 710, 711, 712, 713, 714, 715, 716, 717, 718, 719, 720, 721, 722, 723, 724, 725, 726, 727, 728, 729, 730, 731, 732, 734, 735, 736, 737, 738, 739, 740, 741, 742, 747, 748, 749, 750, 751, 752, 753, 754, 755, 757, 758, 759, 760, 761, 762, 763, 764, 765, 766, 767, 768, 769, 770, 771, 772, 773, 775, 778, 780, 781, 782, 783, 784, 785, 786, 787, 788, 789, 790, 791, 792, 795, 796, 797, 799, 800, 801, 803, 804, 805, 808, 809, 810, 817, 819, 821, 822, 823, 824, 825, 826, 827, 828, 829, 830, 831, 832, 833, 834, 836, 837, 838, 839, 840, 844, 845, 846, 852, 854, 855, 856, 857, 858, 859, 860, 861, 862, 863, 864, 865, 866, 867, 868, 869, 870, 871, 872, 873, 874, 875, 876, 877, 878, 879, 880, 881, 882, 883, 884, 885, 886, 887, 888, 889, 890, 891, 892, 893, 894, 899, 900, 901, 902, 903, 904, 905, 906, 907, 908, 909, 912, 913, 914, 915, 916, 917, 918, 919, 920, 921, 922, 923, 925, 926, 927, 928, 929, 930, 931, 933, 934, 935, 936, 937, 938, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 952, 953, 954, 955, 956, 957, 958, 959, 960, 961, 962, 963, 964, 965, 966, 968, 969, 970, 971, 972, 973, 974, 975, 976, 977, 978, 979, 980, 981, 983, 984, 985, 986, 987, 988, 989, 990, 991, 992, 993, 994, 995, 996, 997, 998, 999, 1000, 1001, 1002, 1003, 1004, 1005, 1006, 1007, 1008, 1009, 1010, 1011, 1012, 1013, 1014, 1015, 1016, 1017, 1018, 1019, 1020, 1021, 1022, 1023, 1024, 1025, 1026, 1027, 1028, 1029, 1030, 1031, 1032, 1033, 1034, 1035, 1036, 1037, 1038, 1039, 1040, 1041, 1043, 1044, 1045, 1046, 1047, 1048, 1051, 1052, 1053, 1054, 1055, 1056, 1057, 1058, 1059, 1060, 1061, 1062, 1063, 1064, 1065, 1066, 1067, 1068, 1069, 1070, 1071, 1072, 1073, 1074, 1075, 1076, 1077, 1078, 1079, 1080, 1081, 1082, 1083, 1084, 1085, 1086, 1087, 1088, 1089, 1090, 1091, 1092, 1093, 1094, 1095, 1096, 1097, 1098, 1099, 1100, 1101, 1102, 1103, 1104, 1105, 1106, 1107, 1108, 1109, 1110, 1111, 1114, 1115, 1116, 1118, 1119, 1120, 1121, 1122, 1124, 1125, 1126, 1127, 1128, 1129, 1130, 1131, 1132, 1133, 1134, 1135, 1136, 1138, 1139, 1140, 1141, 1143, 1144, 1145, 1147, 1148, 1149, 1150, 1151, 1152, 1153, 1154, 1155, 1156, 1157, 1158, 1159, 1160, 1161, 1162, 1163, 1164, 1165, 1166, 1167, 1168, 1169, 1170, 1171, 1172, 1173, 1174, 1175, 1176, 1177, 1178, 1179, 1180, 1181, 1182, 1183, 1184, 1185, 1186, 1187, 1188, 1189, 1190, 1191, 1192, 1193, 1194, 1195, 1196, 1197, 1198, 1199, 1200, 1201, 1202, 1203, 1204, 1206, 1207, 1208, 1211, 1212, 1213, 1214, 1215, 1216, 1217, 1218, 1219, 1220, 1221, 1222, 1223, 1224, 1225, 1226, 1227, 1228, 1229, 1230)
            ) AS u
            INNER JOIN filedata AS
                f ON (f.filedataid = u.filedataid)
            INNER JOIN node AS
                n ON (n.nodeid = u.nodeid)
            WHERE f.filedataid IN(617, 618, 619, 620, 621, 622, 623, 624, 625, 626, 627, 628, 629, 630, 632, 633, 634, 635, 636, 639, 640, 641, 642, 643, 644, 645, 646, 647, 648, 649, 650, 651, 652, 653, 654, 655, 656, 657, 658, 659, 660, 661, 662, 663, 664, 665, 666, 669, 670, 671, 672, 674, 675, 676, 678, 679, 680, 681, 683, 686, 687, 688, 689, 690, 692, 693, 694, 695, 697, 698, 699, 700, 701, 702, 703, 704, 705, 706, 708, 709, 710, 711, 712, 713, 714, 715, 716, 717, 718, 719, 720, 721, 722, 723, 724, 725, 726, 727, 728, 729, 730, 731, 732, 734, 735, 736, 737, 738, 739, 740, 741, 742, 747, 748, 749, 750, 751, 752, 753, 754, 755, 757, 758, 759, 760, 761, 762, 763, 764, 765, 766, 767, 768, 769, 770, 771, 772, 773, 775, 778, 780, 781, 782, 783, 784, 785, 786, 787, 788, 789, 790, 791, 792, 795, 796, 797, 799, 800, 801, 803, 804, 805, 808, 809, 810, 817, 819, 821, 822, 823, 824, 825, 826, 827, 828, 829, 830, 831, 832, 833, 834, 836, 837, 838, 839, 840, 844, 845, 846, 852, 854, 855, 856, 857, 858, 859, 860, 861, 862, 863, 864, 865, 866, 867, 868, 869, 870, 871, 872, 873, 874, 875, 876, 877, 878, 879, 880, 881, 882, 883, 884, 885, 886, 887, 888, 889, 890, 891, 892, 893, 894, 899, 900, 901, 902, 903, 904, 905, 906, 907, 908, 909, 912, 913, 914, 915, 916, 917, 918, 919, 920, 921, 922, 923, 925, 926, 927, 928, 929, 930, 931, 933, 934, 935, 936, 937, 938, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 952, 953, 954, 955, 956, 957, 958, 959, 960, 961, 962, 963, 964, 965, 966, 968, 969, 970, 971, 972, 973, 974, 975, 976, 977, 978, 979, 980, 981, 983, 984, 985, 986, 987, 988, 989, 990, 991, 992, 993, 994, 995, 996, 997, 998, 999, 1000, 1001, 1002, 1003, 1004, 1005, 1006, 1007, 1008, 1009, 1010, 1011, 1012, 1013, 1014, 1015, 1016, 1017, 1018, 1019, 1020, 1021, 1022, 1023, 1024, 1025, 1026, 1027, 1028, 1029, 1030, 1031, 1032, 1033, 1034, 1035, 1036, 1037, 1038, 1039, 1040, 1041, 1043, 1044, 1045, 1046, 1047, 1048, 1051, 1052, 1053, 1054, 1055, 1056, 1057, 1058, 1059, 1060, 1061, 1062, 1063, 1064, 1065, 1066, 1067, 1068, 1069, 1070, 1071, 1072, 1073, 1074, 1075, 1076, 1077, 1078, 1079, 1080, 1081, 1082, 1083, 1084, 1085, 1086, 1087, 1088, 1089, 1090, 1091, 1092, 1093, 1094, 1095, 1096, 1097, 1098, 1099, 1100, 1101, 1102, 1103, 1104, 1105, 1106, 1107, 1108, 1109, 1110, 1111, 1114, 1115, 1116, 1118, 1119, 1120, 1121, 1122, 1124, 1125, 1126, 1127, 1128, 1129, 1130, 1131, 1132, 1133, 1134, 1135, 1136, 1138, 1139, 1140, 1141, 1143, 1144, 1145, 1147, 1148, 1149, 1150, 1151, 1152, 1153, 1154, 1155, 1156, 1157, 1158, 1159, 1160, 1161, 1162, 1163, 1164, 1165, 1166, 1167, 1168, 1169, 1170, 1171, 1172, 1173, 1174, 1175, 1176, 1177, 1178, 1179, 1180, 1181, 1182, 1183, 1184, 1185, 1186, 1187, 1188, 1189, 1190, 1191, 1192, 1193, 1194, 1195, 1196, 1197, 1198, 1199, 1200, 1201, 1202, 1203, 1204, 1206, 1207, 1208, 1211, 1212, 1213, 1214, 1215, 1216, 1217, 1218, 1219, 1220, 1221, 1222, 1223, 1224, 1225, 1226, 1227, 1228, 1229, 1230)
            ORDER BY f.filedataid
        
------------

#0 src/XF/Db/Mysqli/Statement.php(198): XF\Db\AbstractStatement->getException('MySQL statement...', 1270, 'HY000')
#1 src/XF/Db/Mysqli/Statement.php(40): XF\Db\Mysqli\Statement->getException('MySQL statement...', 1270, 'HY000')
#2 src/XF/Db/Mysqli/Statement.php(56): XF\Db\Mysqli\Statement->prepare()
#3 src/XF/Db/AbstractAdapter.php(96): XF\Db\Mysqli\Statement->execute()
#4 src/XF/Db/AbstractAdapter.php(157): XF\Db\AbstractAdapter->query('
            SELECT
                ...', Array)
#5 src/addons/XFI/Import/Importer/vBulletin5.php(1046): XF\Db\AbstractAdapter->fetchAll('
            SELECT
                ...')
#6 src/addons/XFI/Import/Importer/vBulletin4.php(81): XFI\Import\Importer\vBulletin5->getAttachmentsForFileDataIds('617, 618, 619, ...', 'postid')
#7 src/addons/XFI/Import/Importer/vBulletin.php(3883): XFI\Import\Importer\vBulletin4->getAttachmentsGroupedByFile(Array)
#8 src/XF/Import/Runner.php(244): XFI\Import\Importer\vBulletin->stepAttachments(Object(XF\Import\StepState), Array, 8)
#9 src/XF/Import/ParallelRunner.php(213): XF\Import\Runner->runStep('attachments', Object(XF\Import\StepState), 8)
#10 src/XF/Cli/Command/ImportChildProcess.php(77): XF\Import\ParallelRunner->runChildProcess('attachments', 615, 1230, Object(Closure))
#11 src/vendor/symfony/console/Command/Command.php(255): XF\Cli\Command\ImportChildProcess->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#12 src/vendor/symfony/console/Application.php(992): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 src/vendor/symfony/console/Application.php(255): Symfony\Component\Console\Application->doRunCommand(Object(XF\Cli\Command\ImportChildProcess), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 src/vendor/symfony/console/Application.php(148): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 src/XF/Cli/Runner.php(111): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 cmd.php(15): XF\Cli\Runner->run()
#17 {main}

 

[Jeremy P]

Make sure the import was configured with the correct character set for the vBulletin database, and that all tables/columns of the vBulletin database have the corresponding charset/collation (particularly the filedata table).

 

[DaveRage3D]

Hey everyone, I was able to solve the attachment issues. New forum is live! Such a vast improvement over VBulletin

Rage3D Discussion Area

Rage3D Discussion Area

www.rage3d.com