XF 1.5 Import from vBulletin 4.* got stuck on Import Forums step...

[Miso]

While going through the built-in import, once I got half way through to Import Forums step I got stuck at a 500 Error.

So while checking error_log I found this error got generated:

[22-Dec-2016 06:04:21 UTC] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 32 bytes) in /public_html/xenforo/library/Zend/Db/Statement/Mysqli.php on line 294

My PHP memory limit is 512M, and I've tried increasing that to 1024M and more (on a dedicated server, so I can adjust things as needed), and no luck, same error.

I try to resume the import and within literally 10 seconds it gets stuck on that step with that exact same error.

What might be causing it? And more importantly, how do I get around it?

Maybe because the vB_post table that it's probably querying is quite big (2.8GB, 4.6M records)?

If so, again, how do I get around this? Anything I can tweak in the import function itself to force it through somehow?

 

[Mike]

How many forums do you have? Do you have a lot of custom forum permissions?

Those are really the only things handled in this step and I really don't think it should try to use anywhere near that amount of memory.

There is a second part of the step which could potentially be related. Can you see the forums in the "display node tree" part of the control panel?

 

[Miso]

Yep, I see all the forums have been ported over in "Display Node Tree" (there are a few hundred)

And there are 175 usergroups (as per each product we offer) that I think you might be referring to as custom forum permissions, as each forum has those 175 usergroups attached to it, right?

Does that help?

And how do I get to the next step in this process now?

 

[Mike]

Right, it's actually the permission rebuild that's triggered here that's causing the problem then. It's assumed that it's likely fairly fast in this case, but given your setup with that many groups, it probably won't be.

You should be able to get past it by commenting out this line in library/XenForo/Importer/vBulletin.php:

XenForo_Model::create('XenForo_Model_Permission')->rebuildPermissionCache();

And reloading the importer.

To trigger that permission rebuild, go to <url>/install/ and rebuild the master data. (This version uses a paginated system which should hopefully be ok.)

 

[Miso]

That worked. Got past the 500 error, onto the next step, "Import Moderators", hit that button, and boom, 500 Error again.

From the error_log:

PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 5 bytes) in /public_html/xenforo/library/Zend/Db/Statement/Mysqli.php on line 276

 

[Mike]

Can you do the permission rebuild I mentioned? Does that complete without error?

If not, this is likely to be a problem beyond the import. We precalculate permissions for efficiency and given the amount of nodes and permission "combinations" you have, it looks like there's potentially a lot of work to do there.

If it does, then we can probably special case the behavior for the import.

 

[Miso]

Yeah, I tried rebuilding the permissions, but that resulted in that same Fatal Error as mentioned above.

Ok, would it make more sense to have you guys take a look at this instead, by me submitting a ticket about it?

 

[Mike]

Yeah, that might be best. If you can provide phpMyAdmin (or similar) access, that would be very helpful.

I will say that we might be able to change too much here. We approach the permission rebuilding steps by loading a lot of necessary data (permission combinations, nodes, permission values). We have had issues in the past with the amount of time a rebuild takes which we've made changes to handle, but it looks like this may be a data loading issue; changing that is unlikely to be a viable option as it stands.