XF 2.3 VBulletin import errors

[david.steadson]

Running a VB3.7 import and having issues and it's failing.
First, lots of this -

00:00:57 [859] 7.82%libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

And then it fails at -

In AvatarService.php line 236:

Call to a member function getType() on null

Any advice?

 

[david.steadson]

As an addendum ... it appears that if the import fails, then the entire installation process needs to be started from scratch? Uggh. That's horrible.

 

[MentaL]

Running a VB3.7 import and having issues and it's failing.
First, lots of this -

And then it fails at -


Any advice?

Can't fix Post in thread 'libpng warnings during upgrade'

Nov 29, 2017

Yeah, these can be ignored. They're not something we have any control over being output.

• Mike

and

Fixed Post in thread 'Avatar/profile banner uploads don't properly check if the image can be processed, and throw an exception'

Oct 19, 2024

Thank you for reporting this issue, it has now been resolved. We are aiming to include any changes that have been made in a future XF release (2.3.4).

Change log:

Gracefully handle when an avatar cannot be processed

There may be a delay before changes are rolled out to the XenForo Community.

• XF Bug Bot

its a waiting game for you, sorry.

 

[Jeremy P]

Under line 235 of src/XF/Service/User/AvatarService.php, you can add:

        if (!$baseImage)
        {
            return false;
        }

As an addendum ... it appears that if the import fails, then the entire installation process needs to be started from scratch? Uggh. That's horrible.

You can resume the import after making the change above and it should pick up where it left off.

 

[david.steadson]

Under line 235 of src/XF/Service/User/AvatarService.php, you can add:

        if (!$baseImage)
        {
            return false;
        }

You can resume the import after making the change above and it should pick up where it left off.

Thanks, that seems to have fixed that problem. 10+ hours in to the import now, Step 16 of 26, 13.14% of Posts imported. Think I might go to bed ...

 

[MentaL]

Thanks, that seems to have fixed that problem. 10+ hours in to the import now, Step 16 of 26, 13.14% of Posts imported. Think I might go to bed ...

You can use --proc command to boost speed via cmd.php

 

[david.steadson]

You can use --proc command to boost speed via cmd.php

I doubt that will help? I'm transferring from the old VB server to a dedicated database server, so all going via LAN. The VB database is about 23GB

 

[MentaL]

I doubt that will help? I'm transferring from the old VB server to a dedicated database server, so all going via LAN. The VB database is about 23GB

P

XF 2.2 Post in thread 'Speeding up import'

Apr 24, 2023

The import time is dependent on server config, setup, and hardware.

How many cores are available?
You can use the processes argument depending on the number, e.g. php cmd.php xf:import --processes=4 --no-interaction .

• Paul B

• 

 

[Slavik]

I doubt that will help? I'm transferring from the old VB server to a dedicated database server, so all going via LAN. The VB database is about 23GB

Yikes, dont do imports via a network like that. Just transfer the DB first and drop it later.

 

[david.steadson]

P

XF 2.2 Post in thread 'Speeding up import'

Apr 24, 2023

The import time is dependent on server config, setup, and hardware.

How many cores are available?
You can use the processes argument depending on the number, e.g. php cmd.php xf:import --processes=4 --no-interaction .

• Paul B

• 

Yikes, dont do imports via a network like that. Just transfer the DB first and drop it later.

Ok, can you explain more. Here's the setup, on digital ocean.

Drop1 - webserver + database with VB 3.7
Drop2 - webserver with Xenforo
DB1 - database server

Transferring the VB database to DB1 and running the import script on Drop2 still requires LAN transfer via Drop2, right?

I guess I could reconfigure Drop1 to include MySQL, import VB to there, run Xenforo import there, then move everything to DB1? I've had problems finding the info on how to move the XF database after setup though.

 

[MentaL]

Ok, can you explain more. Here's the setup, on digital ocean.

Drop1 - webserver + database with VB 3.7
Drop2 - webserver with Xenforo
DB1 - database server

Transferring the VB database to DB1 and running the import script on Drop2 still requires LAN transfer via Drop2, right?

I guess I could reconfigure Drop1 to include MySQL, import VB to there, run Xenforo import there, then move everything to DB1? I've had problems finding the info on how to move the XF database after setup though.

Install xen on fresh install, make your admin, user 1 with same name and email. Next, use the VB import script to merge from VB to Xen. Sorry, rushing about. Keep it as local as possible for the import.

 

[david.steadson]

Install xen on fresh install, make your admin, user 1 with same name and email. Next, use the VB import script to merge from VB to Xen. Sorry, rushing about. Keep it as local as possible for the import.

That's what I'm doing, but again, note that the webserver does not have a DB, it's a separate server. The import is working, but it's so far been running for about 12 hours, with an estimate of 18:27 hrs left to complete, and the estimate goes up by a minute about .... every minute. Yay, an infinitely long import I suspect that will improve once it gets past the posts table, current at 43% (over 6 million imported so far)

 

[Jeremy P]

Transferring the VB database to DB1 and running the import script on Drop2 still requires LAN transfer via Drop2, right?

It requires the database to be backed up, transferred, and restored, yes. That's usually faster than trying to do the import itself over the network though. You can also use --processes to scale the import across multiple cores.

I've had problems finding the info on how to move the XF database after setup though.

You can dump the database from the source server and restore it on the target server, the same as any other database. You can edit src/config.php if you need to reconfigure the database connection.

 

[david.steadson]

It requires the database to be backed up, transferred, and restored, yes. That's usually faster than trying to do the import itself over the network though. You can also use --processes to scale the import across multiple cores.


You can dump the database from the source server and restore it on the target server, the same as any other database. You can edit src/config.php if you need to reconfigure the database connection.

Yeah, I've worked that out now (where the config is). Really would be nice to have all this info in the Importer docs though, I also have to just assume that's all I need to change?

How long should I expect a 24GB 13 million post VB3.7 import to take, if it's all on the same server?

And then how long to dump the resulting XF database, then reimport the XF database - over LAN - to a dedicated DB server?

 

[Jeremy P]

Yeah, I've worked that out now. Really would be nice to have all this info in the Importer docs though:

We do note that local imports are faster:

The import system | Manual | XenForo

End-user documentation for XenForo

xenforo.com

How long should I expect a 24GB 13 million post VB3.7 import to take, if it's all on the same server?

That depends a lot on the server hardware and configuration, but assuming appropriate specifications and configuration ~1k posts per second is not unreasonable.

 

[MentaL]

That's what I'm doing, but again, note that the webserver does not have a DB, it's a separate server. The import is working, but it's so far been running for about 12 hours, with an estimate of 18:27 hrs left to complete, and the estimate goes up by a minute about .... every minute. Yay, an infinitely long import I suspect that will improve once it gets past the posts table, current at 43% (over 6 million imported so far)

I have the same setup as you (post import), I Just change the server ip in config to the lan ip that has the sql server running on with the db migrated to it after the import is done.

 

[david.steadson]

We do note that local imports are faster:

The import system | Manual | XenForo

End-user documentation for XenForo

xenforo.com

That depends a lot on the server hardware and configuration, but assuming appropriate specifications and configuration ~1k posts per second is not unreasonable.

Noting local imports are faster is kind of stating the obvious. But 13 million posts at 1000/s is 3.5 hrs as apposed to 35+ hours is worth a bit more than a note. You might want to put in the importers docs that separate DB servers should not be used for importing large sites, and if that's the desired config to transfer the DB after the import, as has been suggested here.

So it appears the advice is to start again ... again? Been trying to do this since Thursday ....

 

[MentaL]

Noting local imports are faster is kind of stating the obvious. But 13 million posts at 1000/s is 3.5 hrs as apposed to 35+ hours is worth a bit more than a note. You might want to note on the importers that separate DB servers should not be used for importing large sites, and if that's the desired config to transfer the DB after the import.

So it appears the advice is to start again ... again? Been trying to do this since Thursday ....

Is your setup virtualised instances or each dedicated?

 

[david.steadson]

Is your setup virtualised instances or each dedicated?

Virtualised. Performance has been fine with the existing forum.

 

[MentaL]

Virtualised. Performance has been fine with the existing forum.

We have the same setup. Now, What would I do? Ride it out or start again localised?. Best of luck with your choice.