XF 2.1 In Browser - Horrible Import Speed

[⭐ Alex ⭐]

The import has been running for 8 hours so far and for whatever reason has slowed down tenfold towards the end. Not only that but if the browser window goes unresponsive the import stops.

It uses almost none of the available system resources, I've seen it use 25% of cpu usage max and 50MB/s disk read at max, it should be using much more resources to get it all over with. I don't want to cancel and do command line at this point but I'm sure it would be quicker.

The in-browser should be just a viewer when it's doing the user import step and not command the process every 8 seconds as it does now. There is no UI for cancelling the import so there's no reason for the browser to not just be a viewer during this step.

Should I edit some file to increase the "POST" speed from 8 seconds to 1 second to make it run faster?

 

[Chris D]

There is some overhead with running imports (and any long running tasks) in the browser.

Unfortunately the very mechanics of how the browser import works cannot be changed. It has to work on a principle of refreshing the page every X otherwise your web server would likely kill the request after some period of time and there would be no indication of progress.

There's really no concept of the browser just being "a viewer". The server side processing requires physical activity on the server for its scripts to run, the only way of doing that is by refreshing the page periodically as we do currently, as you noticed, every 8 seconds.

You can change the default using src/config.php:

$config['jobMaxRunTime'] = 30;

This may reduce some of the overhead, but it will be at the expense of fewer progress updates.

It is possible to switch to the command line without losing any progress. You would just close the browser window so progress stops momentarily, and then run the import command from the command line:

php cmd.php xf:import

If you have spare system resources, as it appears you do, you can allow the command line to run multiple processes:

php cmd.php xf-import --processes=4

This will resume the import from where the browser left off.

This still uses the jobMaxRunTime value so you may still wish to increase that.

 

[⭐ Alex ⭐]

Hey @Chris D thanks for your help! I did notice the progress stopped when my computer when to sleep.

Also, I tried to close the tab and do the command line however:

php cmd.php xf:import
This import has been started through another method.

So that didn't work. I have a further question, if I revisit the import page I'm met with this:



Does "Restart" mean continue where it left off? Or start over from scratch? The way I continue right now is by hitting back in my browser, because otherwise I am met with that page above.

I added the config entry you suggested but it didn't improve the import speed.

 

[Chris D]

Oops. I'm sorry, I forgot there was a restriction on resuming the import via another method.

Yeah it will continue where it left off if you click "Restart import".

 

[crondoc]

Is it possible to run an importer command like this in the background so that you can close an SSH terminal?

/opt/rh/rh-php73/root/usr/bin/php cmd.php xf:import

perhaps something like this?

nohup /opt/rh/rh-php73/root/usr/bin/php cmd.php xf:import

(I'm on a Centos7 box running multiple versions of PHP hence the long path with the PHP command)
I ran a test import a few weeks ago and it took more than 24 hours, but I needed to move between different locations so I had to close my laptop which disconnected putty from the machine and stopped the import. It would be cool to be able to run it in the background but also keep an eye on progress somehow.

 

[Chris D]

You likely want to use the screen command:

https://linuxize.com/post/how-to-use-linux-screen/

 

[crondoc]

Thanks for the quick reply, Chris D.

Tried that on a test command sending an email after a long sleep and it works well. I like the way you can start up putty again and type screen -r to resume the session and see the output.