XF 2.1 2.1.0 update taking too long (3+ hours now)

[brunoa]

Wasn't expecting such an epic update from 2.0.9 to 2.1.0. FTP upload took 4:30 hours (using Transmit for Mac) and it seems like the merge files option didn't work this time. Had to recreate the config.php file and I'm not sure yet if other configs were deleted.

Now I'm running the update via command line and it's stuck on this step:
Running upgrade to 2.1.0 Alpha, step 18 (15892888)...

Been almost 3 hours since I started the command line update phase, bringing the total to 7:30 for this update. Is this normal? I'm considering restoring a backup and trying it another day. Any advice?

 

[Chris D]

Is this normal?

No, on both counts. Perhaps most significantly the FTP upload. You may have some serious issues somewhere that you should look at addressing. FTP can be slow, but to take that long you're looking at over 3 seconds per file at that speed. There's a lot of files but on average they are less than 10KB.

I'd be interested to hear how long it takes to upload just the zip file, or some other files. There's potentially some overhead with FTP depending on how the connection is made which could slow things down.

Of course if uploading the zip file is quicker, then you could consider doing that, extracting it on the server and then copying the files into place. That will usually be much quicker anyway.

Onto the actual upgrade itself, the step that's currently running is expected to take a long time. This is converting the xf_post table to support reactions.

That said, it shouldn't be taking this long. The number in brackets (15892888) is the last post ID it worked on. So at this point it has already converted over 15 million posts. How many posts do you have? I'd be interested to understand how long it takes for the number in brackets to change, and what it changes to. That will give us an idea of how many posts it is processing per second. If it has taken up to 3 hours to get to this point though that seems pretty slow to me, so that could indicate there are performance issues with the queries we're running or the database itself. We've not had any other reports of this taking so long, and I'm sure larger boards have attempted by now so the time being taken here does seem unusual.

 

[brunoa]

Thanks for the prompt reply, Chris. Zip upload is fast. Currently uploading at 250kbps, which seems to be my internet upload limit. Transmit seems to be extremely slow when it's zillions of small files, like in the XF upgrades. What I tried to to is I upload the upload folder, than moved the content to the parent directory (forum root), but it didn't ask for the merge option. I think it deleted everything before uploading, so I stopped at the install folder and uploaded src directly from my computer, the normal way.

Iirc, we have close to 20 million posts. It's a large forum.

That number was increasing before it got stuck at that value. While I was writing this, I got a new error message:
Running upgrade to 2.1.0 Alpha, step 18 (15892888)...packet_write_wait: Connection to 172.31.3.168 port 22: Broken pipe

Then I ran the command again and it finished, but the forum and admin page is not opening now. http://forum.outerspace.com.br/

If you want to check it yourself I can give you access to the server & admin. Thanks!

 

[Chris D]

"Broken pipe" is generally just when the SSH connection failed for some reason. The process may not have taken much longer to complete, it depends how long it was stuck on 15892888, I guess. If it was stuck on that post for anything longer than 30 seconds then I suspect it was that point that your SSH connection failed, but the client didn't report "Broken pipe" until later.

I can't really provide much insight as to why the site/admin isn't loading right now other than it appears to be continuously attempting to do a 301 or 302 redirect. That's not going to be something in the software itself. Certainly if our own redirect stuff goes awry, the usual advice is just to go to "admin.php" because none of our redirect stuff operates there (by design) but something on your server seems to be trying to redirect even the admin.php URL.

Not sure if the issue with copying/merging files has done something to remove some custom htaccess/rewrite rules you had.

On that point, though, you might want to actually check that things such as your avatars and attachments are still there. They would be stored in data/avatars and internal_data/attachments and if those files no longer exists then the issue with copying files may have done some damage.

FWIW I tend to use rsync on my server when I'm copying files:

rsync -r upload/ ../public/

That should only copy the required files, overwriting where necessary, without destroying entire directories (which may be what has happened here).

 

[brunoa]

Avatars and attachment are still there. I'm restoring our backup for now and will attempt this update again another day. Seems to be much bigger than I thought and it's a shame that I'll have to waste this huge downtime. Users must be mad!

I'll try to figure out a way to do FTP faster first, and to make sure I'm merging files. Thanks again for helping, Chris!

 

[Andrej]

Transmit seems to be extremely slow when it's zillions of small files, like in the XF upgrades.

I haven't experienced such slow upload with Transmit, nowhere near 3 hours. It has been a while since I did my last file-by-file upload with Transmit (since uploading the ZIP file and extracting it on the server is faster), so I am not even sure whenever it took a few minutes or fifteen. The problem is most certainly not with the FTP client itself.

 

[brunoa]

I haven't experienced such slow upload with Transmit, nowhere near 3 hours. It has been a while since I did my last file-by-file upload with Transmit (since uploading the ZIP file and extracting it on the server is faster), so I am not even sure whenever it took a few minutes or fifteen. The problem is most certainly not with the FTP client itself.

4:30 hours, not 3 hours, lol. Really bizarre and I have no idea why it's so slow, but will keep trying to figure it out.

 

[brunoa]

Hi Chris, I'm trying the upgrade again. Sorted out the ftp problem but the upgrade process still took a long time and now that it's almost finished I'm stuck with the following error:

Current version: 2010070

Upgrade target: 2010070 (2.1.0)

You are already running the latest version. Rebuild the master data? [y/n] y


In Local.php line 199:

[E_WARNING] file_put_contents(/r7/www/forum/public_html/internal_data/code_cache/templates/l2/s25/public/_media_site_embed_applemusic.php): failed to open stream: Permission denied

--

The forum is not working and I can't access the admin yet. Can only access the /install path. Need to act quick as it's been already a long downtime (3 hours). Any advice?

 

[brunoa]

@Chris D

Just making sure you see it.

 

[Chris D]

That error would suggest that the data and internal_data directories are not set to be writable as required. Particularly, any subdirectories within those directories needs to be writable as well. It appears as though the internal_data/code_cache directory and its children is not writable in this specific case.

There's a section about it in the manual:

Install or upgrade? | Manual | XenForo

End-user documentation for XenForo

xenforo.com

 

[brunoa]

Thanks again for your help, Chris. Problem was with my ssh user permissions indeed. Finished the upgrade after about 4 hours. It's an epic one. It got stuck at step 18 again this time, but I could resume and finish. Hopefully the next ones will be easier!