Elasticsearch 7.11

[MattW]

Just a heads up on this one. They have changed how the heap memory is defined and managed:

################################################################
## IMPORTANT: JVM heap size
################################################################
##
## The heap size is automatically configured by Elasticsearch
## based on the available memory in your system and the roles
## each node is configured to fulfill. If specifying heap is
## required, it should be done through a file in jvm.options.d,
## and the min and max should be set to the same value. For
## example, to set the heap to 4 GB, create a new file in the
## jvm.options.d directory containing these lines:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################

It's no longer controlled by the jvm.options file, and if you server automatically restarts ES after the update (like one of my cPanel servers has just done), you'll probably run out of Memory!

On my server running my site, I previously had 1GB allocated to Java, with the new setup, it's restarted, and ballooned to RAM usage to attempt to use 50% (32GB!)

 

[Kier]

Thanks for the heads up, @MattW

 

[Xon]

It wouldn't be elasticsearch without some silent backwards compatible breaking change in a point release

 

[gerryvz]

Is there any action item or config change that is required, as a result of this change in ES 7.11 ?

 

[gerryvz]

Since upgrading to Elasticsearch 7.11, I've gone from having to restart ES once every week or two, to now 10 times a day.

I am getting constant curl 7 errors -- unable to connect to the server to do the indexing. Things run fine for an hour or two after a restart, then fail.

is there any way to downgrade ES from 7.11 to 7.10? Anyone done this or can provide some instructions how to do this?

I think this is a pretty significant problem, and if you are considering going to ES 7.11 I would hold off.

 

[beerForo]

So, my ES seems to update automatically! I don't really know how but it does. So I checked and it is 7.11. And also, yes, today it is failing, for the first time in a LONG time.

 

[gerryvz]

Mine updates when I do software updates via apt update && apt upgrade in ubuntu. I have the Elastic PPA installed, so the system polls it for updates, and then installs if new software is found.

 

[bzcomputers]

It appears the reason for placing your custom Elasticsearch settings in jvm.options.d is that when Elasticsearch upgrades the settings in that location are not overwritten. Unlike the settings which are contained within jvm.options which may be overwritten on version updates.

It appears the default settings (located in jvm.options) for -Xms and -Xmx changed from 7.10 to 7.11. Adding your heap size settings to the new jvm.options.d location, along with any other custom Elasticsearch settings you had, and restarting Elasticsearch should fix your issues.

 

[gerryvz]

It appears the reason for placing your custom Elasticsearch settings in jvm.options.d is that when Elasticsearch upgrades the settings in that location are not overwritten. Unlike the settings which are contained within jvm.options which may be overwritten on version updates.

It appears the default settings (located in jvm.options) for -Xms and -Xmx changed from 7.10 to 7.11. Adding your heap size settings to the new jvm.options.d location, along with any other custom Elasticsearch settings you had, and restarting Elasticsearch should fix your issues.

Thanks for this info and also to @MattW for the info in the original post. I’ll add the settings and see how things go. This is really frustrating.


UPDATE
What I did was to go to /etc/elasticsearch and found the jvm.options file, as well as the jvm.options.d directory.

I did a cp jvm.options jvm.options.d to copy the jvm.options file into the jvm.options.d directory.

Then I did a cd jvm.options.d to go to the new directory. Then nano'd into the copied jvm.options file.

I uncommented the -Xms and -Xmx settings, and changed them from the default 4GB to 2GB, which is exactly one-half of my VPS' system memory, and what my java memory allocation has been (I checked via my system processes).

Saved the file, and issued a restart via /etc/init.d/elasticsearch restart command. I will report back as to how things go.

Cheers,
Gerry

 

[djbaxter]

I'm very new to ElasticSearch (i.e., just installed this month).

How do you know it's stopped working? Is the first indication some sort of server error in the ACP? Can you assume that if you aren't seeing server errors there all is copasetic?

 

[gerryvz]

Hey @djbaxter —

There are a few things you can do. First of all, you should install the following items:

• Elasticsearch Information add-on from @AndyB. This gives you some control and insight into things.
• Search Improvements add-on from the @Xon. This gives a lot of good options and control on things.
• Standard Library add-on from the @Xon. This is an add-on that provides support functions for other @Xon add-ons.
• Elasticsearch Essentials add-on from the @Xon. It is not yet released for XF 2.2, but should be within days. This allows a full Google-style "typeahead" search box and is amazing stuff. I am TOTALLY bummed that it's been disabled since I went live with 2.2. If you are using 2.1, then it works fine as is.

You'll get a mini info panel in your control panel, as shown below. This is super helpful for monitoring things.



With @AndyB's add-on, you can see things from a URL:

You will see via the Admin panel that there are server errors. Will look like this:





From the command line, there are several things that you can do to monitor Elasticsearch, and control it. I use Ubuntu 18.04, so these are the two commands that I use most:

service elasticsearch status



And if you need to reboot your cluster:

service elasticsearch restart or even better, /etc/init.d/elasticsearch restart

 

[djbaxter]

Thank you, @gerryvz

That is super helpful.

Evidently, I am still running Elasticsearch 7.10.2 (Enhanced search 2.2.1 ) so cPanel on Centos hasn't auto-updated me (yet).

 

[gerryvz]

I would think twice about upgrading to 7.11 at this point in time. I will defintely report back as to what happens with the changes that @MattW and @bzcomputers outline above incorporated. It appears that the Elastic folks made some undocumented changes in the config readfile setup. I am very grateful to have folks like the two of them here to find this kind of stuff, and offer solutions to remedy it.

Your commands (per what I outlined above) may be different on CentOS, from what I posted for Ubuntu/Debian-based systems. @AndyB is a CentOS whiz, for reference. More than anyone else in the XF world, he has been the biggest help to me over the past two years.

I know that CentOS and Ubuntu have the Elasticsearch config files in different locations.

From what I read recently, it appears that the future of CentOS is somewhat in doubt?

 

[djbaxter]

Yes, @AndyB has been super helpful with a lot of server configuration issues in the past couple of years. A genuinely nice human being who is enthusiastic about what he does and who is very generous with his time and knowledge.

 

[djbaxter]

From what I read recently, it appears that the future of CentOS is somewhat in doubt?

I'm not sure it's CentOS per se that's in doubt. It's more how they will handle the PHP 8 issue. But in any CentOS is safe enough for the next 2 years or so and that gives them plenty of time to work out what they are going to do moving forward. One of the great things about open source is that even abandoned projects often get taken up by someone new.

 

[gerryvz]

Yes, @AndyB has been super helpful with a lot of server configuration issues in the past couple of years. A genuinely nice human being who is enthusiastic about what he does and who is very generous with his time and knowledge.

I was lucky enough to meet @AndyB in person in late 2019 while on a business trip, and treat him to lunch. Hopefully post-pandemic I can do this again. He's even more kind and generous in person, than he is online.

 

[MattW]

From what I read recently, it appears that the future of CentOS is somewhat in doubt?

CentOS was supposed to be a downstream binary compatible community version of RedHat. Since they announced that CentOS8 is going in front of RedHat in the form of CentOS stream. Also kicking everyone in the nuts who had just started to roll out into production as a replacement for CentOS 6 or 7.

AlmaLinux is so far looking like the best alternative so far. I’ve been testing this as work with RedHat rpms for stuff like centrify for centralised authentication, and it works exactly as expected.

 

[gerryvz]

@MattW -- what do you think of Oracle Linux, which is supposed to be fully compatible with RHEL? Any experience with it?

Is there any compelling reason why folks pick CentOS vs. Ubuntu, or vice versa?

 

[MattW]

@MattW -- what do you think of Oracle Linux, which is supposed to be fully compatible with RHEL? Any experience with it?

The best way to spoil any server is to install oracle on it!

If you were running oracle, yeah, I’d say go for it, but for any other workload, I wouldn’t bother.

Even though we support up to Oracle 19c at work, we aren’t even considering swapping to Oracle Linux.

 

[gerryvz]

I will report back as to how things go.

It's been about six hours since I rebooted the Elasticsearch engine, and it's still up and running with no errors thrown.

Fingers crossed.