ES 2.2 Elastic monitor and restart with WHM chkservd service, radom stops

[philmckrackon]

My service randomly stops an my java logs are empty so I want to monitor via chkservd until I figure this out.

I have added the Elastic monnittor to
in /etc/chkserv.d/chkservd.conf
as
elasticsearch:1

and also vied (editor)
elasticsearch in /etc/chkserv.d
with line
service[elasticsearch]=9200,curl -XGET 'http://localhost:9200',200,/bin/systemctl restart elasticsearch.service,elasticsearch,root

Am I querying correctly?

If not, what should the query be as well as tthe rretun?

THanks,

 

[philmckrackon]

Thoughts? Is '200' the correct return value?

 

[Travis-Mc]

Did this work for you?

 

[philmckrackon]

It works now that I found a list of return codes here -->> https://curl.se/libcurl/c/libcurl-errors.html and changed the expected return code from 200 to 0

service[elasticsearch]=9200,curl -XGET 'http://localhost:9200',200,/bin/systemctl restart elasticsearch.service,elasticsearch,root

to

service[elasticsearch]=9200,curl -XGET 'http://localhost:9200',0,/bin/systemctl restart elasticsearch.service,elasticsearch,root

This is the ftp example I modified. -->> https://support.cpanel.net/hc/en-us...-to-add-a-new-service-for-chkservd-to-monitor

 

[Travis-Mc]

That is very helpful. Thank you so much.

 

[Travis-Mc]

@philmckrackon I am not able to get this to fully work. I keep getting false positives of the service being down when it is not. Did you happen to run into that issue?

 

[philmckrackon]

What I posted works for me but I am on CentOS 7. What is your OS and what is your curl statement? It may be different and may not need the ' '. I would use a ssh terminal to check all your commands and modify as needed.

 

[Travis-Mc]

CentOS 7 as well. I tried each of these one at a time:

service[elasticsearch]=9200,curl -XGET 'http://localhost:9200',0,/bin/systemctl restart elasticsearch.service,elasticsearch,root
service[elasticsearch]=9200,curl -XGET 'http://127.0.0.1:9200',0,/bin/systemctl restart elasticsearch.service,elasticsearch,root
service[elasticsearch]=x,x,x,/bin/systemctl restart elasticsearch.service,elasticsearch,root

I also restarted chkservd and elasticsearch with each change.

In the file /var/run/chkservd/elasticsearch I changed the - to a +, but that didn't seem to do anything either.

Everything done via SSH

 

[Travis-Mc]

The command

# curl -XGET 'http://localhost:9200'

Returns this (I changed some things to ***** because I don't know what is okay to display publicly):

{
  "name" : "*****",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "*****",
  "version" : {
    "number" : "7.16.2",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "*****",
    "build_date" : "2021-12-18T19:42:46.604893745Z",
    "build_snapshot" : false,
    "lucene_version" : "8.10.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

 

[philmckrackon]

Sorry for the late reply, apparently WHM did an update an now elasticsearch is restarting all the time killings my resources - once I get this tackled I'll t back...

 

[Travis-Mc]

I gave up trying to figure this one out for now. If you come up with something, let me know.

 

[philmckrackon]

I gave up trying to figure this one out for now. If you come up with something, let me know.

Sorry for never replying, I thought I had. The reason why I am now is I received two emails last night that the process had failed then recovered and I thought of this thread. Any way, here was the process I used incase it helps anyone in the future.

In summery, the chkservd Elasticsearch service monitor / start command I use is service[elasticsearch]=9200,curl -XGET,HTTP/1.0 400 Ba,/bin/systemctl start elasticsearch.service

This is how I derived this command.

Be sure you have Elasticsearch installed as a service and is monitored.


Check with curl -v localhost:9200 and copy return. (you have to use -v verbose)


Edit Elasticsearch to service[elasticsearch]=9200,curl -XGET,HTTP/1.1 200 OK,/bin/systemctl start elasticsearch.service

Stop Elasticsearch and check for fail service email (should be no longer than five minutes and you need to be sure WHM is correctly configured.)
Look at expected reply not equal ( != ) to actual reply. Yours may differ.


Edit Elasticsearch to correct expected return, start service then stop service. You should get a new fail but reason is as - Connection refused : Died


Wait for service to recover.

 

[Travis-Mc]

Thank you for updating this! I will give it a try.

Which O.S. are you running?

 

[philmckrackon]

Thanks,
CentOS v7.9.2009

 

[philmckrackon]

Thank you for updating this! I will give it a try.

Which O.S. are you running?

Thanks,
CentOS v7.9.2009

FYI @Travis-Mc - A new update. Since CentOS v7 is no longer supported and EOL is slated for Jul. 2024 I proactively changed OS to Almalinux 8.
These instruction (Elasticsearch Service Monitor) apply as well. There was no problem with my OS change.

​

 

[Travis-Mc]

Phil, you've gone above and beyond. Thank you!