Elasticsearch gets killed every morning

[Renegade]

Every morning, between 5AM to 9AM, the elasticsearch service stops. I could see only the following in the syslog regarding elastic search.

hhjjp-172-26-6-79:/var/log# grep -i elasticsearch syslog
Oct 10 11:50:55 ip-172-26-6-79 systemd[1]: Starting Elasticsearch...
Oct 10 11:51:37 ip-172-26-6-79 systemd[1]: Started Elasticsearch.
Oct 11 00:38:24 ip-172-26-6-79 systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
Oct 11 00:38:24 ip-172-26-6-79 systemd[1]: elasticsearch.service: Unit entered failed state.
Oct 11 00:38:24 ip-172-26-6-79 systemd[1]: elasticsearch.service: Failed with result 'signal'.

My servers config is as below and I have over 2M posts.

2GB RAM, Ubuntu (bitnami LAMP stack)

-Xms512m
-Xmx512m

I have noticed that if I try to start the elasticsearch service manually then it fails to start with 'start operation timed out' error. However if I reboot the server, then the service starts fine. I am assuming that this is because the server has more free RAM when it reboots as opposed to a server which has been running for over 20 hours.

 

[Mr. Jinx]

Is this server also running your forum, webserver+database? In that case 2GB memory would be tricky.
I would recommend running elasticsearch on a separate VPS.

 

[Renegade]

Yes it is. Alright, so should I look at upgrading to 4GB memory, or a separate 2GB server only for elasticsearch. Have to keep the costs also in check. Will also explore managed elastic search.

 

[dknife]

I couldn't run Elasticsearch on my 4Gb VPS with 2Gb allocated to InnoDB cache.

 

[Mr. Jinx]

I'm running ES on a 2GB/2CPU Cloud VPS from Hetzner.com. Runs perfect! This machine is dedicated for running ES.
It costs about 4 euro a month, and if you don't like it you can kill your server and stop paying. Very flexible.

Don't forget to protect your ES install with password and only allow the incoming IP from your hostingprovider.
And you should add letsencrypt so you can connect over https.

I also looked at managed ES, but it is more expensive. AWS has a free tier service for ES to play with. If you need more speed you'll have to pay for it.