Elasticsearch stopping on its own every morning

[Feanor]

It's not happening every morning actually, but I set up Elasticsearch and Enhanced Search for XF about a week ago and the forum has had server errors starting around the same time in the morning for 3-4 days since then. When I looked at the error details it was just a standard error about not being able to connect to Elasticsearch. So then all I do is run "service elasticsearch start" and everything is working again.

I think it might be running out of memory, but I'm not really sure. Why does it work fine all day and then have issues at the same time? Or could it be that something else is causing the server to restart and elasticsearch isn't starting on its own?

Another question, but not sure if it's related or not: The top command shows me that elasticsearch is using 30% of the memory when it's running. This seems like a lot to me. Is this normal?

 

[Paul B]

If it's happening at the same time then it's likely due to another process running on the server, such as an automated backup, a cron task, etc.

 

[Feanor]

If it's happening at the same time then it's likely due to another process running on the server, such as an automated backup, a cron task, etc.

Is it normal for elasticsearch to not start on its own when the server restarts?

 

[Yodrak]

You should tell US more about your Server specs. Wich OS and Hoster are you using?

Ah Just now i read you use "Service Elasticsearch Start" then i Think you use Ubuntu?
If so then you need to Check your systemd.

You can read more about it here

How To Configure a Linux Service to Start Automatically After a Crash or Reboot – Part 1: Practical Examples | DigitalOcean

This tutorial shows you how to configure system services to automatically restart after a crash or a server reboot. We cover the three most common init syste…

www.digitalocean.com

 

[Feanor]

CentOS 7. 16 GB memory VPS. Forum has 10 million posts.

I found this in /var/log/messages from around when searching broke on the forum:

Jul 15 08:10:59 servername kernel: Out of memory: Kill process 23707 (mysqld) score 580 or sacrifice child
Jul 15 08:10:59 servername kernel: Killed process 23707 (mysqld) total-vm:13911008kB, anon-rss:9531840kB, file-rss:0kB, shmem-rss:0kB
Jul 15 08:10:59 servername kernel: elasticsearch[n invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Jul 15 08:10:59 servername kernel: elasticsearch[n cpuset=/ mems_allowed=0

So if I'm understanding correctly, the OS killed elasticsearch because it ran out of memory and this would imply it may not be directly related to elasticsearch itself? However, whenever I run the top command it says elasticsearch is using 30% of the server's memory, which seems like a lot to me, but this is my first time using ES, so I'm not sure. I'm also not really sure where to look to see what was happening at 8:10 to cause the server to run out of memory.

 

[Yodrak]

I cant check now my Elasticsearch size in tmpfs but i guess its around 700 MB with 1.000.000 Posts.

So with 10 Mio. Posts the size of your Elasticsearch folder will be something between 5-10GB i guess.

 

[Feanor]

I cant check now my Elasticsearch size in tmpfs but i guess its around 700 MB with 1.000.000 Posts.

So with 10 Mio. Posts the size of your Elasticsearch folder will be something between 5-10GB i guess.

Here are the stats XF is showing:



Does the size here correspond to the amount of memory it needs?

 

[Yodrak]

Im Not a Elasticsearch expert. But i guess the 2GB are the heap size of Elasticsearch.

Elasticsearch recommendation is 50% of your free RAM.

Fid you check your database wich size your xf_search table have now? If it's full you can empty this table, because you're using Elasticsearch.

 

[Feanor]

My Java heap size was at 4GB. It seems like a lot of articles online recommend setting one larger than needed (?). Based on this discussion, I'm thinking it's too high. I now have it set it to 1GB and I'll see how it goes.