Elastic Search keeping crashing on VPS

PJK · Sep 4, 2018

I recently setup elasticsearch on my server and bought the Xenforo addon. However, the elasticsearch process keeps ending and has to be manually restarted. Can this run on a VPS okay? What suggestions do you have to fix this? The error is below when searching on the forum. Thanks.

XFES\Elasticsearch\ConnectException: cURL error 7: Failed to connect to localhost port 9200: Connection refused in src/addons/XFES/Elasticsearch/Api.php at line 368

XFES\Elasticsearch\Api->request() in src/addons/XFES/Elasticsearch/Api.php at line 90

XFES\Elasticsearch\Api->version() in src/addons/XFES/Elasticsearch/Api.php at line 107

XFES\Elasticsearch\Api->majorVersion() in src/addons/XFES/Search/Source/Elasticsearch.php at line 226

XFES\Search\Source\Elasticsearch->getDslFromQuery() in src/addons/XFES/Search/Source/Elasticsearch.php at line 145

XFES\Search\Source\Elasticsearch->search() in src/XF/Search/Search.php at line 164

XF\Search\Search->search() in src/XF/Repository/Search.php at line 34

XF\Repository\Search->runSearch() in src/XF/Pub/Controller/Search.php at line 381

XF\Pub\Controller\Search->runSearch() in src/XF/Pub/Controller/Search.php at line 239

XF\Pub\Controller\Search->actionMember() in src/XF/Mvc/Dispatcher.php at line 249

XF\Mvc\Dispatcher->dispatchClass() in src/XF/Mvc/Dispatcher.php at line 88

XF\Mvc\Dispatcher->dispatchLoop() in src/XF/Mvc/Dispatcher.php at line 41

XF\Mvc\Dispatcher->run() in src/XF/App.php at line 1931

XF\App->run() in src/XF.php at line 328

XF::runApp() in index.php at line 13

MattW · Sep 4, 2018

Yes it can be run on a VPS OK. You need to check the logs for Elasticsearch and the server to see why it's crashing / being killed.

PJK · Sep 4, 2018

MattW said:
Yes it can be run on a VPS OK. You need to check the logs for Elasticsearch and the server to see why it's crashing / being killed.

The process was killed by OOM error:
===
Sep 4 05:32:34 : OOM killed process 807450 (java) score 0 vm:10914168kB, rss:1020388kB, swap:0kB
===

I started it over:
====
root@host [/var/log/elasticsearch]# /etc/init.d/elasticsearch status
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: signal) since Tue 2018-09-04 05:32:34 EDT; 1h 59min ago
Docs: http://www.elastic.co
Process: 29115 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=killed, signal=KILL)
Main PID: 29115 (code=killed, signal=KILL)
CGroup: /system.slice/elasticsearch.service

Sep 03 00:29:36 host systemd[1]: Started Elasticsearch.
Sep 03 00:29:36 host systemd[1]: Starting Elasticsearch...
Sep 04 05:32:34 host systemd[1]: elasticsearch.service: main process exited, code=killed, status=9/KILL
Sep 04 05:32:34 host systemd[1]: Unit elasticsearch.service entered failed state.
Sep 04 05:32:34 host systemd[1]: elasticsearch.service failed.

root@host [/var/log/elasticsearch]# /etc/init.d/elasticsearch start
Starting elasticsearch (via systemctl): [ OK ]

root@host [/var/log/elasticsearch]# /etc/init.d/elasticsearch status
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2018-09-04 07:32:31 EDT; 9s ago
Docs: http://www.elastic.co
Main PID: 5292 (java)
CGroup: /system.slice/elasticsearch.service
├─5292 /bin/java -Xms512m -Xmx512m -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio...
└─5422 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Sep 04 07:32:31 host systemd[1]: Started Elasticsearch.
Sep 04 07:32:31 host systemd[1]: Starting Elasticsearch...
====

The host recommends upgrading. There is 5GB of RAM on the server. How much is recommended?
Thanks.

MattW · Sep 4, 2018

It depends on the size of your board.

How much RAM have you allocated to Elasticsearch (or are you allocating the default 2GB by default)?

PJK · Sep 4, 2018

MattW said:
It depends on the size of your board.

How much RAM have you allocated to Elasticsearch (or are you allocating the default 2GB by default)?

The board is 1.25M posts. I'l check with the host about the RAM but I assume it is the default.

MattW · Sep 4, 2018

If it's a VPS you manage, have at look in /etc/elasticsearch/jvm.options

Code:

# cat /etc/elasticsearch/jvm.options | egrep "Xms|Xmx"
-Xms64m
-Xmx64m

The latest version of ES can get away with a couple of hundred MB of RAM per million posts (obviously more if you have additional content types being indexed such as conversations).

PJK · Sep 4, 2018

MattW said:
If it's a VPS you manage, have at look in /etc/elasticsearch/jvm.options

Code:

# cat /etc/elasticsearch/jvm.options | egrep "Xms|Xmx" -Xms64m -Xmx64m

The latest version of ES can get away with a couple of hundred MB of RAM per million posts (obviously more if you have additional content types being indexed such as conversations).

Thanks, can you recommend your settings so I can let my host know? My host is suggesting adding more RAM to the server. What do you limit the elasticsearch's RAM to?

TPerry · Sep 4, 2018

If you are getting OOM errors (out of memory) then a RAM upgrade may be in order unless you fine tune ES.

Alpha1 · Sep 4, 2018

I am using a 4GB RAM 2vCPU VPS from digital ocean for ES alone. In the past I had too many issues from having ES on the same server. Its using 86% of the RAM. I have 2 million posts if you count deleted posts. 1.6 million without.

PJK · Sep 5, 2018

Tracy Perry said:
If you are getting OOM errors (out of memory) then a RAM upgrade may be in order unless you fine tune ES.

Can you recommend what ES settings to make? Thanks.

Alfa1 said:
I am using a 4GB RAM 2vCPU VPS from digital ocean for ES alone. In the past I had too many issues from having ES on the same server. Its using 86% of the RAM. I have 2 million posts if you count deleted posts. 1.6 million without.

Ah, so you have a server just for ES alone? What's that server cost each month to run? My host is recommending I add RAM, but I'm not sure how much. I host other sites on the server as well and up until installing ES, the server has worked great for years.

Alpha1 · Sep 5, 2018

I am using a cheap Digital Ocean droplet (VPS) for it. It costs me $20 a month, but you can likely use a cheaper one.
https://www.digitalocean.com/pricing/
Separating it avoids issues spilling over between ES and webserver and database server. Separating everything into cheap segments avoids one segment affecting the whole site and also avoids paying top dollar for things that can easily be hosted on cheap nodes. I run my big board on 6 droplets: ES, DB, Web Server, Mail, Http Proxy, Backup. All are cheap droplets, except the database one. This slashed my hosting costs in half and no longer have ES issues.

PJK · Sep 11, 2018

I went ahead and added RAM to the server, and now enabled ES search again, rebuilding the index and it stops not long after the rebuild and the server error is:

Elasticsearch indexing error (queued): cURL error 28: Operation timed out after 20000 milliseconds with 0 bytes received 3 minutes ago src/addons/XFES/Elasticsearch/Api.php:368

What do I need to do to fix this?
Thanks.

TPerry · Sep 11, 2018

PJK said:
What do I need to do to fix this?
Thanks.

Is outbound ports open to allow curl to connect? Sounds like a firewall issue.

PJK · Sep 11, 2018

Tracy Perry said:
Is outbound ports open to allow curl to connect? Sounds like a firewall issue.

I had the host disable the firewall and still getting this issue. I also see this on the error log:

Elasticsearch indexing error: cURL error 28: Operation timed out after 20000 milliseconds with 0 bytes received

3 minutes ago

src/addons/XFES/Elasticsearch/Api.php:368

Elasticsearch error: all shards failed

Sanmu · Apr 29, 2022

Hi there, I'm on Linus.

bash: /var/log/elasticsearch: Is a directory
[root@iZj6c2xxxxxxxjahou8rZ admin]#

What to do next?
What's the command to check the logs for Elasticsearch and the server?

duderuud · Apr 29, 2022

Check the files in that directory. Normally it's called <clustername>.log

Elastic Search keeping crashing on VPS

PJK

Active member

MattW

Well-known member

PJK

Active member

MattW

Well-known member

PJK

Active member

MattW

Well-known member

PJK

Active member

TPerry

Well-known member

Alpha1

Well-known member

PJK

Active member

Alpha1

Well-known member

PJK

Active member

TPerry

Well-known member

PJK

Active member

Sanmu

Active member

duderuud

Well-known member

Similar threads

We value your privacy