Elasticsearch 7.11

MattW

MattW

Well-known member
Just a heads up on this one. They have changed how the heap memory is defined and managed:

Code: 
################################################################
## IMPORTANT: JVM heap size
################################################################
##
## The heap size is automatically configured by Elasticsearch
## based on the available memory in your system and the roles
## each node is configured to fulfill. If specifying heap is
## required, it should be done through a file in jvm.options.d,
## and the min and max should be set to the same value. For
## example, to set the heap to 4 GB, create a new file in the
## jvm.options.d directory containing these lines:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################

It's no longer controlled by the jvm.options file, and if you server automatically restarts ES after the update (like one of my cPanel servers has just done), you'll probably run out of Memory!

On my server running my site, I previously had 1GB allocated to Java, with the new setup, it's restarted, and ballooned to RAM usage to attempt to use 50% (32GB!)
 
Sort by date Sort by votes
Elasticsearch hasn't thrown a single error in almost 24 hours, and has been working as normal since yesterday morning when I created a new prefs file with heap limit, as indicated above in post #9. All seems to be well, and I will tentatively call the problem solved.

If you are using Elasticsearch 7.11, hopefully this thread will help you.

Again, thanks to @bzcomputers and @MattW for identifying the issue and suggesting a fix.
 
Upvote 1 Downvote
gerryvz said:
Thanks for this info and also to @MattW for the info in the original post. I’ll add the settings and see how things go. This is really frustrating.


UPDATE
What I did was to go to /etc/elasticsearch and found the jvm.options file, as well as the jvm.options.d directory.

I did a cp jvm.options jvm.options.d to copy the jvm.options file into the jvm.options.d directory.

Then I did a cd jvm.options.d to go to the new directory. Then nano'd into the copied jvm.options file.

I uncommented the -Xms and -Xmx settings, and changed them from the default 4GB to 2GB, which is exactly one-half of my VPS' system memory, and what my java memory allocation has been (I checked via my system processes).

Saved the file, and issued a restart via /etc/init.d/elasticsearch restart command. I will report back as to how things go.

Cheers,
Gerry
Click to expand...
Thanks for these, Gerry.

One comment/correction is that the items marked DIRECTORY above are actually just FILES, perhaps this is a CENTOS difference? It was a bit confusing for me. I also have not been upgraded yet.

so, for Cent OS/cpanel i used: (i prefer pico to nano)
EDIT: DO NOT FOLLOW THIS, jump ahead to
xenforo.com

Elasticsearch 7.11

Just a heads up on this one. They have changed how the heap memory is defined and managed: ################################################################ ## IMPORTANT: JVM heap size ################################################################ ## ## The heap size is automatically...
xenforo.com xenforo.com

Code: 
cd /etc/elasticsearch
cp jvm.options jvm.options.d
pico jvm.options.d

and everything was already the same as before so i just X'ed out of the file.

and finally issued the restart
Code: 
/etc/init.d/elasticsearch restart

So, not sure where you are getting directories....?


Hopefully this will prevent issues when it does update.
 
Last edited:
Upvote 0 Downvote
Yes, as mentioned there may be differences in location of the file between Ubuntu/Debian and CentOS systems. I run Ubuntu, and the instructions above are 100% correct.

When you do a ls -l of the etc directory in Ubuntu, you will see the following:

Screen Shot 2021-02-15 at 10.22.47 AM.jpg


And then when you do a cd elasticsearch, you will see the following:

Screen Shot 2021-02-15 at 10.23.11 AM.jpg


The jvm.options.d directory is where you want to put the new config file. The jvm.options.d directory was already there (pre-existing) and all I did was copy the jvm.options file into that jvm.options.d directory, and then tweak it as mentioned.

I don't know where elasticsearch puts files and directories in a CentOS system, as I don't run that operating system.
 
Last edited:
Upvote 0 Downvote
For CentOS, all you need to do is create a new file in the jvm.options.d folder that is appended with the .options extension

You set any additional configuration parameters in there, such as the Xms / Xmx values.

Code: 
[15:29][root@XXX jvm.options.d]# pwd
/etc/elasticsearch/jvm.options.d
[15:29][root@XXX jvm.options.d]# ll
total 12
drwxr-s--- 2 root elasticsearch 4096 Feb 15 15:29 ./
drwxr-s--- 3 root elasticsearch 4096 Feb 10 22:24 ../
-rw-r--r-- 1 root elasticsearch   14 Feb 10 22:24 heap.options
[15:29][root@XXX jvm.options.d]# cat heap.options
-Xms4g
-Xmx4g
[15:30][root@hz1.mattwservices.uk jvm.options.d]#
 
Upvote 0 Downvote
got it.

ok, so because i am pre-creating this before i actually have been upgraded, i didn't have the directory jvm.options.d

So,
don't do this either:
Code: 
cd /etc/elasticsearch
mkdir jvm.options.d
cp jvm.options ./jvm.options.d/heap.options
 
Last edited:
Upvote 0 Downvote
No, you don't need to do this before the upgrade. Upgrade Elasticsearch, stop the service, create the required new options file. You don't need to copy the whole jvm.options file into the new directory, just create the file with the required values you want to specify. Start elasticsearch back up.
 
Upvote 0 Downvote
Like @MattW showed, you can just create a simple text file in the jvm.options.d directory with ONLY the heap settings. I opted to copy over my entire jvm.options file and then un-comment & modify the two lines in it with the heap settings.

@MattW's way is simpler/cleaner, though what I outlined above works just fine as well.
 
Upvote 0 Downvote
I backed out my changes.
ran the upgrade process from 6.8 to 7.11 as outlined here: https://www.elastic.co/guide/en/elasticsearch/reference/current/restart-upgrade.html

and now it won't start.

# sudo -i service elasticsearch start
Starting elasticsearch (via systemctl): Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
[FAILED]

nothing meaningful in either of those logs i don't think...
Code: 
# systemctl status elasticsearch.service -l
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2021-02-15 12:20:58 EST; 21s ago
     Docs: https://www.elastic.co
  Process: 22183 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 22183 (code=exited, status=1/FAILURE)
   CGroup: /system.slice/elasticsearch.service

Feb 15 12:20:58 myhost.com systemd-entrypoint[22183]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
Feb 15 12:20:58 myhost.com systemd-entrypoint[22183]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
Feb 15 12:20:58 myhost.com systemd-entrypoint[22183]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
Feb 15 12:20:58 myhost.com systemd-entrypoint[22183]: at org.elasticsearch.cli.Command.main(Command.java:79)
Feb 15 12:20:58 myhost.com systemd-entrypoint[22183]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
Feb 15 12:20:58 myhost.com systemd-entrypoint[22183]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)
Feb 15 12:20:58 myhost.com systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Feb 15 12:20:58 myhost.com systemd[1]: Failed to start Elasticsearch.
Feb 15 12:20:58 myhost.com systemd[1]: Unit elasticsearch.service entered failed state.
Feb 15 12:20:58 smyhost.com systemd[1]: elasticsearch.service failed.

Code: 
-- Subject: Unit elasticsearch.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has begun starting up.

Feb 15 12:37:02myhost.comsystemd-entrypoint[27604]: fatal error in thread [main], exiting
Feb 15 12:37:02myhost.comsystemd-entrypoint[27604]: java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
Feb 15 12:37:02myhost.comsystemd-entrypoint[27604]: at org.elasticsearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:23)
Feb 15 12:37:02myhost.comsystemd-entrypoint[27604]: at java.base/java.security.AccessController.doPrivileged(AccessController.java:312)
Feb 15 12:37:02myhost.comsystemd-entrypoint[27604]: at org.elasticsearch.systemd.Libsystemd.<clinit>(Libsystemd.java:22)
Feb 15 12:37:02myhost.comsystemd-entrypoint[27604]: at org.elasticsearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:115)
Feb 15 12:37:02myhost.comsystemd-entrypoint[27604]: at org.elasticsearch.systemd.SystemdPlugin.onNodeStarted(SystemdPlugin.java:126)
Feb 15 12:37:02myhost.comsystemd-entrypoint[27604]: at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Feb 15 12:37:02myhost.comsystemd-entrypoint[27604]: at org.elasticsearch.node.Node.start(Node.java:909)
 
Last edited:
Upvote 0 Downvote
Upvote 0 Downvote
Yeah, that was introduced in 7.5.2 which I found out back in January 2020:

1613411163367.png

The actual log files give more specific information relating to that issue (I created my own article on it as I keep all my fault finding notes)

Code: 
[15:50][root@ovh1.mattwservices.uk elasticsearch]# cat elasticsearch.log
[2020-01-22T15:30:51,415][WARN ][o.e.b.Natives            ] [ovh1.mattwservices.uk] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-17621466510065462312/jna--1985354563/jna2353222043769968013.tmp: /tmp/elasticsearch-17621466510065462312/jna--1985354563/jna2353222043769968013.tmp: failed to map segment from shared object: Operation not permitted
        at java.lang.ClassLoader$NativeLibrary.load0(Native Method) ~[?:?]
        at java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2440) ~[?:?]
        at java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2497) ~[?:?]
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2694) ~[?:?]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2627) ~[?:?]
        at java.lang.Runtime.load0(Runtime.java:744) ~[?:?]
        at java.lang.System.load(System.java:1873) ~[?:?]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:?]
        at java.lang.Class.forName(Class.java:333) ~[?:?]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:104) [elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:190) [elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:369) [elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) [elasticsearch-cli-7.5.2.jar:7.5.2]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.5.2.jar:7.5.2]
[2020-01-22T15:30:51,422][WARN ][o.e.b.Natives            ] [ovh1.mattwservices.uk] cannot check if running as root because JNA is not available
[2020-01-22T15:30:51,423][WARN ][o.e.b.Natives            ] [ovh1.mattwservices.uk] cannot install system call filter because JNA is not available
[2020-01-22T15:30:51,423][WARN ][o.e.b.Natives            ] [ovh1.mattwservices.uk] cannot register console handler because JNA is not available
[2020-01-22T15:30:51,424][WARN ][o.e.b.Natives            ] [ovh1.mattwservices.uk] cannot getrlimit RLIMIT_NPROC because JNA is not available
[2020-01-22T15:30:51,424][WARN ][o.e.b.Natives            ] [ovh1.mattwservices.uk] cannot getrlimit RLIMIT_AS because JNA is not available
[2020-01-22T15:30:51,424][WARN ][o.e.b.Natives            ] [ovh1.mattwservices.uk] cannot getrlimit RLIMIT_FSIZE because JNA is not available
[2020-01-22T15:30:51,859][INFO ][o.e.e.NodeEnvironment    ] [ovh1.mattwservices.uk] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [375.4gb], net total_space [411.4gb], types [rootfs]
[2020-01-22T15:30:51,860][INFO ][o.e.e.NodeEnvironment    ] [ovh1.mattwservices.uk] heap size [989.8mb], compressed ordinary object pointers [true]
[2020-01-22T15:30:51,862][INFO ][o.e.n.Node               ] [ovh1.mattwservices.uk] node name [ovh1.mattwservices.uk], node ID [ODcNZ0kcRUu4H6Qg3ZNCyA], cluster name [elasticsearch]
[2020-01-22T15:30:51,862][INFO ][o.e.n.Node               ] [ovh1.mattwservices.uk] version[7.5.2], pid[32278], build[default/rpm/8bec50e1e0ad29dad5653712cf3bb580cd1afcdf/2020-01-15T12:11:52.313576Z], OS[Linux/3.10.0-1062.1.1.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13.0.1/13.0.1+9]
[2020-01-22T15:30:51,862][INFO ][o.e.n.Node               ] [ovh1.mattwservices.uk] JVM home [/usr/share/elasticsearch/jdk]
[2020-01-22T15:30:51,862][INFO ][o.e.n.Node               ] [ovh1.mattwservices.uk] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-17621466510065462312, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2020-01-22T15:30:52,955][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [aggs-matrix-stats]
[2020-01-22T15:30:52,955][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [analysis-common]
[2020-01-22T15:30:52,955][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [flattened]
[2020-01-22T15:30:52,955][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [frozen-indices]
[2020-01-22T15:30:52,956][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [ingest-common]
[2020-01-22T15:30:52,956][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [ingest-geoip]
[2020-01-22T15:30:52,956][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [ingest-user-agent]
[2020-01-22T15:30:52,956][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [lang-expression]
[2020-01-22T15:30:52,956][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [lang-mustache]
[2020-01-22T15:30:52,956][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [lang-painless]
[2020-01-22T15:30:52,956][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [mapper-extras]
[2020-01-22T15:30:52,956][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [parent-join]
[2020-01-22T15:30:52,956][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [percolator]
[2020-01-22T15:30:52,957][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [rank-eval]
[2020-01-22T15:30:52,957][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [reindex]
[2020-01-22T15:30:52,957][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [repository-url]
[2020-01-22T15:30:52,957][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [search-business-rules]
[2020-01-22T15:30:52,957][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [spatial]
[2020-01-22T15:30:52,957][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [systemd]
[2020-01-22T15:30:52,957][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [transform]
[2020-01-22T15:30:52,957][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [transport-netty4]
[2020-01-22T15:30:52,957][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [vectors]
[2020-01-22T15:30:52,958][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [x-pack-analytics]
[2020-01-22T15:30:52,958][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [x-pack-ccr]
[2020-01-22T15:30:52,958][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [x-pack-core]
[2020-01-22T15:30:52,958][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [x-pack-deprecation]
[2020-01-22T15:30:52,958][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [x-pack-enrich]
[2020-01-22T15:30:52,958][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [x-pack-graph]
[2020-01-22T15:30:52,958][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [x-pack-ilm]
[2020-01-22T15:30:52,958][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [x-pack-logstash]
[2020-01-22T15:30:52,958][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [x-pack-ml]
[2020-01-22T15:30:52,959][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [x-pack-monitoring]
[2020-01-22T15:30:52,959][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [x-pack-rollup]
[2020-01-22T15:30:52,959][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [x-pack-security]
[2020-01-22T15:30:52,959][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [x-pack-sql]
[2020-01-22T15:30:52,959][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [x-pack-voting-only-node]
[2020-01-22T15:30:52,959][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] loaded module [x-pack-watcher]
[2020-01-22T15:30:52,959][INFO ][o.e.p.PluginsService     ] [ovh1.mattwservices.uk] no plugins loaded
[2020-01-22T15:30:55,120][INFO ][o.e.x.s.a.s.FileRolesStore] [ovh1.mattwservices.uk] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2020-01-22T15:30:55,411][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [ovh1.mattwservices.uk] [controller/32484] [Main.cc@110] controller (64 bit): Version 7.5.2 (Build 68f6981dfb8e2d) Copyright (c) 2020 Elasticsearch BV
[2020-01-22T15:30:55,703][DEBUG][o.e.a.ActionModule       ] [ovh1.mattwservices.uk] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2020-01-22T15:30:55,808][INFO ][o.e.d.DiscoveryModule    ] [ovh1.mattwservices.uk] using discovery type [zen] and seed hosts providers [settings]
[2020-01-22T15:30:56,376][INFO ][o.e.n.Node               ] [ovh1.mattwservices.uk] initialized
[2020-01-22T15:30:56,377][INFO ][o.e.n.Node               ] [ovh1.mattwservices.uk] starting ...
[2020-01-22T15:30:56,492][INFO ][o.e.t.TransportService   ] [ovh1.mattwservices.uk] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2020-01-22T15:30:56,603][WARN ][o.e.b.BootstrapChecks    ] [ovh1.mattwservices.uk] system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2020-01-22T15:30:56,603][WARN ][o.e.b.BootstrapChecks    ] [ovh1.mattwservices.uk] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2020-01-22T15:30:56,612][INFO ][o.e.c.c.ClusterBootstrapService] [ovh1.mattwservices.uk] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2020-01-22T15:30:59,616][INFO ][o.e.c.c.Coordinator      ] [ovh1.mattwservices.uk] setting initial configuration to VotingConfiguration{ODcNZ0kcRUu4H6Qg3ZNCyA}
[2020-01-22T15:30:59,678][INFO ][o.e.c.s.MasterService    ] [ovh1.mattwservices.uk] elected-as-master ([1] nodes joined)[{ovh1.mattwservices.uk}{ODcNZ0kcRUu4H6Qg3ZNCyA}{yoP8OB00QXKeabzvTsESnw}{127.0.0.1}{127.0.0.1:9300}{dilm}{ml.machine_memory=33237557248, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: master node changed {previous [], current [{ovh1.mattwservices.uk}{ODcNZ0kcRUu4H6Qg3ZNCyA}{yoP8OB00QXKeabzvTsESnw}{127.0.0.1}{127.0.0.1:9300}{dilm}{ml.machine_memory=33237557248, xpack.installed=true, ml.max_open_jobs=20}]}
[2020-01-22T15:30:59,693][INFO ][o.e.c.c.CoordinationState] [ovh1.mattwservices.uk] cluster UUID set to [GC_gUG5ETBuuWfaB0sq9Aw]
[2020-01-22T15:30:59,700][INFO ][o.e.c.s.ClusterApplierService] [ovh1.mattwservices.uk] master node changed {previous [], current [{ovh1.mattwservices.uk}{ODcNZ0kcRUu4H6Qg3ZNCyA}{yoP8OB00QXKeabzvTsESnw}{127.0.0.1}{127.0.0.1:9300}{dilm}{ml.machine_memory=33237557248, xpack.installed=true, ml.max_open_jobs=20}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
[2020-01-22T15:30:59,733][INFO ][o.e.h.AbstractHttpServerTransport] [ovh1.mattwservices.uk] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2020-01-22T15:30:59,733][INFO ][o.e.n.Node               ] [ovh1.mattwservices.uk] started
[2020-01-22T15:30:59,735][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [ovh1.mattwservices.uk] fatal error in thread [main], exiting
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
        at org.elasticsearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:34) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
        at org.elasticsearch.systemd.Libsystemd.<clinit>(Libsystemd.java:33) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:114) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.onNodeStarted(SystemdPlugin.java:124) ~[?:?]
        at java.util.ArrayList.forEach(ArrayList.java:1507) ~[?:?]
        at org.elasticsearch.node.Node.start(Node.java:762) ~[elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:293) ~[elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:378) ~[elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.5.2.jar:7.5.2]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.5.2.jar:7.5.2]
 
Last edited:
Upvote 0 Downvote
Following up on this situation, I am still having to restart Elasticsearch via my Terminal about two times per day. I am still getting the same curl error 7, failure to connect to the server.

Something happened with the Elasticsearch 7.11 update that exacerbated this condition, causing this error condition to happen multiple times per day from about once every week or two previously.

I have not delved into the error logs to see what is happening and being recorded that is causing this problem.
 
Upvote 0 Downvote
Upgraded 2 days ago and everything is running OK.

Just add same "option file" to jvm.options.d folder.
Restart ES and voila

Code: 
# service elasticsearch status
â— elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2021-02-15 14:34:27 CET; 1 day 22h ago
     Docs: https://www.elastic.co
 Main PID: 23290 (java)
   CGroup: /system.slice/elasticsearch.service
           â”─23290 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress...
           └─23490 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Feb 15 14:34:12 upcloud.pijanitvor.com systemd[1]: Starting Elasticsearch...
Feb 15 14:34:13 upcloud.pijanitvor.com systemd-entrypoint[23290]: [0.003s][warning][logging] Output options for existin...ed.
Feb 15 14:34:27 upcloud.pijanitvor.com systemd[1]: Started Elasticsearch.
Hint: Some lines were ellipsized, use -l to show in full.

13_13_55.webp
 
Upvote 0 Downvote
Sunka said:
Upgraded 2 days ago and everything is running OK.

Just add same "option file" to jvm.options.d folder.
Restart ES and voila

Code: 
# service elasticsearch status
â— elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2021-02-15 14:34:27 CET; 1 day 22h ago
     Docs: https://www.elastic.co
Main PID: 23290 (java)
   CGroup: /system.slice/elasticsearch.service
           â”─23290 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress...
           └─23490 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Feb 15 14:34:12 upcloud.pijanitvor.com systemd[1]: Starting Elasticsearch...
Feb 15 14:34:13 upcloud.pijanitvor.com systemd-entrypoint[23290]: [0.003s][warning][logging] Output options for existin...ed.
Feb 15 14:34:27 upcloud.pijanitvor.com systemd[1]: Started Elasticsearch.
Hint: Some lines were ellipsized, use -l to show in full.

View attachment 246551
Click to expand...
What was the memory setting you used? How much overall system memory does your server have?
 
Upvote 0 Downvote
You must log in or register to reply here.

Similar threads

Sim
ElasticSearch 8 on a system with 1GB RAM
Replies
1
Views
673
ivp
I
N
  • Question Question
ES 2.2 Allocated memory Elasticsearch
Replies
1
Views
743
nanto
N
Sunka
  • Question Question
How To Install Elasticsearch 7.x for Xenforo on CentOS
2 3
Replies
47
Views
13K
phucvandinh
phucvandinh
Chromaniac
  • Question Question
ES 2.2 Disappearing Elastic Index on a random basis
Replies
11
Views
2K
Chromaniac
Chromaniac
Fred.
  • Question Question
New to ES - A few questions about optimising ES
Replies
11
Views
4K
Deebs
Deebs
Top Bottom