• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

Elasticsearch 5

Sunka

Well-known member
#44
Elasticsearch 5.2.0 is out there
Release notes

Code:
[root@upcloud ~]# yum history info 107
Loaded plugins: fastestmirror, priorities
Transaction ID : 107
Begin time     : Tue Jan 31 17:53:51 2017
Begin rpmdb    : 780:c4ecfaf4409ab378a30cffcbe386d12fb8d90763
End time       :            17:53:52 2017 (1 seconds)
End rpmdb      : 780:67c68ed802b58da59e5083429635ba9141488cf1
User           : root <root>
Return-Code    : Success
Command Line   : update
Transaction performed with:
    Installed     rpm-4.11.3-21.el7.x86_64                      @base
    Installed     yum-3.4.3-150.el7.centos.noarch               @base
    Installed     yum-metadata-parser-1.1.4-10.el7.x86_64       @anaconda
    Installed     yum-plugin-fastestmirror-1.1.31-40.el7.noarch @base
Packages Altered:
    Updated elasticsearch-5.1.2-1.noarch @elasticsearch-5.x
    Update                5.2.0-1.noarch @elasticsearch-5.x
history info


[root@upcloud ~]# curl -XGET '127.0.0.1:9200'
{
  "name" : "-QxWHO2",
  "cluster_name" : "xxxxx",
  "cluster_uuid" : "xxxxxxx",
  "version" : {
    "number" : "5.2.0",
    "build_hash" : "24e05b9",
    "build_date" : "2017-01-24T19:52:35.800Z",
    "build_snapshot" : false,
    "lucene_version" : "6.4.0"
  },
  "tagline" : "You Know, for Search"
}
 

Anthony Parsons

Well-known member
#47
I honestly don't see any major improvement for the upgrade. I've tested the hell out of this over the past months, using 2.4x and 5.x and I just don't see any significant savings anywhere.

What pulled my curiosity the most, was why I need 4 - 8gb of ram to run a couple million search entries when I did the same thing with 1gb using Amazon.... so I've been exploring, and using 5.x I've got it running on 2gb ram, stable, using NGINX as the connector and proxying a distributed load to ES locally.

I now have a super stable ES instance that doesn't crash or such, uses next to no resources other than RAM... and with 150 live online at any given second with around 50% of that pulling search queries with each page load... SUPER STABLE with super little resources.

Whilst I am going to increase my ES instance in the future to 2 cores and 4gb RAM just for good measure and future proofing... I wanted to understand and acquire stable ES with the least resources to understand how to run an optimised ES instance.

I still believe I can improve it yet... and will keep playing with it over the coming month or two as I learn more about ES itself, and NGINX proxying.... writing to it when rebuilding the index is the greatest issue, though the logs seem to show that is where the minimal use of resources cause issues, yet I'm trying to work through that too and find the optimal rebuild solution.

I found 2.4.x easier though to work with for overall stability than 5.x has been.
 

Xon

Well-known member
#48
5.x has been rock stable for me (once you get over all the breaking changes...), but I run mine as a 3 node cluster with 30 million search entries and only ~3gb ram across the cluster allocated to ES.
 

Anthony Parsons

Well-known member
#49
I heard your ES is huge Xon. But how much RAM does the actual ES instance use in total? Including Java?

Just fixed my nginx rebuild issue.... multi_accept on; increase workers, issue fixed, whilst using ES as an upstream server so everything is then proxied with keep alive. Slowly getting there.
 

Xon

Well-known member
#50
Top reports the following (on each node):

USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
elastics 20 0 14.8g 1.9g 958m S 1.0 50.2 128:28.43 java
elastics 20 0 14.7g 1.9g 943m S 1.7 50.1 111:00.15 java
elastics 20 0 14.7g 1.9g 942m S 11.9 49.8 126:02.23 java

So 1.9gb of physical memory, of which 958mb of that is "sharable" or can be discard if faced with memory pressure; thus 1gb ram usage for data. JVM options limit the heap to 640m, so there is a hilarious amount of overhead somewhere.

Just fixed my nginx rebuild issue.... multi_accept on; increase workers, issue fixed, whilst using ES as an upstream server so everything is then proxied with keep alive. Slowly getting there.
Definitely interesting how proxying via nginx helps, I'll need to look into that!
 

Anthony Parsons

Well-known member
#51
So 6gb ram for 30 million? Sounds pretty good. With just 2gb and the things I'm doing with it right now.... sounds promising for myself for the future to not worry too much about it. :)

I only find issue when building the index, which I have just gotten around for the most part now with the above fix.... but usage... it barely blips with 150 live online per second.

I think I'm really going to have to stress test this sucker, just to see what it can do. :cautious::whistle::coffee:

I need to learn more about the nodes, shards, etc... within the ES, to get optimal results going forward. Right now its small enough that I don't have to worry about it, but I figure there are better optimisations using them correctly even for small instances of a couple million results.
 

Sunka

Well-known member
#53
Elasticsearch 5.3.1 is out there
Release notes

Code:
[root@upcloud ~]# yum update
Loaded plugins: fastestmirror, priorities
.....

=====================================================================================================
 Package                   Arch          Version                      Repository                Size
=====================================================================================================
Updating:
 bind-libs                 x86_64        32:9.9.4-38.el7_3.3          updates                  1.0 M
 bind-libs-lite            x86_64        32:9.9.4-38.el7_3.3          updates                  730 k
 bind-license              noarch        32:9.9.4-38.el7_3.3          updates                   83 k
 bind-utils                x86_64        32:9.9.4-38.el7_3.3          updates                  202 k
 device-mapper             x86_64        7:1.02.135-1.el7_3.4         updates                  269 k
 device-mapper-libs        x86_64        7:1.02.135-1.el7_3.4         updates                  333 k
 elasticsearch             noarch        5.3.1-1                      elasticsearch-5.x         32 M

Transaction Summary
=====================================================================================================
Upgrade  7 Packages

Total download size: 35 M
Is this ok [y/d/N]: y
Downloading packages:
No Presto metadata available for elasticsearch-5.x
updates/7/x86_64/prestodelta                                                  | 615 kB  00:00:00    
Delta RPMs reduced 2.0 M of updates to 478 k (77% saved)
(1/7): bind-libs-9.9.4-38.el7_3.2_9.9.4-38.el7_3.3.x86_64.drpm                | 172 kB  00:00:00    
(2/7): bind-libs-lite-9.9.4-38.el7_3.2_9.9.4-38.el7_3.3.x86_64.drpm           | 148 kB  00:00:00    
(3/7): device-mapper-libs-1.02.135-1.el7_3.3_1.02.135-1.el7_3.4.x86_64.drpm   | 159 kB  00:00:00    
(4/7): bind-license-9.9.4-38.el7_3.3.noarch.rpm                               |  83 kB  00:00:00    
(5/7): bind-utils-9.9.4-38.el7_3.3.x86_64.rpm                                 | 202 kB  00:00:00    
(6/7): device-mapper-1.02.135-1.el7_3.4.x86_64.rpm                            | 269 kB  00:00:00    
(7/7): elasticsearch-5.3.1.rpm                                                |  32 MB  00:00:02    
Finishing delta rebuilds of 1 package(s) (333 k)
-----------------------------------------------------------------------------------------------------
Total                                                                 12 MB/s |  33 MB  00:00:02    
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : 32:bind-license-9.9.4-38.el7_3.3.noarch                                          1/14 
  Updating   : 32:bind-libs-9.9.4-38.el7_3.3.x86_64                                             2/14 
  Updating   : 7:device-mapper-1.02.135-1.el7_3.4.x86_64                                        3/14 
  Updating   : 7:device-mapper-libs-1.02.135-1.el7_3.4.x86_64                                   4/14 
  Updating   : 32:bind-utils-9.9.4-38.el7_3.3.x86_64                                            5/14 
  Updating   : 32:bind-libs-lite-9.9.4-38.el7_3.3.x86_64                                        6/14 
  Updating   : elasticsearch-5.3.1-1.noarch                                                     7/14 
  Cleanup    : elasticsearch-5.3.0-1.noarch                                                     8/14 
  Cleanup    : 7:device-mapper-1.02.135-1.el7_3.3.x86_64                                        9/14 
  Cleanup    : 7:device-mapper-libs-1.02.135-1.el7_3.3.x86_64                                  10/14 
  Cleanup    : 32:bind-utils-9.9.4-38.el7_3.2.x86_64                                           11/14 
  Cleanup    : 32:bind-libs-9.9.4-38.el7_3.2.x86_64                                            12/14 
  Cleanup    : 32:bind-libs-lite-9.9.4-38.el7_3.2.x86_64                                       13/14 
  Cleanup    : 32:bind-license-9.9.4-38.el7_3.2.noarch                                         14/14 
  Verifying  : elasticsearch-5.3.1-1.noarch                                                     1/14 
  Verifying  : 32:bind-libs-lite-9.9.4-38.el7_3.3.x86_64                                        2/14 
  Verifying  : 32:bind-utils-9.9.4-38.el7_3.3.x86_64                                            3/14 
  Verifying  : 32:bind-libs-9.9.4-38.el7_3.3.x86_64                                             4/14 
  Verifying  : 7:device-mapper-libs-1.02.135-1.el7_3.4.x86_64                                   5/14 
  Verifying  : 32:bind-license-9.9.4-38.el7_3.3.noarch                                          6/14 
  Verifying  : 7:device-mapper-1.02.135-1.el7_3.4.x86_64                                        7/14 
  Verifying  : elasticsearch-5.3.0-1.noarch                                                     8/14 
  Verifying  : 32:bind-libs-lite-9.9.4-38.el7_3.2.x86_64                                        9/14 
  Verifying  : 32:bind-utils-9.9.4-38.el7_3.2.x86_64                                           10/14 
  Verifying  : 7:device-mapper-1.02.135-1.el7_3.3.x86_64                                       11/14 
  Verifying  : 32:bind-libs-9.9.4-38.el7_3.2.x86_64                                            12/14 
  Verifying  : 32:bind-license-9.9.4-38.el7_3.2.noarch                                         13/14 
  Verifying  : 7:device-mapper-libs-1.02.135-1.el7_3.3.x86_64                                  14/14 

Updated:
  bind-libs.x86_64 32:9.9.4-38.el7_3.3           bind-libs-lite.x86_64 32:9.9.4-38.el7_3.3          
  bind-license.noarch 32:9.9.4-38.el7_3.3        bind-utils.x86_64 32:9.9.4-38.el7_3.3              
  device-mapper.x86_64 7:1.02.135-1.el7_3.4      device-mapper-libs.x86_64 7:1.02.135-1.el7_3.4    
  elasticsearch.noarch 0:5.3.1-1                

Complete!
Code:
# curl -XGET '127.0.0.1:9200'
{
  "name" : "-QxWHO2",
  "cluster_name" : "xxx",
  "cluster_uuid" : "ZLDl3yghRIa6MBvRGLBPzg",
  "version" : {
    "number" : "5.3.1",
    "build_hash" : "5f9cf58",
    "build_date" : "2017-04-17T15:52:53.846Z",
    "build_snapshot" : false,
    "lucene_version" : "6.4.2"
  },
  "tagline" : "You Know, for Search"
}
18-56-01.png
 

Sunka

Well-known member
#54
Elasticsearch 5.3.2 is out there
Release notes

Code:
# yum update
Loaded plugins: fastestmirror, priorities
base                                                                          | 3.6 kB  00:00:00    
centos-sclo-rh                                                                | 2.9 kB  00:00:00    
centos-sclo-sclo                                                              | 2.9 kB  00:00:00    
elasticsearch-5.x                                                             | 1.3 kB  00:00:00    
epel/x86_64/metalink                                                          |  25 kB  00:00:00    
epel                                                                          | 4.3 kB  00:00:00    
extras                                                                        | 3.4 kB  00:00:00    
mariadb                                                                       | 2.9 kB  00:00:00    
updates                                                                       | 3.4 kB  00:00:00    
varnish-4.1                                                                   |  951 B  00:00:00    
(1/11): base/7/x86_64/group_gz                                                | 155 kB  00:00:00    
(2/11): centos-sclo-sclo/x86_64/primary_db                                    | 149 kB  00:00:00    
(3/11): epel/x86_64/group_gz                                                  | 170 kB  00:00:00    
(4/11): epel/x86_64/updateinfo                                                | 787 kB  00:00:00    
(5/11): extras/7/x86_64/primary_db                                            | 151 kB  00:00:00    
(6/11): base/7/x86_64/primary_db                                              | 5.6 MB  00:00:00    
(7/11): epel/x86_64/primary_db                                                | 4.7 MB  00:00:00    
(8/11): centos-sclo-rh/x86_64/primary_db                                      | 2.2 MB  00:00:00    
(9/11): updates/7/x86_64/primary_db                                           | 4.8 MB  00:00:00    
(10/11): mariadb/primary_db                                                   |  18 kB  00:00:00    
(11/11): elasticsearch-5.x/primary                                            |  50 kB  00:00:00    
varnish-4.1/x86_64/primary                                                    |  10 kB  00:00:00    
Determining fastest mirrors
 * base: ftp.hosteurope.de
 * epel: mirror.de.leaseweb.net
 * extras: ftp.hosteurope.de
 * updates: ftp.hosteurope.de
elasticsearch-5.x                                                                            122/122
varnish-4.1                                                                                    35/35
281 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package elasticsearch.noarch 0:5.3.1-1 will be updated
---> Package elasticsearch.noarch 0:5.3.2-1 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

=====================================================================================================
 Package                  Arch              Version               Repository                    Size
=====================================================================================================
Updating:
 elasticsearch            noarch            5.3.2-1               elasticsearch-5.x             32 M

Transaction Summary
=====================================================================================================
Upgrade  1 Package

Total download size: 32 M
Is this ok [y/d/N]: y
Downloading packages:
No Presto metadata available for elasticsearch-5.x
elasticsearch-5.3.2.rpm                                                       |  32 MB  00:00:02    
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : elasticsearch-5.3.2-1.noarch                                                      1/2 
  Cleanup    : elasticsearch-5.3.1-1.noarch                                                      2/2 
  Verifying  : elasticsearch-5.3.2-1.noarch                                                      1/2 
  Verifying  : elasticsearch-5.3.1-1.noarch                                                      2/2 

Updated:
  elasticsearch.noarch 0:5.3.2-1                                                                    

Complete!
Code:
# curl -XGET '127.0.0.1:9200'
{
  "name" : "-QxWHO2",
  "cluster_name" : "xxx",
  "cluster_uuid" : "ZLDl3yghRIa6MBvRGLBPzg",
  "version" : {
    "number" : "5.3.2",
    "build_hash" : "3068195",
    "build_date" : "2017-04-24T16:15:59.481Z",
    "build_snapshot" : false,
    "lucene_version" : "6.4.2"
  },
  "tagline" : "You Know, for Search"
}
 

Sunka

Well-known member
#58
Elasticsearch 5.4.0 is out there
Release notes

Code:
# yum history info 174
Loaded plugins: fastestmirror, priorities
Transaction ID : 174
Begin time     : Fri May  5 04:40:48 2017
Begin rpmdb    : 803:6987088e17f94f0ab10d03c67e52204e2d645b7f
End time       :            04:40:49 2017 (1 seconds)
End rpmdb      : 803:b0c8a5964772052e6760d0d2da3a9d356807f516
User           : root <root>
Return-Code    : Success
Command Line   : update
Transaction performed with:
    Installed     rpm-4.11.3-21.el7.x86_64                      @base
    Installed     yum-3.4.3-150.el7.centos.noarch               @base
    Installed     yum-metadata-parser-1.1.4-10.el7.x86_64       @anaconda
    Installed     yum-plugin-fastestmirror-1.1.31-40.el7.noarch @base
Packages Altered:
    Updated elasticsearch-5.3.2-1.noarch @elasticsearch-5.x
    Update                5.4.0-1.noarch @elasticsearch-5.x
history info
Code:
# curl -XGET '127.0.0.1:9200'
{
  "name" : "-QxWHO2",
  "cluster_name" : "xxx",
  "cluster_uuid" : "ZLDl3yghRIa6MBvRGLBPzg",
  "version" : {
    "number" : "5.4.0",
    "build_hash" : "780f8c4",
    "build_date" : "2017-04-28T17:43:27.229Z",
    "build_snapshot" : false,
    "lucene_version" : "6.5.0"
  },
  "tagline" : "You Know, for Search"
}
12-49-29.png
 
#59
I Upgraded elasticsearch 5 on centos 7 with this steps.

Code:
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
Open /etc/yum.repos.d/elasticsearch.repo

Add this:

Code:
[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
Then Run:

Code:
yum update
That will Upgraded elasticsearch 5 from elasticsearch 2.x and donot need change any setting for this Upgrading.
not that easy, had to still edit config files, as some of the syntax depreciated..
 

Sunka

Well-known member
#60
Elasticsearch 5.4.2 is out there
Release notes



Code:
[root@upcloud ~]# curl -XGET '127.0.0.1:9200'
{
  "name" : "-QxWHO2",
  "cluster_name" : "elastictvor",
  "cluster_uuid" : "xxxxxx",
  "version" : {
    "number" : "5.4.2",
    "build_hash" : "929b078",
    "build_date" : "2017-06-15T02:29:28.122Z",
    "build_snapshot" : false,
    "lucene_version" : "6.5.1"
  },
  "tagline" : "You Know, for Search"
}

15-32-04.png