Elasticsearch v5.0 is out there.
Anyone installed?
https://www.elastic.co/guide/en/elasticsearch/reference/5.0/setup-upgrade.html
Anyone installed?
https://www.elastic.co/guide/en/elasticsearch/reference/5.0/setup-upgrade.html
Elasticsearch 5
- Thread starter Sunka
- Start date
If it doesn't give you an error saying is it working on that screen then it's working. It would say something along the lines as are you sure ES is running.
Yer... but you don't need the latest... their stable version works effectively. V2 basically went to V5 for stable versions.
Elasticsearch 5.2.0 is out there
Release notes
Release notes
Code:
[root@upcloud ~]# yum history info 107
Loaded plugins: fastestmirror, priorities
Transaction ID : 107
Begin time : Tue Jan 31 17:53:51 2017
Begin rpmdb : 780:c4ecfaf4409ab378a30cffcbe386d12fb8d90763
End time : 17:53:52 2017 (1 seconds)
End rpmdb : 780:67c68ed802b58da59e5083429635ba9141488cf1
User : root <root>
Return-Code : Success
Command Line : update
Transaction performed with:
Installed rpm-4.11.3-21.el7.x86_64 @base
Installed yum-3.4.3-150.el7.centos.noarch @base
Installed yum-metadata-parser-1.1.4-10.el7.x86_64 @anaconda
Installed yum-plugin-fastestmirror-1.1.31-40.el7.noarch @base
Packages Altered:
Updated elasticsearch-5.1.2-1.noarch @elasticsearch-5.x
Update 5.2.0-1.noarch @elasticsearch-5.x
history info
[root@upcloud ~]# curl -XGET '127.0.0.1:9200'
{
"name" : "-QxWHO2",
"cluster_name" : "xxxxx",
"cluster_uuid" : "xxxxxxx",
"version" : {
"number" : "5.2.0",
"build_hash" : "24e05b9",
"build_date" : "2017-01-24T19:52:35.800Z",
"build_snapshot" : false,
"lucene_version" : "6.4.0"
},
"tagline" : "You Know, for Search"
}Nice... it upgraded smoothly and hopefully is more stable, reading those changes.
I honestly don't see any major improvement for the upgrade. I've tested the hell out of this over the past months, using 2.4x and 5.x and I just don't see any significant savings anywhere.
What pulled my curiosity the most, was why I need 4 - 8gb of ram to run a couple million search entries when I did the same thing with 1gb using Amazon.... so I've been exploring, and using 5.x I've got it running on 2gb ram, stable, using NGINX as the connector and proxying a distributed load to ES locally.
I now have a super stable ES instance that doesn't crash or such, uses next to no resources other than RAM... and with 150 live online at any given second with around 50% of that pulling search queries with each page load... SUPER STABLE with super little resources.
Whilst I am going to increase my ES instance in the future to 2 cores and 4gb RAM just for good measure and future proofing... I wanted to understand and acquire stable ES with the least resources to understand how to run an optimised ES instance.
I still believe I can improve it yet... and will keep playing with it over the coming month or two as I learn more about ES itself, and NGINX proxying.... writing to it when rebuilding the index is the greatest issue, though the logs seem to show that is where the minimal use of resources cause issues, yet I'm trying to work through that too and find the optimal rebuild solution.
I found 2.4.x easier though to work with for overall stability than 5.x has been.
What pulled my curiosity the most, was why I need 4 - 8gb of ram to run a couple million search entries when I did the same thing with 1gb using Amazon.... so I've been exploring, and using 5.x I've got it running on 2gb ram, stable, using NGINX as the connector and proxying a distributed load to ES locally.
I now have a super stable ES instance that doesn't crash or such, uses next to no resources other than RAM... and with 150 live online at any given second with around 50% of that pulling search queries with each page load... SUPER STABLE with super little resources.
Whilst I am going to increase my ES instance in the future to 2 cores and 4gb RAM just for good measure and future proofing... I wanted to understand and acquire stable ES with the least resources to understand how to run an optimised ES instance.
I still believe I can improve it yet... and will keep playing with it over the coming month or two as I learn more about ES itself, and NGINX proxying.... writing to it when rebuilding the index is the greatest issue, though the logs seem to show that is where the minimal use of resources cause issues, yet I'm trying to work through that too and find the optimal rebuild solution.
I found 2.4.x easier though to work with for overall stability than 5.x has been.
I heard your ES is huge Xon. But how much RAM does the actual ES instance use in total? Including Java?
Just fixed my nginx rebuild issue.... multi_accept on; increase workers, issue fixed, whilst using ES as an upstream server so everything is then proxied with keep alive. Slowly getting there.
Just fixed my nginx rebuild issue.... multi_accept on; increase workers, issue fixed, whilst using ES as an upstream server so everything is then proxied with keep alive. Slowly getting there.
Top reports the following (on each node):
USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
elastics 20 0 14.8g 1.9g 958m S 1.0 50.2 128:28.43 java
elastics 20 0 14.7g 1.9g 943m S 1.7 50.1 111:00.15 java
elastics 20 0 14.7g 1.9g 942m S 11.9 49.8 126:02.23 java
So 1.9gb of physical memory, of which 958mb of that is "sharable" or can be discard if faced with memory pressure; thus 1gb ram usage for data. JVM options limit the heap to 640m, so there is a hilarious amount of overhead somewhere.
USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
elastics 20 0 14.8g 1.9g 958m S 1.0 50.2 128:28.43 java
elastics 20 0 14.7g 1.9g 943m S 1.7 50.1 111:00.15 java
elastics 20 0 14.7g 1.9g 942m S 11.9 49.8 126:02.23 java
So 1.9gb of physical memory, of which 958mb of that is "sharable" or can be discard if faced with memory pressure; thus 1gb ram usage for data. JVM options limit the heap to 640m, so there is a hilarious amount of overhead somewhere.
Definitely interesting how proxying via nginx helps, I'll need to look into that!
So 6gb ram for 30 million? Sounds pretty good. With just 2gb and the things I'm doing with it right now.... sounds promising for myself for the future to not worry too much about it. 
I only find issue when building the index, which I have just gotten around for the most part now with the above fix.... but usage... it barely blips with 150 live online per second.
I think I'm really going to have to stress test this sucker, just to see what it can do.
I need to learn more about the nodes, shards, etc... within the ES, to get optimal results going forward. Right now its small enough that I don't have to worry about it, but I figure there are better optimisations using them correctly even for small instances of a couple million results.
I only find issue when building the index, which I have just gotten around for the most part now with the above fix.... but usage... it barely blips with 150 live online per second.
I think I'm really going to have to stress test this sucker, just to see what it can do.
I need to learn more about the nodes, shards, etc... within the ES, to get optimal results going forward. Right now its small enough that I don't have to worry about it, but I figure there are better optimisations using them correctly even for small instances of a couple million results.
Elasticsearch 5.3.1 is out there
Release notes
Release notes
Code:
[root@upcloud ~]# yum update
Loaded plugins: fastestmirror, priorities
.....
=====================================================================================================
Package Arch Version Repository Size
=====================================================================================================
Updating:
bind-libs x86_64 32:9.9.4-38.el7_3.3 updates 1.0 M
bind-libs-lite x86_64 32:9.9.4-38.el7_3.3 updates 730 k
bind-license noarch 32:9.9.4-38.el7_3.3 updates 83 k
bind-utils x86_64 32:9.9.4-38.el7_3.3 updates 202 k
device-mapper x86_64 7:1.02.135-1.el7_3.4 updates 269 k
device-mapper-libs x86_64 7:1.02.135-1.el7_3.4 updates 333 k
elasticsearch noarch 5.3.1-1 elasticsearch-5.x 32 M
Transaction Summary
=====================================================================================================
Upgrade 7 Packages
Total download size: 35 M
Is this ok [y/d/N]: y
Downloading packages:
No Presto metadata available for elasticsearch-5.x
updates/7/x86_64/prestodelta | 615 kB 00:00:00
Delta RPMs reduced 2.0 M of updates to 478 k (77% saved)
(1/7): bind-libs-9.9.4-38.el7_3.2_9.9.4-38.el7_3.3.x86_64.drpm | 172 kB 00:00:00
(2/7): bind-libs-lite-9.9.4-38.el7_3.2_9.9.4-38.el7_3.3.x86_64.drpm | 148 kB 00:00:00
(3/7): device-mapper-libs-1.02.135-1.el7_3.3_1.02.135-1.el7_3.4.x86_64.drpm | 159 kB 00:00:00
(4/7): bind-license-9.9.4-38.el7_3.3.noarch.rpm | 83 kB 00:00:00
(5/7): bind-utils-9.9.4-38.el7_3.3.x86_64.rpm | 202 kB 00:00:00
(6/7): device-mapper-1.02.135-1.el7_3.4.x86_64.rpm | 269 kB 00:00:00
(7/7): elasticsearch-5.3.1.rpm | 32 MB 00:00:02
Finishing delta rebuilds of 1 package(s) (333 k)
-----------------------------------------------------------------------------------------------------
Total 12 MB/s | 33 MB 00:00:02
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : 32:bind-license-9.9.4-38.el7_3.3.noarch 1/14
Updating : 32:bind-libs-9.9.4-38.el7_3.3.x86_64 2/14
Updating : 7:device-mapper-1.02.135-1.el7_3.4.x86_64 3/14
Updating : 7:device-mapper-libs-1.02.135-1.el7_3.4.x86_64 4/14
Updating : 32:bind-utils-9.9.4-38.el7_3.3.x86_64 5/14
Updating : 32:bind-libs-lite-9.9.4-38.el7_3.3.x86_64 6/14
Updating : elasticsearch-5.3.1-1.noarch 7/14
Cleanup : elasticsearch-5.3.0-1.noarch 8/14
Cleanup : 7:device-mapper-1.02.135-1.el7_3.3.x86_64 9/14
Cleanup : 7:device-mapper-libs-1.02.135-1.el7_3.3.x86_64 10/14
Cleanup : 32:bind-utils-9.9.4-38.el7_3.2.x86_64 11/14
Cleanup : 32:bind-libs-9.9.4-38.el7_3.2.x86_64 12/14
Cleanup : 32:bind-libs-lite-9.9.4-38.el7_3.2.x86_64 13/14
Cleanup : 32:bind-license-9.9.4-38.el7_3.2.noarch 14/14
Verifying : elasticsearch-5.3.1-1.noarch 1/14
Verifying : 32:bind-libs-lite-9.9.4-38.el7_3.3.x86_64 2/14
Verifying : 32:bind-utils-9.9.4-38.el7_3.3.x86_64 3/14
Verifying : 32:bind-libs-9.9.4-38.el7_3.3.x86_64 4/14
Verifying : 7:device-mapper-libs-1.02.135-1.el7_3.4.x86_64 5/14
Verifying : 32:bind-license-9.9.4-38.el7_3.3.noarch 6/14
Verifying : 7:device-mapper-1.02.135-1.el7_3.4.x86_64 7/14
Verifying : elasticsearch-5.3.0-1.noarch 8/14
Verifying : 32:bind-libs-lite-9.9.4-38.el7_3.2.x86_64 9/14
Verifying : 32:bind-utils-9.9.4-38.el7_3.2.x86_64 10/14
Verifying : 7:device-mapper-1.02.135-1.el7_3.3.x86_64 11/14
Verifying : 32:bind-libs-9.9.4-38.el7_3.2.x86_64 12/14
Verifying : 32:bind-license-9.9.4-38.el7_3.2.noarch 13/14
Verifying : 7:device-mapper-libs-1.02.135-1.el7_3.3.x86_64 14/14
Updated:
bind-libs.x86_64 32:9.9.4-38.el7_3.3 bind-libs-lite.x86_64 32:9.9.4-38.el7_3.3
bind-license.noarch 32:9.9.4-38.el7_3.3 bind-utils.x86_64 32:9.9.4-38.el7_3.3
device-mapper.x86_64 7:1.02.135-1.el7_3.4 device-mapper-libs.x86_64 7:1.02.135-1.el7_3.4
elasticsearch.noarch 0:5.3.1-1
Complete!
Code:
# curl -XGET '127.0.0.1:9200'
{
"name" : "-QxWHO2",
"cluster_name" : "xxx",
"cluster_uuid" : "ZLDl3yghRIa6MBvRGLBPzg",
"version" : {
"number" : "5.3.1",
"build_hash" : "5f9cf58",
"build_date" : "2017-04-17T15:52:53.846Z",
"build_snapshot" : false,
"lucene_version" : "6.4.2"
},
"tagline" : "You Know, for Search"
}Elasticsearch 5.3.2 is out there
Release notes
Release notes
Code:
# yum update
Loaded plugins: fastestmirror, priorities
base | 3.6 kB 00:00:00
centos-sclo-rh | 2.9 kB 00:00:00
centos-sclo-sclo | 2.9 kB 00:00:00
elasticsearch-5.x | 1.3 kB 00:00:00
epel/x86_64/metalink | 25 kB 00:00:00
epel | 4.3 kB 00:00:00
extras | 3.4 kB 00:00:00
mariadb | 2.9 kB 00:00:00
updates | 3.4 kB 00:00:00
varnish-4.1 | 951 B 00:00:00
(1/11): base/7/x86_64/group_gz | 155 kB 00:00:00
(2/11): centos-sclo-sclo/x86_64/primary_db | 149 kB 00:00:00
(3/11): epel/x86_64/group_gz | 170 kB 00:00:00
(4/11): epel/x86_64/updateinfo | 787 kB 00:00:00
(5/11): extras/7/x86_64/primary_db | 151 kB 00:00:00
(6/11): base/7/x86_64/primary_db | 5.6 MB 00:00:00
(7/11): epel/x86_64/primary_db | 4.7 MB 00:00:00
(8/11): centos-sclo-rh/x86_64/primary_db | 2.2 MB 00:00:00
(9/11): updates/7/x86_64/primary_db | 4.8 MB 00:00:00
(10/11): mariadb/primary_db | 18 kB 00:00:00
(11/11): elasticsearch-5.x/primary | 50 kB 00:00:00
varnish-4.1/x86_64/primary | 10 kB 00:00:00
Determining fastest mirrors
* base: ftp.hosteurope.de
* epel: mirror.de.leaseweb.net
* extras: ftp.hosteurope.de
* updates: ftp.hosteurope.de
elasticsearch-5.x 122/122
varnish-4.1 35/35
281 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package elasticsearch.noarch 0:5.3.1-1 will be updated
---> Package elasticsearch.noarch 0:5.3.2-1 will be an update
--> Finished Dependency Resolution
Dependencies Resolved
=====================================================================================================
Package Arch Version Repository Size
=====================================================================================================
Updating:
elasticsearch noarch 5.3.2-1 elasticsearch-5.x 32 M
Transaction Summary
=====================================================================================================
Upgrade 1 Package
Total download size: 32 M
Is this ok [y/d/N]: y
Downloading packages:
No Presto metadata available for elasticsearch-5.x
elasticsearch-5.3.2.rpm | 32 MB 00:00:02
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : elasticsearch-5.3.2-1.noarch 1/2
Cleanup : elasticsearch-5.3.1-1.noarch 2/2
Verifying : elasticsearch-5.3.2-1.noarch 1/2
Verifying : elasticsearch-5.3.1-1.noarch 2/2
Updated:
elasticsearch.noarch 0:5.3.2-1
Complete!
Code:
# curl -XGET '127.0.0.1:9200'
{
"name" : "-QxWHO2",
"cluster_name" : "xxx",
"cluster_uuid" : "ZLDl3yghRIa6MBvRGLBPzg",
"version" : {
"number" : "5.3.2",
"build_hash" : "3068195",
"build_date" : "2017-04-24T16:15:59.481Z",
"build_snapshot" : false,
"lucene_version" : "6.4.2"
},
"tagline" : "You Know, for Search"
}
Elastic Search 5.3.x will spam the logfile <cluster>_deprecation.log with "Content type detection for rest requests is deprecated. Specify the content type using the [Content-Type] header." everytime you do a bulk re-index.
Bug report; https://xenforo.com/community/threa...ction-for-rest-requests-is-deprecated.128765/
Bug report; https://xenforo.com/community/threa...ction-for-rest-requests-is-deprecated.128765/
Elasticsearch 5.4.0 is out there
Release notes
Release notes
Code:
# yum history info 174
Loaded plugins: fastestmirror, priorities
Transaction ID : 174
Begin time : Fri May 5 04:40:48 2017
Begin rpmdb : 803:6987088e17f94f0ab10d03c67e52204e2d645b7f
End time : 04:40:49 2017 (1 seconds)
End rpmdb : 803:b0c8a5964772052e6760d0d2da3a9d356807f516
User : root <root>
Return-Code : Success
Command Line : update
Transaction performed with:
Installed rpm-4.11.3-21.el7.x86_64 @base
Installed yum-3.4.3-150.el7.centos.noarch @base
Installed yum-metadata-parser-1.1.4-10.el7.x86_64 @anaconda
Installed yum-plugin-fastestmirror-1.1.31-40.el7.noarch @base
Packages Altered:
Updated elasticsearch-5.3.2-1.noarch @elasticsearch-5.x
Update 5.4.0-1.noarch @elasticsearch-5.x
history info
Code:
# curl -XGET '127.0.0.1:9200'
{
"name" : "-QxWHO2",
"cluster_name" : "xxx",
"cluster_uuid" : "ZLDl3yghRIa6MBvRGLBPzg",
"version" : {
"number" : "5.4.0",
"build_hash" : "780f8c4",
"build_date" : "2017-04-28T17:43:27.229Z",
"build_snapshot" : false,
"lucene_version" : "6.5.0"
},
"tagline" : "You Know, for Search"
}
not that easy, had to still edit config files, as some of the syntax depreciated..
Elasticsearch 5.4.2 is out there
Release notes
Release notes
Code:
[root@upcloud ~]# curl -XGET '127.0.0.1:9200'
{
"name" : "-QxWHO2",
"cluster_name" : "elastictvor",
"cluster_uuid" : "xxxxxx",
"version" : {
"number" : "5.4.2",
"build_hash" : "929b078",
"build_date" : "2017-06-15T02:29:28.122Z",
"build_snapshot" : false,
"lucene_version" : "6.5.1"
},
"tagline" : "You Know, for Search"
}Similar threads
- Question
