New to ES - A few questions about optimising ES

Fred.

Well-known member
Hi,

I'm new to Elasticsearch. I installed it and have it running on my test forum now.
I've set the memory settings in /etc/elasticsearch/jvm.options to
Code:
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms256m
-Xmx256m
Because it will be used on a small forum with less than 100 000 posts. And that should be enough. (I can set it higher if I want because I have more ram)

While it's only active on my test forum with no active usage I still see many processes and ram usage.
Is this normal?

This is what I mean
Screen Shot 2017-07-27 at 12.47.23 pm.png
 

kontrabass

Well-known member
Looks fine to me, we use 2G heap size, the virtual memory requested is going to be about the same, the resident memory of our processes is 5363M. 20 million posts.
 

Fred.

Well-known member
Thanks, It seems to run fine. I was worried about the virtual memory but it seems ok.
 

Xon

Well-known member
@Fred the most important optimization is to use the latest version.

With just a single node, you want to disable the replica shards (ie having multiple copies of your data); For ElasticSearch 5v.x;
Code:
curl -XDELETE 'http://localhost:9200/_template/default'
curl -XPUT 'http://localhost:9200/_template/default' -d'{
"template": "*",
"settings": {
"number_of_replicas": "0"
}
}'
curl -XPUT 'http://localhost:9200/_settings' -d '{
"index" : { "number_of_replicas" : "0" }
}'
1st argument deletes any default template (since merging is a *****), 2nd builds a new one for any new indexes so they have zero replicas.

Last curl command changes any existing indexes.
 

Fred.

Well-known member
Thanks Xon (y)
I'm using the last version.
I will try that with the replica shards.
 

Sunka

Well-known member
@Fred the most important optimization is to use the latest version.

With just a single node, you want to disable the replica shards (ie having multiple copies of your data); For ElasticSearch 5v.x;
Code:
curl -XDELETE 'http://localhost:9200/_template/default'
curl -XPUT 'http://localhost:9200/_template/default' -d'{
"template": "*",
"settings": {
"number_of_replicas": "0"
}
}'
curl -XPUT 'http://localhost:9200/_settings' -d '{
"index" : { "number_of_replicas" : "0" }
}'
1st argument deletes any default template (since merging is a *****), 2nd builds a new one for any new indexes so they have zero replicas.

Last curl command changes any existing indexes.
@Xon
Just one question regarding your code.
We should do that in three steps? Right?
  1. execute curl -XDELETE 'http://localhost:9200/_template/default'
  2. then execute curl -XPUT 'http://localhost:9200/_template/default' -d'{"template": "*","settings": {"number_of_replicas": "0"}}'
  3. then execute curl -XPUT 'http://localhost:9200/_settings' -d '{"index" : { "number_of_replicas" : "0" }}'

If someone just copy/paste in cli all code or just one step it will do nothing right?
Like this:
Code:
curl -XPUT 'http://localhost:9200/_template/default' -d'{
"template": "*",
"settings": {
"number_of_replicas": "0"
}
}'
Maybe I am wrong, but in above case, should not that code be like this?
Code:
curl -XPUT 'http://localhost:9200/_template/default' -d'{ \
"template": "*", \
"settings": { \
"number_of_replicas": "0" \
} \
}'
or this
curl -XPUT 'http://localhost:9200/_template/default' -d'{"template": "*","settings": {"number_of_replicas": "0"}}'
 

Xon

Well-known member
You do need to run each command separately, but it shouldn't matter if it is 1 line or not, as the ' character will extend the data entry over multiple lines.

At least with a bash shell. Other shell may vary :p
 

Nuno

Well-known member
@Fred the most important optimization is to use the latest version.

With just a single node, you want to disable the replica shards (ie having multiple copies of your data); For ElasticSearch 5v.x;
Code:
curl -XDELETE 'http://localhost:9200/_template/default'
curl -XPUT 'http://localhost:9200/_template/default' -d'{
"template": "*",
"settings": {
"number_of_replicas": "0"
}
}'
curl -XPUT 'http://localhost:9200/_settings' -d '{
"index" : { "number_of_replicas" : "0" }
}'
1st argument deletes any default template (since merging is a *), 2nd builds a new one for any new indexes so they have zero replicas.

Last curl command changes any existing indexes.
Sorry about bumping this old thread, but is this available in ElasticSearch 6?
I tested this in a new/empty ES6 installation, eventually this only works after install Enhanced Search and create some indexes.

Code:
# curl -XDELETE 'http://localhost:9200/_template/default'

{"error":{"root_cause":[{"type":"index_template_missing_exception","reason":"index_template [default] missing"}],"type":"index_template_missing_exception","reason":"index_template [default] missing"},"status":404}

Code:
# curl -XPUT 'http://localhost:9200/_template/default' -d'{"template": "*","settings": {"number_of_replicas": "0"}}'

{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}

Code:
# curl -XPUT 'http://localhost:9200/_settings' -d '{"index" : { "number_of_replicas" : "0" }}'

{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}
Thanks
 
Last edited:

Deebs

Well-known member
OK .. reading the docs, for 6 we need to add

Code:
-H'Content-Type: application/json'

Now, for a single node it's advisable to set number_of_shards = 1 or should I left with 5?
Starting with version 7 they are setting the default number of shards to 1.

On a single node (and without wanting the ability to scale out) it is better to use a single shard as it means a single lucene instance and if you have multiple shards then the search has to occur across all shards then an internal process merges the results before returning them. A single shard will not have to do this step.

In prep for v7 I have rebuilt all my indexes in use to have 0 replicas and 1 shard within my single node system.
 

Deebs

Well-known member
Basically I ran the following:

Code:
curl -XPUT "localhost:9200/_template/all" -H 'Content-Type: application/json' -d'
{
  "template": "*",
  "settings": {
    "number_of_replicas": 0,
    "number_of_shards": 1,
  }
}
'
Then deleted the index. After that I went to ElasticSearch Setup, set my analyzer to English, then rebuilt the index without deleting.

198010
 
Top