New to ES - A few questions about optimising ES

[Fred.]

Hi,

I'm new to Elasticsearch. I installed it and have it running on my test forum now.
I've set the memory settings in /etc/elasticsearch/jvm.options to

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms256m
-Xmx256m

Because it will be used on a small forum with less than 100 000 posts. And that should be enough. (I can set it higher if I want because I have more ram)

While it's only active on my test forum with no active usage I still see many processes and ram usage.
Is this normal?

This is what I mean

 

[kontrabass]

Looks fine to me, we use 2G heap size, the virtual memory requested is going to be about the same, the resident memory of our processes is 5363M. 20 million posts.

 

[Fred.]

Thanks, It seems to run fine. I was worried about the virtual memory but it seems ok.

 

[Xon]

@Fred the most important optimization is to use the latest version.

With just a single node, you want to disable the replica shards (ie having multiple copies of your data); For ElasticSearch 5v.x;

curl -XDELETE 'http://localhost:9200/_template/default'
curl -XPUT 'http://localhost:9200/_template/default' -d'{
"template": "*",
"settings": {
"number_of_replicas": "0"
}
}'
curl -XPUT 'http://localhost:9200/_settings' -d '{
"index" : { "number_of_replicas" : "0" }
}'

1st argument deletes any default template (since merging is a *****), 2nd builds a new one for any new indexes so they have zero replicas.

Last curl command changes any existing indexes.

 

[Fred.]

Thanks Xon
I'm using the last version.
I will try that with the replica shards.

 

[Sunka]

@Fred the most important optimization is to use the latest version.

With just a single node, you want to disable the replica shards (ie having multiple copies of your data); For ElasticSearch 5v.x;

curl -XDELETE 'http://localhost:9200/_template/default'
curl -XPUT 'http://localhost:9200/_template/default' -d'{
"template": "*",
"settings": {
"number_of_replicas": "0"
}
}'
curl -XPUT 'http://localhost:9200/_settings' -d '{
"index" : { "number_of_replicas" : "0" }
}'

1st argument deletes any default template (since merging is a *****), 2nd builds a new one for any new indexes so they have zero replicas.

Last curl command changes any existing indexes.

@Xon
Just one question regarding your code.
We should do that in three steps? Right?

1. execute curl -XDELETE 'http://localhost:9200/_template/default'
2. then execute curl -XPUT 'http://localhost:9200/_template/default' -d'{"template": "*","settings": {"number_of_replicas": "0"}}'
3. then execute curl -XPUT 'http://localhost:9200/_settings' -d '{"index" : { "number_of_replicas" : "0" }}'

If someone just copy/paste in cli all code or just one step it will do nothing right?
Like this:

curl -XPUT 'http://localhost:9200/_template/default' -d'{
"template": "*",
"settings": {
"number_of_replicas": "0"
}
}'

Maybe I am wrong, but in above case, should not that code be like this?

curl -XPUT 'http://localhost:9200/_template/default' -d'{ \
"template": "*", \
"settings": { \
"number_of_replicas": "0" \
} \
}'

or this
curl -XPUT 'http://localhost:9200/_template/default' -d'{"template": "*","settings": {"number_of_replicas": "0"}}'

 

[Xon]

You do need to run each command separately, but it shouldn't matter if it is 1 line or not, as the ' character will extend the data entry over multiple lines.

At least with a bash shell. Other shell may vary

 

[Nuno]

@Fred the most important optimization is to use the latest version.

With just a single node, you want to disable the replica shards (ie having multiple copies of your data); For ElasticSearch 5v.x;

curl -XDELETE 'http://localhost:9200/_template/default'
curl -XPUT 'http://localhost:9200/_template/default' -d'{
"template": "*",
"settings": {
"number_of_replicas": "0"
}
}'
curl -XPUT 'http://localhost:9200/_settings' -d '{
"index" : { "number_of_replicas" : "0" }
}'

1st argument deletes any default template (since merging is a *), 2nd builds a new one for any new indexes so they have zero replicas.

Last curl command changes any existing indexes.

Sorry about bumping this old thread, but is this available in ElasticSearch 6?
I tested this in a new/empty ES6 installation, eventually this only works after install Enhanced Search and create some indexes.

# curl -XDELETE 'http://localhost:9200/_template/default'

{"error":{"root_cause":[{"type":"index_template_missing_exception","reason":"index_template [default] missing"}],"type":"index_template_missing_exception","reason":"index_template [default] missing"},"status":404}

# curl -XPUT 'http://localhost:9200/_template/default' -d'{"template": "*","settings": {"number_of_replicas": "0"}}'

{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}

# curl -XPUT 'http://localhost:9200/_settings' -d '{"index" : { "number_of_replicas" : "0" }}'

{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}

Thanks

 

[Nuno]

OK .. reading the docs, for 6 we need to add

-H'Content-Type: application/json'

Strict Content-Type Checking for Elasticsearch REST Requests

Elasticsearch 6.0 will enforce strict content-type checking. All REST requests that include a body must also provide the correct Content-Type header....

www.elastic.co

Now, for a single node it's advisable to set number_of_shards = 1 or should I left with 5?

 

[Deebs]

OK .. reading the docs, for 6 we need to add

-H'Content-Type: application/json'

Strict Content-Type Checking for Elasticsearch REST Requests

Elasticsearch 6.0 will enforce strict content-type checking. All REST requests that include a body must also provide the correct Content-Type header....

www.elastic.co

Now, for a single node it's advisable to set number_of_shards = 1 or should I left with 5?

Starting with version 7 they are setting the default number of shards to 1.

On a single node (and without wanting the ability to scale out) it is better to use a single shard as it means a single lucene instance and if you have multiple shards then the search has to occur across all shards then an internal process merges the results before returning them. A single shard will not have to do this step.

In prep for v7 I have rebuilt all my indexes in use to have 0 replicas and 1 shard within my single node system.

 

[Nuno]

In prep for v7 I have rebuilt all my indexes in use to have 0 replicas and 1 shard within my single node system

Can you share the steps you toke to make these changes?

Thanks

 

[Deebs]

Basically I ran the following:

curl -XPUT "localhost:9200/_template/all" -H 'Content-Type: application/json' -d'
{
  "template": "*",
  "settings": {
    "number_of_replicas": 0,
    "number_of_shards": 1,
  }
}
'

Then deleted the index. After that I went to ElasticSearch Setup, set my analyzer to English, then rebuilt the index without deleting.