Elasticsearch memory usage

[eggplant_casserole]

I'm an admin on a Xenforo forum. We're preparing to move to a new server and are considering purchasing Enhanced Search.

The server specs are as follows:

• Intel Xeon E3-1231 v3 @ 3.40 GHz
• 8 Gb RAM
• 256 Gb SSD
• CentOS 7
• Plesk Onyx 17

Everything is up-to-date. Other than that, I've installed Java and Elasticsearch, set up Gzip compression (well, mod_deflate) and that's it.

I hadn't checked the usage stats before, but I looked after installing the above and they'd gone to 2.7 Gb used. Seemed excessive for a server running nothing! So I checked and Elasticsearch was the guilty party.

It's been a couple of days and I've been busy. So I check again. Nothing has changed, but now Plesk reports:

Total 7.6 Gb Used 6.25 Gb Free 1.35 Gb Shared 0 B Buffer 254.94 Mb Cached 2.57 Gb Usage 48.42%

There's nothing on the server other than a test HTML file I downloaded to check the compression was working.

I checked with vmstat -s and the result was a little different:

7969744 K total memory
2987892 K used memory
3972704 K active memory
1855860 K inactive memory
1396312 K free memory
261056 K buffer memory
3324484 K swap cache
8388604 K total swap
0 K used swap
8388604 K free swap

Is that only counting physical RAM used?

So I tried top. Elasticsearch would come and go every 5 seconds or so, but it reported 29.9% RAM usage. So it still seems to be using 2 Gb or more doing nothing!

I've not used Elasticsearch before - I'm only doing so because users are complaining about poor search results and not being able to search for 3 letter words (there are many used in our topic area).

There's not much info on the buy page regarding requirements - it says you need Java and Elasticsearch and it's probably only suited for a dedicated server. We are running a dedicated server. However, most stuff I've seen online about configuring Elasticsearch talks about using server clusters, or at least a dedicated server JUST for Elasticsearch! We can't really afford to do that.

So....

Is the RAM usage going to jump drastically when we have copied a Xenforo forum across with over 850,000 messages and nearly 25,000 users? Is the server going to be adequate for our needs? What can we do to stop Elasticsearch hogging RAM... and is it a good idea?

 

[Paul B]

If your only requirement is to allow searching for 3 letter words, then you can do that with standard MySQL search.
You just need to set the value in the ACP and in my.cnf.

Having said that, the general rule of thumb is once you hit 1 million posts, you need to look at alternative search solutions, of which ES is one, so you may need to move away from MySQL search sooner or later anyway.

 

[eggplant_casserole]

If your only requirement is to allow searching for 3 letter words, then you can do that with standard MySQL search.
You just need to set the value in the ACP and in my.cnf.

That's interesting to know, although we've had complaints about poor quality search results in general. I suspect a lot of that is people using short keywords, but that is sometimes unavoidable (Wii U for example). As Enhanced Search is advertised as allowing "higher quality results" and speed for larger forums, it sounds like it would make sense to use it now and not have to worry when that time comes.

Oh, I forgot to ask... whilst maybe not ideal, would it be ok to share Elasticsearch with Mediawiki on the same server? That was one attraction, if it's not going to use too many resources. The wiki shouldn't be massive, really... no Wikipedia, for sure! I'm not a huge fan of having Java enabled unless necessary... but this sounds like a nice solution to both the current issue and the forthcoming one.

 

[AndyB]

So it still seems to be using 2 Gb or more doing nothing!

I just checked my server and the same amount of memory is being used, although I have 1.5 million posts indexed. So it seams this is normal.

 

[AndyB]

we've had complaints about poor quality search results in general.

The Enhanced Search will greatly improve search results when you select stemming.

https://xenforo.com/community/threads/how-to-elasticsearch-stemming-analyzer-setup.63835/

 

[eggplant_casserole]

I just checked my server and the same amount of memory is being used, although I have 1.5 million posts indexed. So it seams this is normal.

Ah, that's reassuring! If it's going to sit at that and not really go any higher, we can live with that. Thanks for checking!

The Enhanced Search will greatly improve search results when you select stemming.

Great tip - I'll certainly give that a go when we get it! Thanks again!

 

[Alpha1]

There are a couple of great Enhanced Search addons by @Xon which will give your users autocomplete suggestions.
Mind that stemming only works for one language.

 

[eggplant_casserole]

There are a couple of great Enhanced Search addons by @Xon which will give your users autocomplete suggestions.
Mind that stemming only works for one language.

Thanks for pointing that out. We request users write in English, so the language isn't a problem.

I was looking at Similar Threads, but ElasticSearch Essentials seems as though it may be more elegant (albeit more expensive). Thanks for the suggestion!

 

[Alpha1]

Mind there is a similar threads plus which uses elastic search.