Elastic search configuration

[FloV]

Hey everybody,

for a couple of month know, i'm running into memory issues with elasticsearch (0.20.5). Everytime we have a lot of traffic on the site (around 2.000 active users) elasticsearch is running out of memory and also kills the process of the apache.

Is anyone having the same issues? Maybe i can tune a little on the configuration? As soon as i deactive ES, the servers are running smoothly.

Thanks for your help.

Florian

 

[Paul B]

How big is your post table/search index and how much RAM do you have in your server?

 

[FloV]

The post table is 4,3 GiB and we have 32 GB of memory in the server

 

[Paul B]

That should be enough.

You mention that it only happens when there are a lot of visitors (2000+).
Do you have many add-ons installed?

 

[Mike]

1. Consider upgrading to the latest version of ES.
2. Ensure your indexes are optimized (use the latest version of the XFES add-on and optimize them via the admin CP).
3. If all else fails, you'll need to allocate more memory to ES.

 

[FloV]

We already upgraded to the latest ES Version (0.20.6.) and the index is optimized. There are also 2GB of memory allocated to ES.

That should be enough.

You mention that it only happens when there are a lot of visitors (2000+).
Do you have many add-ons installed?

Yeah, sadly we have. It seems that that's a problem too, i guess. But it's weird that the server runs smoothly, as soon as we deactivate the XFES add-on.

 

[Paul B]

I suspect you need to allocate more RAM to ES as Mike has said.

A rough rule of thumb seems to be around 1GB per 1 million posts.

 

[Mike]

We already upgraded to the latest ES Version (0.20.6.) and the index is optimized. There are also 2GB of memory allocated to ES.

I was actually referring to 0.90: http://www.elasticsearch.org/downloads/0-90-0-rc2/

 

[CTXMedia]

Are you on XF 1.1.4 and XFES add-on v1.0.1 too?

 

[FloV]

Right now i'm on XF 1.1.3 and XFES 1.0.1, but i will update XF this week

 

[CTXMedia]

Did you upgrade XFES from 1.0 to 1.0.1? The only reason I ask is that the newer version can optimise your search index and often reduces the size by up to 50% (on average) - so if you haven't yet optimised (in Tools > Elasticsearch Options) it might be worth doing.

Cheers,
Shaun

 

[FloV]

Yep, i've upgraded from 1.0 -> 1.0.1. I've already tried optimize the index. After it failed, i've deleted it and created a new one.

Our server host said, there are were lot of issues with java processes.

 

[kontrabass]

@Crazy-Achmet Did you find a solution? Did you try increasing the memory allocated to ES? I don't know your board but I'm guessing you have 10M+ posts, so 2GB doesn't seem like near enough.