1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Elasticsearch memory requirements for big boards

Discussion in 'Enhanced Search Support' started by kontrabass, Oct 2, 2013.

  1. kontrabass

    kontrabass Well-Known Member

    With vbulletin 3 + sphinx, we have sphinx running on the DB server, and it runs beautifully with 16 gigs of ram and SSD's. Server loads barely register . We have 13M posts, and from what I've read from @Slavik we'll need like 13 gigs of ram just for Elasticsearch? So we'd pretty much need to buy a new dedicated search server with this many posts? If so how robust should this server be?
     
  2. Dinh Thanh

    Dinh Thanh Well-Known Member

    30m post on 16GB RAM Server. Running very smooth.
     
    kontrabass likes this.
  3. Slavik

    Slavik XenForo Moderator Staff Member

    With the latest updates, i've been finding 512mb per mil has been working well.
     
    The Forum Heroes and kontrabass like this.
  4. kontrabass

    kontrabass Well-Known Member

    Wow. I also love how on your http://www.tinhte.vn site, you offer Google Custom Search results and XF search results on the same results page, with tabs. There are many odd searches for bass/amp models that Google just does better, and I'd love to offer a similar solution with the user not having to do two separate searches.
     
  5. The Forum Heroes

    The Forum Heroes Well-Known Member

    We have 6 clients using ES, about 18m posts between them and ES runs smooth with 10GB RAM dedicated to ES.
     
  6. Dinh Thanh

    Dinh Thanh Well-Known Member

    Google does better jobs, but some content types need XF Search (user, tag and advanced search)
     
  7. DeltaHF

    DeltaHF Well-Known Member

    What are the consequences of running ES on a server with less RAM than what is required to store the entire data set in memory?
     
  8. Slavik

    Slavik XenForo Moderator Staff Member

    Slower search returns.
     
  9. DeltaHF

    DeltaHF Well-Known Member

    How much slower are we talking about?
     
  10. Slavik

    Slavik XenForo Moderator Staff Member

    How long is a peice of string.
     
    CreationNation likes this.
  11. digitalpoint

    digitalpoint Well-Known Member

    For purposes of the original question, you can ignore everything except "Index Size" and "Documents", but this is how ours looks as far as how much memory the indexes take:

    upload_2013-10-22_14-23-13.png

    The searchable document number is actually 22,145,684 (since our setup has all shards replicated so any ES node can fail without service disruption). We have about 18M posts, but we also made other content types searchable (users, reports, conversations, etc).

    So if we weren't replicating ES data to multiple servers, our index size of 22,145,684 searchable documents would be 6.6GB.

    For us, we have 13.2GB of index data (again, we have 2 copies of everything) spread across 8 servers (so about 1.6GB per server is used for indexes), and I allocate 4GB of RAM for ES on each server just for good measure (servers each have 256GB RAM, so not short on RAM and rather over-allocate than under-allocate).
     
    HWS, MattW and Walter like this.

Share This Page