How to: Basic Elasticsearch installation. (RHEL/SUSE)

Slavik

XenForo moderator
Staff member
Basic Elasticsearch Installation (RHEL / SUSE)

@Floren has an amazing repo for RHEL 6 and 7, the Elasticsearch RPM he provides is perfectly set up and currently I suggest using it.

The old manual setup guide can be found below.

Step 1) Install the Axivo Repo: https://www.axivo.com/resources/repository-setup.1/

Step 2) Install ElasticSearch: https://www.axivo.com/resources/elasticsearch-setup.11/




This guide is provided to show how to do a basic (vanilla get up and go) install of Elasticsearch (0.90.0 Beta 1), the Elasticsearch Service Wrapper and the required Java Runtime Environment (JRE) (1.7.0_17) on RHEL / SUSE. This guide will not cover running a dedicated Elasticsearch user.

For Debian/Ubuntu users, a guide can be found here.

This guide assumes the user has basic knowledge of SSH and prior to starting the steps below has logged in as root. This guide also assumes the user does not currently have any JRE installed. You can check if you have JRE installed by typing

Code:
java -version

As of writing, the current file locations for JRE are as follows:

32 bit
Code:
http://download.oracle.com/otn-pub/java/jdk/7u17-b02/jre-7u17-linux-i586.rpm

64 bit
Code:
http://download.oracle.com/otn-pub/java/jdk/7u17-b02/jre-7u17-linux-x64.rpm

The guide will be shown using the 64 bit install, however if you are using a 32 bit system, change the file names as appropriate.

Please note, whilst this is a simple and easy setup, I take no responsibility for any damages or losses that may occur to your system by following the steps below. If you are unsure at any stage, please ask for assistance or seek the help of a qualified Linux Systems Administrator.

Installing the JRE
Type the following commands into your SSH terminal.
Code:
cd /tmp
wget http://download.oracle.com/otn-pub/java/jdk/7u17-b02/jre-7u17-linux-x64.rpm
rpm -ivh jre-7u17-linux-x64.rpm
java -version

Assuming everything was done correctly, you should get the following output.

Code:
# java -version
java version "1.7.0_17"
Java(TM) SE Runtime Environment (build 1.7.0_17)
Java HotSpot(TM) 64-Bit Server VM (build 22.0-b10, mixed mode)

Install Elasticsearch

Code:
cd /
curl -L -O -k https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.0.Beta1.zip
unzip elasticsearch-0.90.0.Beta1.zip
mv elasticsearch-0.90.0.Beta1 elasticsearch

Install the Elasticsearch Service Wrapper

Code:
curl -L -k http://github.com/elasticsearch/elasticsearch-servicewrapper/tarball/master | tar -xz
mv *servicewrapper*/service elasticsearch/bin/
elasticsearch/bin/service/elasticsearch install
ln -s `readlink -f elasticsearch/bin/service/elasticsearch` /usr/local/bin/rcelasticsearch
rcelasticsearch start

Assuming everything was done correctly, you should see the following output.

Code:
rcelasticsearch start
Starting ElasticSearch...
Waiting for ElasticSearch......
running: PID: xxxxx

Basic Configuration

You should do some basic configuration of Elasticsearch before installing the addon in XenForo.


1) Open up /elasticsearch/config/elasticsearch.yml and on line 32 edit

Code:
# cluster.name: elasticsearch

To

Code:
cluster.name: PUT-SOMETHING-UNIQUE-HERE

on line 199 edit

Code:
# network.host: 192.168.0.1

to

Code:
network.host: 127.0.0.1

On line 211 edit

Code:
# http.port: 9200

to

Code:
http.port: 9200

Save and Close


2) Open up /elasticsearch/bin/service/elasticsearch.conf on line 2 edit

Code:
set.default.ES_HEAP_SIZE=1024

To a number suitable for the size of your forum.

I reccomend approximately 1 GB for the HEAP_SIZE per 1 million posts on your forum.

1 Million Posts: 1024
2 Million Posts: 2048
3 Million Posts: 3072
4 Million Posts: 4096
etc

This will not mean the service will use all that available memory, however it will have it at its disposal if required.

So for example a 3 Million Post forum would edit

Code:
set.default.ES_HEAP_SIZE=1024

to

Code:
set.default.ES_HEAP_SIZE=3072



Save and Exit.


3) Optional - Move the Elasticsearch data directory.

Your linux install may be configured in such a way that your install partition is only a few Gb in size, and placing a large Elasticsearch index there is not ideal.

In which case you will want to move the index directory to a different, larger, location (in this example /var/elasticsearch)

Code:
cd /var
mkdir elasticsearch

Open up /elasticsearch/config/elasticsearch.yml on line 143 edit

Code:
# path.data: /path/to/data

to

Code:
path.data: /var/elasticsearch

Save and Exit

4) Restart the Elasticsearch Service

In SSH type

Code:
rcelasticsearch restart

You should get the following output

Code:
rcelasticsearch restart
Stopping ElasticSearch...
Stopped ElasticSearch.
Starting ElasticSearch...
Waiting for ElasticSearch......
running: PID: xxxxx

Elasticsearch is now runing with your updated config.



Install the XenForo Enhanced Search Addon

1) Turn your board off into maintainance mode*

2) Download the addon from your customer area at http://xenforo.com/customers/

3) Follow the instructions found at http://xenforo.com/help/enhanced-search/

4) Wait for your indexes to be rebuilt

5) Open your board.

6) Install the index pre-warmer.

As of 0.90.0 Beta an index pre-warmer is available. This keeps your search index "warm" in active memory so when a search is done, the access time latency is highly reduced.

Installing this is simple, in SSH simply run the following replacing the *INDEX NAME* with the name of your ES index.

Code:
curl -XPUT localhost:9200/*INDEX NAME*/_warmer/warmer_1 -d '{
    "query" : {
        "match_all" : {}
    }
}'

You should have the following returned

Code:
{"ok":true,"acknowledged":true}


*You may leave your board open during the re-index process.

Congratulations. Your board should now be running XenForo Enhanced Search.
 
Last edited:
@Slavik, I'm looking at Elastic as we speak.
I'm confused with the warmer part. I looked at the elasticsearch.yml file and I don't see anywhere mentioned the index name?
Rich (BB code):
curl -XPUT localhost:9200/*INDEX NAME*/_warmer/warmer_1 -d '{
    "query" : {
        "match_all" : {}
    }
}'
Thanks for your clarification.
The index name is the name of the index you put in the XenForo configuration (Options -> Search options -> Elasticsearch index name
 
Nice, I did not get to install the add-on yet... just purchased yesterday. :)
Now, I'm looking at this template:
https://github.com/elasticsearch/elasticsearch/issues/1917

Something like:
Code:
curl -XPUT localhost:9200/test/warmers/warmer_1 -d '{
    "query" : {
        "match_all" : {}
    },
    "facets" : {
        "facet_1" : {
            "terms" : {
                "field" : "field"
            }
        }
    }
}'
Or:
Code:
curl -XPUT localhost:9200/test -d '{
    "warmers" : {
        "warmer_1" : {
            "types" : [],
            "source" : {
                "query" : {
                    "match_all" : {}
                }
            }
        }
    }
}'
Let me know if is possible to set the warmers like that.
 
Last edited:
I'm going to bed, I worked all night and I need some sleep. If you can please try the above config in a test forum, thanks. I'll look at it when I wakeup... The new elasticsearch rpm is really neat, you will like it. :)

I don't have any testbeds running at the moment im afraid :( Nuked them all after doing the latest work with LiteSpeed.
 
@Mouth posted this into Debian thread:
Now even easier again to install ElasticSearch (and automatically keep up-to-date with latest version) onto your Debian/Ubuntu server ...
  1. sudo wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | apt-key add -
  2. sudo echo "deb http://packages.elasticsearch.org/elasticsearch/0.90/debian stable main" > /etc/apt/sources.list.d/elastic_search.list
  3. sudo apt-get update
  4. sudo apt-get install elasticsearch
Follow the instructions in first post from the section Basic Configuration onwards
I did not checked Debian but for RHEL their rpm package is a disaster. :)
I invite you to look at on the Debian side also, as I'm sure they have few things wrong. I emailed the developers and let them know about the issues, while asking them how they build the packages as they don't follow any standards. Basically, when a new version is released, they build everything with a script. The BIG problem is: they don't follow the package standards or the proper locations where files should be stored. Plus, for RHEL, they add a bunch of files that conflict with the system. Example for RHEL 6, see the wrong files added below:
Rich (BB code):
/etc/elasticsearch
/etc/elasticsearch/elasticsearch.yml
/etc/elasticsearch/logging.yml
/etc/rc.d/init.d
/etc/rc.d/init.d/elasticsearch
/etc/sysconfig
/etc/sysconfig/elasticsearch
/etc/systemd/system
/etc/systemd/system/elasticsearch.service
/etc/tmpfiles.d
/etc/tmpfiles.d/elasticsearch.conf
/usr/lib/sysctl.d
/usr/lib/sysctl.d/elasticsearch.conf
/usr/share/elasticsearch/bin
/usr/share/elasticsearch/bin/elasticsearch
/usr/share/elasticsearch/bin/elasticsearch.in.sh
/usr/share/elasticsearch/bin/plugin
/usr/share/elasticsearch/lib/elasticsearch-0.90.9.jar
/usr/share/elasticsearch/lib/jna-3.3.0.jar
/usr/share/elasticsearch/lib/jts-1.12.jar
/usr/share/elasticsearch/lib/log4j-1.2.17.jar
/usr/share/elasticsearch/lib/lucene-analyzers-common-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-codecs-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-core-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-expressions-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-grouping-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-highlighter-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-join-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-memory-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-misc-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-queries-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-queryparser-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-sandbox-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-spatial-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-suggest-4.6.0.jar
/usr/share/elasticsearch/lib/sigar
/usr/share/elasticsearch/lib/sigar/libsigar-amd64-linux.so
/usr/share/elasticsearch/lib/sigar/libsigar-ia64-linux.so
/usr/share/elasticsearch/lib/sigar/libsigar-x86-linux.so
/usr/share/elasticsearch/lib/sigar/sigar-1.6.4.jar
/usr/share/elasticsearch/lib/spatial4j-0.3.jar
/usr/share/elasticsearch/LICENSE.txt
/usr/share/elasticsearch/NOTICE.txt
/usr/share/elasticsearch/README.textile
/var/lib/elasticsearch
/var/log/elasticsearch
/var/run/elasticsearch
Pretty much the entire package architecture is wrong. They build a noarch package when in fact they should build separate packages. They don't use for example sigar 1.6.5 which is provided by default in RHEL and have their own custom version. That is fine... they probably need it this way, but is wrong. They should build their package around existing software in RHEL. I kept also the sigar version they produce for sanity reasons, as I have no time to dive into deeper testing but I will email them and let them know. Another thing, tmpfiles and systemd is specific to RHEL 7/Fedora 20, RHEL 6 does not have that. And /usr/lib/sysctl.d does not exist in RHEL either. Well, it exists in /etc/sysctl.d if you know where to look for, as is hidden (bug in RHEL or crazy decision on their side to hide it?):
Rich (BB code):
$ grep sysctl /etc/init.d/functions
# Apply sysctl settings, including files in /etc/sysctl.d
apply_sysctl() {
    sysctl -e -p /etc/sysctl.conf >/dev/null 2>&1
    for file in /etc/sysctl.d/* ; do
        test -f "$file" && sysctl -e -p "$file" >/dev/null 2>&1
$ ls -lah /etc/sysctl.d/
ls: cannot access /etc/sysctl.d/: No such file or directory
I released a proper package for CentOS/Red Hat 6.5 into Axivo repository:
Code:
# yum --enablerepo=axivo install elasticsearch
# yum info elasticsearch
Installed Packages
Name        : elasticsearch
Arch        : x86_64
Version     : 0.90.9
Release     : 1.el6
Size        : 19 M
Repo        : installed
Summary     : Distributed RESTful full-text search engine
URL         : http://www.elasticsearch.org
License     : ASL 2.0
Description : ElasticSearch is a distributed, scalable, multitenant-capable full-text search
            : engine with a RESTful web interface and schema-free JSON documents. It uses
            : Lucene and tries to make all features of it available through the JSON and Java
            : API. It supports real-time GET requests, faceting and percolating, which can be
            : useful for notifying if new documents match for registered queries.
# chkconfig elasticsearch on
# service elasticsearch start
# service elasticsearch status
elasticsearch (pid  13975) is running...
In /etc/sysconfig/elasticsearch, you can adjust the maximum memory allocation pool size.
Also, look at /etc/security/limits.d/60-elasticsearch.conf. You are done. :)
Give it a try and let me know if you have any issues with the package.

@Slavik, if you find this useful, please link it to your OP so others can read it.
 
Last edited:
@Mouth posted this into Debian thread:

I did not checked Debian but for RHEL their rpm package is a disaster. :)
I invite you to look at on the Debian side also, as I'm sure they have few things wrong. I emailed the developers and let them know about the issues, while asking them how they build the packages as they don't follow any standards. Basically, when a new version is released, they build everything with a script. The BIG problem is: they don't follow the package standards or the proper locations where files should be stored. Plus, for RHEL, they add a bunch of files that conflict with the system. Example for RHEL 6, see the wrong files added below:
Rich (BB code):
/etc/elasticsearch
/etc/elasticsearch/elasticsearch.yml
/etc/elasticsearch/logging.yml
/etc/rc.d/init.d
/etc/rc.d/init.d/elasticsearch
/etc/sysconfig
/etc/sysconfig/elasticsearch
/etc/systemd/system
/etc/systemd/system/elasticsearch.service
/etc/tmpfiles.d
/etc/tmpfiles.d/elasticsearch.conf
/usr/lib/sysctl.d
/usr/lib/sysctl.d/elasticsearch.conf
/usr/share/elasticsearch/bin
/usr/share/elasticsearch/bin/elasticsearch
/usr/share/elasticsearch/bin/elasticsearch.in.sh
/usr/share/elasticsearch/bin/plugin
/usr/share/elasticsearch/lib/elasticsearch-0.90.9.jar
/usr/share/elasticsearch/lib/jna-3.3.0.jar
/usr/share/elasticsearch/lib/jts-1.12.jar
/usr/share/elasticsearch/lib/log4j-1.2.17.jar
/usr/share/elasticsearch/lib/lucene-analyzers-common-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-codecs-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-core-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-expressions-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-grouping-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-highlighter-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-join-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-memory-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-misc-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-queries-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-queryparser-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-sandbox-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-spatial-4.6.0.jar
/usr/share/elasticsearch/lib/lucene-suggest-4.6.0.jar
/usr/share/elasticsearch/lib/sigar
/usr/share/elasticsearch/lib/sigar/libsigar-amd64-linux.so
/usr/share/elasticsearch/lib/sigar/libsigar-ia64-linux.so
/usr/share/elasticsearch/lib/sigar/libsigar-x86-linux.so
/usr/share/elasticsearch/lib/sigar/sigar-1.6.4.jar
/usr/share/elasticsearch/lib/spatial4j-0.3.jar
/usr/share/elasticsearch/LICENSE.txt
/usr/share/elasticsearch/NOTICE.txt
/usr/share/elasticsearch/README.textile
/var/lib/elasticsearch
/var/log/elasticsearch
/var/run/elasticsearch
Pretty much the entire package architecture is wrong. They build a noarch package when in fact they should build separate packages. They don't use for example sigar 1.6.5 which is provided by default in RHEL and have their own custom version. That is fine... they probably need it this way, but is wrong. They should build their package around existing software in RHEL. I kept also the sigar version they produce for sanity reasons, as I have no time to dive into deeper testing but I will email them and let them know. Another thing, tmpfiles and systemd is specific to RHEL 7/Fedora 20, RHEL 6 does not have that. And /usr/lib/sysctl.d does not exist in RHEL either. Well, it exists in /etc/sysctl.d if you know where to look for, as is hidden (bug in RHEL or crazy decision on their side to hide it?):
Rich (BB code):
$ grep sysctl /etc/init.d/functions
# Apply sysctl settings, including files in /etc/sysctl.d
apply_sysctl() {
    sysctl -e -p /etc/sysctl.conf >/dev/null 2>&1
    for file in /etc/sysctl.d/* ; do
        test -f "$file" && sysctl -e -p "$file" >/dev/null 2>&1
$ ls -lah /etc/sysctl.d/
ls: cannot access /etc/sysctl.d/: No such file or directory
I released a proper package for CentOS/Red Hat 6.5 into Axivo repository:
Code:
# yum --enablerepo=axivo install elasticsearch
# yum info elasticsearch
Installed Packages
Name        : elasticsearch
Arch        : x86_64
Version     : 0.90.9
Release     : 1.el6
Size        : 19 M
Repo        : installed
Summary     : Distributed RESTful full-text search engine
URL         : http://www.elasticsearch.org
License     : ASL 2.0
Description : ElasticSearch is a distributed, scalable, multitenant-capable full-text search
            : engine with a RESTful web interface and schema-free JSON documents. It uses
            : Lucene and tries to make all features of it available through the JSON and Java
            : API. It supports real-time GET requests, faceting and percolating, which can be
            : useful for notifying if new documents match for registered queries.
# chkconfig elasticsearch on
# service elasticsearch start
# service elasticsearch status
elasticsearch (pid  13975) is running...
In /etc/sysconfig/elasticsearch, you can adjust the maximum memory allocation pool size.
Also, look at /etc/security/limits.d/60-elasticsearch.conf. You are done. :)
Give it a try and let me know if you have any issues with the package.

@Slavik, if you find this useful, please link it to your OP so others can read it.

Excellent work Floren.
 
@Slavik
about
Elasticsearch
hmmmm is it good for a small vps ?

40GB
1GB ram
2core ?!

and does it have some issue or conflict with openlitespeed ?

i want it cuz i stop the search in my forum :( its take around 2GB size in my database
and i cannot make backup as it will make some issue
when the whole database r 6 to 7GB

so is Elasticsearch will solve this issue
 
hmmm i will see now i will try to add the google search maybe it will be good !!
if its usless i will use this
cuz i really need the search
and the problem the search index table get big size !!

That will happen in any system where you use MySQL search, its not usually an issue though unless your limited on disk space.
 
That will happen in any system where you use MySQL search, its not usually an issue though unless your limited on disk space.
so in VB they dont use this way ?!
as before in VB my DB was around 2GB after converting it was 4 ^_^ and now its 7 and i delete the search index table from time to time when i have problems :( cuz i can`t take backup for my DB
 
Back
Top Bottom