How To Install Elasticsearch 7.x for Xenforo on CentOS

Sunka

Well-known member
How To Install Elasticsearch 7.x for Xenforo


This is tutorial for installing (not upgrading) elasticsearch 7 on centos.
If I am missing something, please post here.

Delete first old one if you have installed and install fresh new one elasticsearch v7
Code:
yum remove elasticsearch

Installing elasticsearch 7

Code:
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
Code:
nano /etc/yum.repos.d/elasticsearch.repo
insert this:
Code:
[elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
If you do not have java installed, install it (elasticsearch itself includes a bundled version of OpenJDK, but, just to be sure).
Java v11 is what you want to install. You can have both version (v.8 and v.11) but I removed v8 and installed v11
Code:
yum remove java-1.8.0-openjdk
Code:
yum -y install java-11-openjdk

Code:
yum install --enablerepo=elasticsearch elasticsearch
Code:
nano /etc/elasticsearch/elasticsearch.yml
at the end of file add this:
Code:
cluster.name: CUSTOM NAME OF YOUR CLUSTER (same name "should" be inserted into your XenForo field in Admin Control Panel in field Elasticsearch Index Name)
network.host: localhost
http.port: 9200
node.name: ANOTHER CUSTOM NAME
cluster.initial_master_nodes: ANOTHER CUSTOM NAME (same as line above, and use all lowercase)

Also, in same above template change/modify the path -Djava.io.tmpdir=
Code:
-Djava.io.tmpdir=/home/elasticsearch

Then (thanks @rdan )
Code:
mkdir /home/elasticsearch

and

Code:
chown -R elasticsearch:elasticsearch /home/elasticsearch/


set how much ram you will add for elasticsearch - 512 megabytes is set in example (both Xms and Xmx must have same value)
Code:
nano /etc/elasticsearch/jvm.options
Code:
-Xms512m
-Xmx512m


Code:
service elasticsearch start
Code:
systemctl daemon-reload
Code:
systemctl enable elasticsearch.service
Code:
systemctl start elasticsearch.service

test if everything OK (give 1-2 minute after restarting elasticsearch)
Code:
curl -XGET 'localhost:9200'
If everything OK, you will see something like this in your CLI
Code:
# curl -XGET 'localhost:9200'
{
  "name" : "xxxx",
  "cluster_name" : "xxxxx",
  "cluster_uuid" : "4nAffvfpSGG-RE4ou1ttFA",
  "version" : {
    "number" : "7.6.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "7f634e9f44834fbc12724506cc1da681b0c3b1e3",
    "build_date" : "2020-02-06T00:09:00.449973Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Also, just in case run command and choose Java 11 (if it will list more than 1 java version):
Code:
alternatives --config java

Also, it is better that you not use shard replica if Elasticsearch is on same server.
So for make by default no replica shards, run this command

Code:
curl -XPUT "localhost:9200/_template/all" -H'Content-Type: application/json' -d'
{
  "template": "*",
  "settings": {
    "number_of_replicas": 0,
    "number_of_shards": 1
  }
}
'

To see if you have unassigned replica shards run:

Code:
curl -s -XGET http://localhost:9200/_cat/shards | fgrep UNASSIGNED

To delete unassigned replica shards run:

Code:
curl -XGET http://localhost:9200/_cat/shards | grep UNASSIGNED | awk {'print $1'} | xargs -i curl -XDELETE "http://localhost:9200/{}"

Then deleted the index. After that go to ElasticSearch Setup in Xenforo ACP then rebuilt the index without deleting.

Minimum version of XenForo Enhanced Search you have installed for elasticsearch 7 is 2.0 I think!

After you installed elastic search, configure relevant options for enhanced search in your Xenforo ACP and rebuild index (all).
It is better and faster to rebuild index through cli. So:

Trunucate old index first (if you have it):
Code:
cd /PATH/TO/YOUR/XENFERO/INSTALL/public/

// SOMETHING LIKE: cd /home/nginx/domains/your_domain.com/public/
and then

Rebuild search with trunucating index before that
Code:
php cmd.php xf-rebuild:search --truncate
or just rebuild without trunucating index
Code:
php cmd.php xf-rebuild:search

And how it looks in elasticsearch setup page in xenforo ACP

setup.webp
 
I wish there were some XenForo oriented professional support for Elasticsearch.

I have now spent about 30 hours trying to upgrade the server to the latest version. The elasticsearch.co site is very confusing about downloads and upgrades that seem to imply it is now only available as a Docker image. In further looking I found a reference to Composer but I think that is only if I am developing an alternative to xF Advanced Search.

Attempts to use the directions given above in this thread consistently return a 404 message when I try to get the repo.
 
What OS are you using? Centos I assume?

Without wishing to be unkind, you seem very much a beginner with Linux server admin. Installing software like Elasticsearch is a little more involved as it isn't in the default repositories.

sudo is a command used to run privileged commands as a non-root user. Therefore it is unnecessary to use sudo if you are root. Whatever went wrong or right was not down to sudo.

The commands you are trying to run aren't a magic incantation. Its 4 simple commands.

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.10.0-x86_64.rpm

Downloads the package. (Alter version number for newer versions)

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.10.0-x86_64.rpm.sha512

Downloads a checksum file you can use to verify the authenticity of the download. You don't really need this.

shasum -a 512 -c elasticsearch-7.10.0-x86_64.rpm.sha512

Checks the downloaded rpm is authentic. You can skip this step and the one before unless you are especially paranoid.

sudo rpm --install elasticsearch-7.10.0-x86_64.rpm

Actually installs the package.

As the poster before suggests, read the instructions on the Elastic website. I'd use the repo based method personally.

The Elastic rpm install page is literally the first page in the Google search for "install elasticsearch Centos" so I struggle to see how you ended up reading Docker instructions.
 
I have been and still am confused by the way cPanel/WHM handles this package. It seems to disappear when some automatic updates are installed.

I am not sure I am naïve, but I am certainly ignorant of the conventions of Linux systems. I end up spending twice as much time reading the help files as I do actually entering useful commands or content. The NAWCC uses an in house computer for XenForo and we do not have anyone on staff who can do this job. With over 50 years developing software I get the short stick because I am the only one who will volunteer.

I successfully installed Elasticsearch using the simple wget and rpm approach you gave above.

I am still curious if the file /etc/yum.repos.d/elasticsearch.repo and its contents play any role in the installation when I use the rpm command. I did edit it to have the contents:
Code:
[elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/
gpgcheck=1
gpgkey=https//artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
but perhaps it is only used with the yum command.

In any case thank you @Overscan . Your approach brought it all down to a level I was capable of following.
 
I have been and still am confused by the way cPanel/WHM handles this package. It seems to disappear when some automatic updates are installed.

I am not sure I am naïve, but I am certainly ignorant of the conventions of Linux systems. I end up spending twice as much time reading the help files as I do actually entering useful commands or content. The NAWCC uses an in house computer for XenForo and we do not have anyone on staff who can do this job. With over 50 years developing software I get the short stick because I am the only one who will volunteer.

I successfully installed Elasticsearch using the simple wget and rpm approach you gave above.

I am still curious if the file /etc/yum.repos.d/elasticsearch.repo and its contents play any role in the installation when I use the rpm command. I did edit it to have the contents:
Code:
[elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/
gpgcheck=1
gpgkey=https//artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
but perhaps it is only used with the yum command.

In any case thank you @Overscan . Your approach brought it all down to a level I was capable of following.
Creating the /etc/yum.repos.d/elasticsearch.repo file is intended to give yum, the standard package manager for rpm based Linux, the ability to download and install elasticsearch rpms for you. The standard Centos repositories don't contain Elastic packages for various reasons. This file has no bearing on the manual rpm install process and is not needed for this.

cPanel/WHM has its own packages but not Elasticsearch I think. It is probably issuing a regular 'yum update' process for you, so having this config for yum might be how your manually installed Elasticsearch disappears. In which case, you could remove the /etc/yum.repos.d/elasticsearch.repo file, and rely on manually update Elasticsearch from rpm as you have been. Bear in mind that you'll have to remember to do this regularly and Elastic could end up out of date and insecure if you don't- that's the whole reason to automate the process with yum.

I have my Ubuntu Xenforo server set to update Elastic via apt (Ubuntu/Debian equivalent of yum) which means install Elasticsearch updates along with ordinary OS updates. Its not been a problem so far.

Having not used cPanel, the intricacies of how it interacts with OS packages is not in my experience. In my opinion cPanel type products cause almost as much trouble as they fix, by abstracting the user from understanding what they are doing underneath. Learning curve without it is a bit steeper but I don't regret ditching web control panels and don't miss them.
 
I am not sure about the reboot or the upgrade restart functions.

Reboot seems to able to include Elasticsearch with the following commands executed once before the system is rebooted. i.e. no need to do this every time.
/bin/systemctl daemon-reload
/bin/systemctl enable elasticsearch.service

The file /etc/sysconfig/elasticsearch has the configuration values for Elasticsearch configuration. In particular RESTART_ON_UPGRADE defaults to False to avoid issues with multiple Elasticsearch images on complex systems. We only have a single instance so I think I could set this to True by removing the hashmark from the beginning of the line.
 
Last edited:
I am not sure about the reboot or the upgrade restart functions.

Reboot seems to able to include Elasticsearch with the following commands executed once before the system is rebooted. i.e. no need to do this every time.


The file /etc/sysconfig/elasticsearch has the configuration values for Elasticsearch configuration. In particular RESTART_ON_UPGRADE defaults to False to avoid issues with multiple Elasticsearch images on complex systems. We only have a single instance so I think I could set this to True by removing the hashmark from the beginning of the line.
yes that is correct. Just making sure you weren't 'losing' Elasticsearch by rebooting.

systemctl enable elasticsearch.service

This command enables a pre-created (installed by the rpm install) systemd 'service' which runs Elasticsearch automatically at boot. The parameters for this are described in a text file - on my server this is located at /usr/lib/systemd/system/elasticsearch.service. You can easily create such s file to run literally anything at boot.

Almost all Linux config is done via text files.
 
I am getting lots of errors from Elasticsearch on my system, so I keep staring at it and trying to understand what is going on.

One of the things I worry abouot is the status report for ES
Code:
[HEADING=1]Elasticsearch information[/HEADING]
Version 7.14.0
Documents 1,627,751 (1,000 MB)
Index updates 1,868,327 (0 milliseconds average)
Searches 1,744 (23 milliseconds average)
Allocated memory 91.4 KB
Cluster status yellow
Active shards 2 (67%)
Unassigned shards 1

I think on my reltively small system I should be running with a single shard instead of 3, but I cannot figure out how to fix that.

I could edit the elasticsearch.yml file for the server, but the values there now are missing and the default for 7.14.0 should be 1. I do not know where the 3 is coming from.

I just rebuilt the index and deleted all the system error records but I would really like to understand the configuration better.
 
Thank you for the guide. I refer to this very often and you explained things very well.

I just upgraded to 7.16 and it seems that after the upgrade, an unassigned replica shard was created. It makes my status “yellow” and I have been unable to solve this. I’d like to understand why this was created and find a way to remove it.

Has this happened to anyone else during the latest upgrade?
 
This is what solved it for me, assuming my single node configuration(replace index name):

Code:
curl -XPUT "localhost:9200/<INDEX_NAME>/_settings?pretty" -H 'Content-Type: application/json' -d' { "number_of_replicas": 0 }'
 
No that's generally a good thing.
You generally want all your shards to be active and status green.
The common cause of this to be yellow is you have a single node Elasticsearch cluster but Elasticsearch is configured to make replicas. In a cluster each shard is active on one node in a cluster, and the other nodes have replicas of the active node's shard. One node can't have both an active shard and a replica of that shard live.

Default config is often replicas : 1, which means on a single node you may have only 50% of shards active, the other shards exist (as you said replicas:1) but have nowhere to start and cannot be replicated to - hence yellow state.
If you have a single elasticsearch server you can safely set the number of replicas to 0.

I fixed this on my server. Here's some links I found useful.

 
just for reference elasticsearch 8 on almalinux/centos

/etc/yum.repos.d/elasticsearch.repo
Code:
[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md
run
Code:
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
yum install --enablerepo=elasticsearch elasticsearch
systemctl enable elasticsearch.service
yum install java-11-openjdk-devel
edit /etc/elasticsearch/elasticsearch.yml
Code:
cluster.name: forumsearch
node.name: node-forumsearch
http.port: 9200
xpack.security.enabled: false
run
Code:
service elasticsearch start
result
Code:
[root@localhost ~]# curl -XGET 'localhost:9200'
{
  "name" : "node-forumsearch",
  "cluster_name" : "forumsearch",
  "cluster_uuid" : "AJTUbKM-Ql2-7ELr8MMopQ",
  "version" : {
    "number" : "8.7.1",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "f229ed3f893a515d590d0f39b05f68913e2d9b53",
    "build_date" : "2023-04-27T04:33:42.127815583Z",
    "build_snapshot" : false,
    "lucene_version" : "9.5.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
 
just for reference elasticsearch 8 on almalinux/centos

/etc/yum.repos.d/elasticsearch.repo
Code:
[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md
run
Code:
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
yum install --enablerepo=elasticsearch elasticsearch
systemctl enable elasticsearch.service
yum install java-11-openjdk-devel
edit /etc/elasticsearch/elasticsearch.yml
Code:
cluster.name: forumsearch
node.name: node-forumsearch
http.port: 9200
xpack.security.enabled: false
run
Code:
service elasticsearch start
result
Code:
[root@localhost ~]# curl -XGET 'localhost:9200'
{
  "name" : "node-forumsearch",
  "cluster_name" : "forumsearch",
  "cluster_uuid" : "AJTUbKM-Ql2-7ELr8MMopQ",
  "version" : {
    "number" : "8.7.1",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "f229ed3f893a515d590d0f39b05f68913e2d9b53",
    "build_date" : "2023-04-27T04:33:42.127815583Z",
    "build_snapshot" : false,
    "lucene_version" : "9.5.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
Did you upgrade from ES 7?
 
Top Bottom