• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

Heavy server load, host pointed this out to us...

Ludachris

Well-known member
#1
Our host was troubleshooting some heavy server loads and found that it was due to Elastisearch. Here's what they pointed out:
Sorry for the delayed response, but it looks like these processes were all started by the "elasticsearch" user. I'm seeing most of these originating out of the /tmp directory on your server.

In checking out this directory, I am seeing a lot of suspiciously named directories (all of which are owned by elasticsearch). Do any of these look familiar to you?:

drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 31 05:19 .1h2dh1h21/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 31 05:38 .1h2dh1h2d1/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Mar 31 05:13 .1h2dh1hd21/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 13 04:52 .1m1m1m1m1m/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Apr 3 06:59 .1zbxzcbzcbzxcz/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 13 07:20 .aapzozuuqrq/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 8 07:36 .anxuqywqkqwr/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 31 13:59 .aooicwewewe/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 10 08:40 .blieisixuqqr/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 10 11:06 .byzyukdlq/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 8 10:51 .clcixixueq/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 8 09:31 .emdiucjwkexx/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 31 13:09 .fksoixuhyhgfwef/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 2 09:17 .gutlakxiuw/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Apr 6 18:07 .h11b1b1b/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Apr 3 09:18 .j1n1nn1n1n1/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 31 06:02 .ldkjweuiiwe/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 2 04:03 .m1m1m1m1m1/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Apr 2 04:02 .masmasmma/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 10 09:19 .mizjixhuierwe/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 2 06:01 .mjducxuywdiuw/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 2 06:14 .muicisixuqr/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 6 18:34 .n1n1n1n1n/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 7 03:50 .nabb1b1b1b1/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 9 10:13 .nahsabb1ds1/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Apr 1 04:26 .nxcxzczc/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Apr 8 07:17 .nzxnzxnzn/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 6 07:25 .oiuxqkwudxkxqwr/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 7 08:33 .ooixqiwqrqr/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Apr 13 04:46 .p012hj12/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 10 09:42 .pqoayyxtrqweyuqis/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 2 07:46 .wisquxklwxwet/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 7 08:14 .wkioxxcuiwkerwe/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 13 12:53 .wksioxuiqjwrq/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 7 08:10 .wksioxuxjklqw/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 8 06:46 .wlosixuklwqsqw/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 10 08:12 .wmaoxixqlwrq/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 13 11:25 .wmiixuixqwuriq/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 9 12:22 .wmsixjzsoqweiqw/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 31 10:31 .wmskokijrwewe/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 9 07:15 .wmsxjqxuhwwqr/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 6 08:38 .wmwiuxiqwrwqsfg/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 10 09:31 .wqjsqwuweutwg/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Apr 3 06:55 .zbxzcbzcbzxcz/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Apr 2 04:01 .zxchzbxcc/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Apr 1 05:01 .zxcxczxczczc/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Apr 1 04:33 .zxhzhzc/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Apr 6 05:17 .zxmzmxmzm/

Also seeing these files in here which seems suspect:

-rw-r--r-- 1 elasticsearch elasticsearch 763474 Jan 5 2007 73932
-rwxr-xr-x 1 elasticsearch elasticsearch 1128800 Apr 13 10:17 awm*
-rwxr-xr-x 1 elasticsearch elasticsearch 841596 Mar 22 22:02 bbbs*
-rwxrwxrwx 1 elasticsearch elasticsearch 1128800 Sep 10 2013 bbos-998*
-rwxr-xr-x 1 elasticsearch elasticsearch 1128800 Apr 13 05:46 butty-c.a*
-rwxr-xr-x 1 elasticsearch elasticsearch 1128808 Sep 10 2013 cmd*
-rwxrwxrwx 1 elasticsearch elasticsearch 417936 Apr 10 17:26 .Mm2*
-rwxr-xr-x 1 elasticsearch elasticsearch 1128800 Apr 5 15:40 okk*
-rwxr-xr-x 1 elasticsearch elasticsearch 1254630 Apr 7 09:42 .sshd*
-rwxrwxrwx 1 elasticsearch elasticsearch 1524643 Apr 10 17:33 .TSm*
-rwxr-xr-x 1 elasticsearch elasticsearch 1135000 Apr 12 01:12 uname*
-rw-r--r-- 1 elasticsearch elasticsearch 1128800 Dec 13 2013 wangzai
-rwxrwxrwx 1 elasticsearch elasticsearch 1128800 Sep 10 2013 xupxlc*
Is this normal? Or should I be worried?
 

rainmotorsports

Well-known member
#2
What is your hosting setup? Elastisearch is usually only appropriate for large VPS or dedicated servers. Not sure if load is as much a problem as memory is but still.
 

Mike

XenForo developer
Staff member
#4
What version of elasticsearch are you running? Is it publicly accessible?

Seeing files called uname, cmd, and .sshd make me think your server has been compromised (likely by using an older version of elasticsearch/one with dynamic scripting enabled without firewalling it).
 

imthebest

Formerly Super120
#5
Yup, pretty sure that your server got hacked.

You should perform a clean install of the OS and then restore your files/DB from a clean backup. Then install the latest version of ES 1.5.1 and make sure to properly configure your iptables.
 

Ludachris

Well-known member
#7
Yup, pretty sure that your server got hacked.

You should perform a clean install of the OS and then restore your files/DB from a clean backup. Then install the latest version of ES 1.5.1 and make sure to properly configure your iptables.
So if this happened several days ago then I'd potentially have to restore the DB to a week old backup? Ouch.
 
Last edited:

imthebest

Formerly Super120
#8
No that necessary to restore from such an old DB backup. I was talking basically to the PHP files of XenForo.

Try running the File Health Check tool.
 

HWS

Well-known member
#9
I also think you got hacked. You should never run elasticsearch at a public accessible port. This is a huge security problem.
 

Ludachris

Well-known member
#10
I also think you got hacked. You should never run elasticsearch at a public accessible port. This is a huge security problem.
Can you explain what that means? I don't remember anything about public accessible port in the set up or configuration instructions when it was installed.
 

HWS

Well-known member
#11
Can you explain what that means? I don't remember anything about public accessible port in the set up or configuration instructions when it was installed.
I don't know your setup. But given the assumption you run a single server for web, database, elasticsearch and all the other stuff (like java, etc) you most likely have an open port (usually 9200) for elasticsearch/java at your servers main ip address.

If you haven't blocked ALL non-local access to that port with a firewall it is just a matter of time until you'll get hacked.

I would find it to be a good idea to add this warning to the XFES installation instructions page.
 
Last edited:

HWS

Well-known member
#12
And IF you got hacked, you need to take your whole server down ASAP and restore the OS from scratch. After that you need to install all daemons again you need for your site. Also XF has to be installed from scratch. In addition I would compare the most current database backup with a backup before your server got hacked and check if suspicious database entries have been made. If not, you can use the most recent backup for your restore so your members won't loose any data.

It is also very important that you change ALL passwords at your server. Including the os root and user passwords, the database passwords and ALL passwords of all your XF user accounts. The intruder could have got possession of them from your database.

Getting hacked on the server root level is the most problematic and painful scenario you can experience with a server.