Heavy server load, host pointed this out to us...

Ludachris

Ludachris

Well-known member
Our host was troubleshooting some heavy server loads and found that it was due to Elastisearch. Here's what they pointed out:
Sorry for the delayed response, but it looks like these processes were all started by the "elasticsearch" user. I'm seeing most of these originating out of the /tmp directory on your server.

In checking out this directory, I am seeing a lot of suspiciously named directories (all of which are owned by elasticsearch). Do any of these look familiar to you?:

drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 31 05:19 .1h2dh1h21/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 31 05:38 .1h2dh1h2d1/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Mar 31 05:13 .1h2dh1hd21/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 13 04:52 .1m1m1m1m1m/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Apr 3 06:59 .1zbxzcbzcbzxcz/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 13 07:20 .aapzozuuqrq/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 8 07:36 .anxuqywqkqwr/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 31 13:59 .aooicwewewe/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 10 08:40 .blieisixuqqr/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 10 11:06 .byzyukdlq/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 8 10:51 .clcixixueq/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 8 09:31 .emdiucjwkexx/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 31 13:09 .fksoixuhyhgfwef/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 2 09:17 .gutlakxiuw/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Apr 6 18:07 .h11b1b1b/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Apr 3 09:18 .j1n1nn1n1n1/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 31 06:02 .ldkjweuiiwe/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 2 04:03 .m1m1m1m1m1/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Apr 2 04:02 .masmasmma/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 10 09:19 .mizjixhuierwe/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 2 06:01 .mjducxuywdiuw/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 2 06:14 .muicisixuqr/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 6 18:34 .n1n1n1n1n/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 7 03:50 .nabb1b1b1b1/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 9 10:13 .nahsabb1ds1/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Apr 1 04:26 .nxcxzczc/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Apr 8 07:17 .nzxnzxnzn/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 6 07:25 .oiuxqkwudxkxqwr/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 7 08:33 .ooixqiwqrqr/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Apr 13 04:46 .p012hj12/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 10 09:42 .pqoayyxtrqweyuqis/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 2 07:46 .wisquxklwxwet/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 7 08:14 .wkioxxcuiwkerwe/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 13 12:53 .wksioxuiqjwrq/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 7 08:10 .wksioxuxjklqw/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 8 06:46 .wlosixuklwqsqw/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 10 08:12 .wmaoxixqlwrq/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 13 11:25 .wmiixuixqwuriq/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 9 12:22 .wmsixjzsoqweiqw/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Mar 31 10:31 .wmskokijrwewe/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 9 07:15 .wmsxjqxuhwwqr/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 6 08:38 .wmwiuxiqwrwqsfg/
drwxr-xr-x 3 elasticsearch elasticsearch 4096 Apr 10 09:31 .wqjsqwuweutwg/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Apr 3 06:55 .zbxzcbzcbzxcz/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Apr 2 04:01 .zxchzbxcc/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Apr 1 05:01 .zxcxczxczczc/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Apr 1 04:33 .zxhzhzc/
drwxr-xr-x 2 elasticsearch elasticsearch 4096 Apr 6 05:17 .zxmzmxmzm/

Also seeing these files in here which seems suspect:

-rw-r--r-- 1 elasticsearch elasticsearch 763474 Jan 5 2007 73932
-rwxr-xr-x 1 elasticsearch elasticsearch 1128800 Apr 13 10:17 awm*
-rwxr-xr-x 1 elasticsearch elasticsearch 841596 Mar 22 22:02 bbbs*
-rwxrwxrwx 1 elasticsearch elasticsearch 1128800 Sep 10 2013 bbos-998*
-rwxr-xr-x 1 elasticsearch elasticsearch 1128800 Apr 13 05:46 butty-c.a*
-rwxr-xr-x 1 elasticsearch elasticsearch 1128808 Sep 10 2013 cmd*
-rwxrwxrwx 1 elasticsearch elasticsearch 417936 Apr 10 17:26 .Mm2*
-rwxr-xr-x 1 elasticsearch elasticsearch 1128800 Apr 5 15:40 okk*
-rwxr-xr-x 1 elasticsearch elasticsearch 1254630 Apr 7 09:42 .sshd*
-rwxrwxrwx 1 elasticsearch elasticsearch 1524643 Apr 10 17:33 .TSm*
-rwxr-xr-x 1 elasticsearch elasticsearch 1135000 Apr 12 01:12 uname*
-rw-r--r-- 1 elasticsearch elasticsearch 1128800 Dec 13 2013 wangzai
-rwxrwxrwx 1 elasticsearch elasticsearch 1128800 Sep 10 2013 xupxlc*
Click to expand...
Is this normal? Or should I be worried?
 
Sort by date Sort by votes
What version of elasticsearch are you running? Is it publicly accessible?

Seeing files called uname, cmd, and .sshd make me think your server has been compromised (likely by using an older version of elasticsearch/one with dynamic scripting enabled without firewalling it).
 
  • Like
Reactions: HWS
Upvote 0 Downvote
Yup, pretty sure that your server got hacked.

You should perform a clean install of the OS and then restore your files/DB from a clean backup. Then install the latest version of ES 1.5.1 and make sure to properly configure your iptables.
 
Upvote 0 Downvote
Super120 said:
Yup, pretty sure that your server got hacked.

You should perform a clean install of the OS and then restore your files/DB from a clean backup. Then install the latest version of ES 1.5.1 and make sure to properly configure your iptables.
Click to expand...
So if this happened several days ago then I'd potentially have to restore the DB to a week old backup? Ouch.
 
Last edited:
Upvote 0 Downvote
HWS said:
I also think you got hacked. You should never run elasticsearch at a public accessible port. This is a huge security problem.
Click to expand...
Can you explain what that means? I don't remember anything about public accessible port in the set up or configuration instructions when it was installed.
 
Upvote 0 Downvote
Ludachris said:
Can you explain what that means? I don't remember anything about public accessible port in the set up or configuration instructions when it was installed.
Click to expand...

I don't know your setup. But given the assumption you run a single server for web, database, elasticsearch and all the other stuff (like java, etc) you most likely have an open port (usually 9200) for elasticsearch/java at your servers main ip address.

If you haven't blocked ALL non-local access to that port with a firewall it is just a matter of time until you'll get hacked.

I would find it to be a good idea to add this warning to the XFES installation instructions page.
 
Last edited:
Upvote 0 Downvote
And IF you got hacked, you need to take your whole server down ASAP and restore the OS from scratch. After that you need to install all daemons again you need for your site. Also XF has to be installed from scratch. In addition I would compare the most current database backup with a backup before your server got hacked and check if suspicious database entries have been made. If not, you can use the most recent backup for your restore so your members won't loose any data.

It is also very important that you change ALL passwords at your server. Including the os root and user passwords, the database passwords and ALL passwords of all your XF user accounts. The intruder could have got possession of them from your database.

Getting hacked on the server root level is the most problematic and painful scenario you can experience with a server.
 
Upvote 0 Downvote
You must log in or register to reply here.
Back
Top Bottom