The hackers were using our server to run a DDOS against a Chinese IP. There is no evidence that they took the database (we would see SQL dumps and none are visible).
Tim shut the server down immediately. This meant that our remaining server could not cope with the search load and AVForums ground to a halt with errors.
We have one addon which relies on the search and does not obey the search enabled/disabled option in the admin settings. So even disabling the search meant we were getting Elastic Search errors continuing. Disabling the [XenMods] [Widgets] (bd Widget Framework) addon meant the forums could come back online.
The best idea of how we were hacked is that a known vulnerability in our search engine Elastic Search was used to gain root access to the server.
Elasticsearch.org Elasticsearch 1.4.3 And 1.3.8 Released | Blog | Elasticsearch
So we were hacked because
- There was a vulnerability in our Elastic Search installation because
- we were using an older version of Elastic Search which contained the vulnerability and
- we had the Elastic Search port open to the public which mean hackers could get to it.
That means we have to recreate the search index and that is ongoing. With 20 million posts and near 2 million threads, it takes hours. Until that happens, the search will either be offline or online with limited results.
What have we done to ensure this doesn't happen again?
Tim has installed the latest version of Elastic Search on the servers (which patches the vulnerability) and the 'recon' function which automatically installs Elastic Search updates so that we won't lag behind security patches in the future.
He has also taken the Elastic Search port away from public view so that nobody can access is externally any more.
We can't say for sure that the database wasn't taken, but we don't think it was as the purpose of the hack seems to have been just to launch the DDOS.
Passwords are encrypted, so they won't have been compromised. For the purposes of transparency, I have declared on AVForums that there is a small possibility that peoples' emails were taken.
If you're running Elastic Search, make sure it's running the latest version and that the post is not public facing.