My server has been compromised.

[jgas]

What should I do to understand what's caused the problem and how to fix it?

Hi, recently my server starter to suffer from "massive amounts of outbound udp packets", just from time to time. When it happens, my hosting company turns the server down for a day, then it re-enable it and for a couple of weeks everything seems ok, but after the problem happens again.

The hosting company is doing nothing to help me because it's an unmanaged server, and since I'm not a programmer or technician (I'm doing marketing!) I didn't know that I could encounter such big problems like these.
If they are going to solve the problem in my place, they want me to pay for their technical services that will cost me hundreds of pounds, so the best thing for me will just be to change host...

So, I started to learn from scratch the basics of ssh and with the help of some tutorial I installed fail2ban (that ban an ip for example after 3 times it tries to connect to ssh) and rkhunter , that search for maliciuous scripts in my server.

Both were not so useful... rkhunter only pointed out these warnings:

Warning: Application 'httpd', version '2.2.3', is out of date, and possibly a security risk.
Warning: Application 'openssl', version '0.9.8e', is out of date, and possibly a security risk.
Warning: Application 'php', version '5.2.10', is out of date, and possibly a security risk.
Warning: Application 'sshd', version '4.3p2', is out of date, and possibly a security risk.

Do you think that they REALLY need to be updated?
I asked my hosting company to update them, but they are REALLY SLOW to answer me. Imagine that to have my server online again I had to CALL THEM in the uk and wait 20 minutes for the customer service to answer...

So, in the end, my question is: I know that in the future someone will try to launch an attack from my server again, so how can I track what he is doing, and finally understand which is the hole in the server?

Many thanks!

 

[TPerry]

Warning: Application 'openssl', version '0.9.8e', is out of date, and possibly a security risk.

Not sure what the current version is, but looking at the others it is probably way out of date also.

Warning: Application 'php', version '5.2.10', is out of date, and possibly a security risk.

This is WAY out of date. 5.4.xx is current and 5.5 is right around the corner - so yes, it DOES need to be upgraded.

Warning: Application 'sshd', version '4.3p2', is out of date, and possibly a security risk.

Considering that this is what you SSH via, it is vitally important that it be kept up to date.
I believe that the current version is around 6.2 (again, you are WAY out of date).

Do you think that they REALLY need to be updated?

Uh, yeah - like yesterday.

I asked my hosting company to update them, but they are REALLY SLOW to answer me. Imagine that to have my server online again I had to CALL THEM in the uk and wait 20 minutes for the customer service to answer...
So, in the end, my question is: I know that in the future someone will try to launch an attack from my server again, so how can I track what he is doing, and finally understand which is the hole in the server?

If you are going to get an unmanaged system then you need to either do 1 of 2 things
1) Hire and administrator for it
2) learn to administer it yourself

The most important thing is keeping the system up to date with the latest versions that are stable.

 

[Slavik]

Fixing an already comprimised and outdated server is akin to using band-aids to fix a stab wound.

If your current company is providing such terrible service, the first thing i'd do is look to find a new host.

Then, get a new server get it all up to date, secure and running and ready for your site (If you cant do this, then hire someone who can), and then start transfering your data over, checking it through for any scripts or malicious code that may have been put in your web files allowing the attacker access.

 

[TPerry]

Fixing an already comprimised and outdated server is akin to using band-aids to fix a stab wound.

If your current company is providing such terrible service, the first thing i'd do is look to find a new host.

Then, get a new server get it all up to date, secure and running and ready for your site (If you cant do this, then hire someone who can), and then start transfering your data over, checking it through for any scripts or malicious code that may have been put in your web files allowing the attacker access.

From what he indicated from his current hosting comments - I'd be running as fast as I could to find another hosting solution.
If he's going to maintain it himself then he REALLY needs to brush up on administration of a Linux server (I'll be the first to say it's not for the faint of heart and not for everybody).

 

[Biker]

I'm sorry, but this is one of those cases where the OP needs to bring someone on board that knows what they're doing.

Don't get me wrong, I'm all for folks learning how to do things on their own. But you don't do it on a live, production server that talks to the world. All that does is create another compromised box that everyone else has to worry about.

If the OP wants to learn Linux and how to run a secure server, do it on a home network. THEN apply those skills to a live server.

 

[TPerry]

If the OP wants to learn Linux and how to run a secure server, do it on a home network. THEN apply those skills to a live server.

Agreed 100%. That's how I taught myself (and am still teaching - up now to 7 flavors of Linux installed on computers here in the computer room - 6 desktops and 2 actual servers).

 

[jgas]

Thank you guys! I understand that learning everything will be to much for me... I just don't have time to do that.
Maybe, the easiest solution will be to find a fully managed virtual server, and that's it, as you clearly pointed out
Any recommendation for a virtual managed server with 1 gb ram, 20 gb webspace, 500 gb of bandwidth, 1 core of at least 2 gb? (most of the traffic will be from Italy and Europe).

 

[Slavik]

Thank you guys! I understand that learning everything will be to much for me... I just don't have time to do that.
Maybe, the easiest solution will be to find a fully managed virtual server, and that's it, as you clearly pointed out
Any recommendation for a virtual managed server with 1 gb ram, 20 gb webspace, 500 gb of bandwidth, 1 core of at least 2 gb? (most of the traffic will be from Italy and Europe).

I believe nimbus hosting are fully managed servers, and are quite highly reccomended around here.

 

[jgas]

Nimbus can't take me onboard (I already tried...) because due to a legal restriction that is in place between Nimbus and my hosting, Nimbus can't take me as a client. That's what Nimbus customer care wrote me...


What about a small orange?

Me: Hi! I'd like to buy one of your vps with 2gb ram.
Are your vps fully managed, semi managed or unmanaged?
Thanks

Support: Hello
We manage the important aspects of the VPS Server, we install and include WHM/cPanel for you. We handle any requests related to the server, such as OS Upgrade, PHP Upgrades, etc. We monitor the server for ping, and should your site become unavailable we investigate why.

Me: Thanks Kelly. What about if my server get compromised?

Support: well we have our internal firewalls as well as a custom firewall you can configure on your VPS
we can help you set up backups on your VPS instance
and in the case where you server was compromised we can assist you in reverting to an earlier backup

Me: and everything is included in the monthly fee?
Support: you would just need to contact support and we can help you set up a weekly backup schedule
yes, those things are standard
if you needed additional IP addresses for SSL Certificates etc those would be extra

 

[TPerry]

Any recommendation for a virtual managed server with 1 gb ram, 20 gb webspace, 500 gb of bandwidth, 1 core of at least 2 gb? (most of the traffic will be from Italy and Europe).

Spend some time at WHT, especially in their review areas and then the VPS offerings.
http://www.webhostingtalk.com/forumdisplay.php?f=103
http://www.webhostingtalk.com/forumdisplay.php?f=104

 

[Slavik]

Nimbus can't take me onboard (I already tried...) because due to a legal restriction that is in place between Nimbus and my hosting, Nimbus can't take me as a client. That's what Nimbus customer care wrote me...


What about a small orange?

Jake Bunce uses them and says their fantastic.

 

[dbembibre]

I dont think that the hole can be in yours outdated binaries, maybe you have the hole in the web, review your /tmp partition to see if anyone upload an script there that is using your server with any malware or something similar. Maybe a network misconfiguration is generating a lot of broadcast packages and flood the routers.

Take a look to this web http://apache-range-exploit.com/
and this web too http://www.incapsula.com/ddos/ddos-attacks
I remember a apache script (ping of death) that flood your server with a lot of udp outgoing packages.

You can prevent an udp flood with a easy iptables command. Anyway i recommend that install a firewall like apf

/sbin/iptables -A OUTPUT -p udp -m state --state NEW -j ACCEPT
/sbin/iptables -A OUTPUT -p udp -m limit --limit 50/s -j ACCEPT
/sbin/iptables -A OUTPUT -p udp -j DROP

Excuse my broken English

 

[SneakyDave]

Make sure all of your web applications: blog, forum, etc. And their addons are up to date.

 

[WSWD]

You can give Knownhost a try, though I don't believe they have any servers in Europe.

 

[TPerry]

@jgas
Here is just a little sample of why you want to make sure everything is updated and as secured as possible. Over 700 invalid attempts at SSH on one of my servers.

 

[p4guru]

@jgas
Here is just a little sample of why you want to make sure everything is updated and as secured as possible. Over 700 invalid attempts at SSH on one of my servers.

+1 i see alot of these in CSF firewall logs as well but CSF firewall would of never allowed 700+ failed attempts i don't think unless they're all different ips ? Of course great case for changing default SSH port 22 to something else

Reason all my CentOS servers have CSF Firewall installed http://configserver.com/cp/csf.html

I also use Centmin Mod and that automatically installs CSF as well http://centminmod.com/csf_firewall.html

Also install Malware Detect (maldet) as well http://www.rfxn.com/projects/linux-malware-detect/

There's also free AVG Antivirus for Linux that Centmin Mod posted at http://centminmod.com/avg_antivirus_free.html.

While AVG is way faster at scanning that Maldet, I have had problems with AVG virus definition updates failing to update on some servers while on others it works fine. So i'd use AVG and Maldet in conjunction to complement each other (if AVG works for you).

How much speed difference ? On a server with 190,000 files and directories to scan, AVG took around 30-45 mins versus Maldet took just under 12 hrs !

Maldet and AVG both managed to detect malware on one of my clients' servers although AVG picked up a few email message I-Worm/Netsky, I-Worm/Bagle.BDE and HTML/Framer.FA in addition to what both reported for some PHP/backdoor/PHP Cmdshell hits.

p.s. AVG has debian and rpm and source downloads available at http://free.avg.com/us-en/download.prd-alf and was one of the first malware/anti-virus able to detect SSHD Rootkit exploit discussed at http://www.webhostingtalk.com/showthread.php?p=8569541#post8569541

 

[Biker]

p.s. AVG has debian and rpm and source downloads available at http://free.avg.com/us-en/download.prd-alf and was one of the first malware/anti-virus able to detect SSHD Rootkit exploit discussed at http://www.webhostingtalk.com/showthread.php?p=8569541#post8569541

That title needs to be changed as it was NOT an SSHD rootkit. In the early hours and days, it was thought it might be, but if you read that entire thread, you'll see that it wasn't related to SSHD at all.

 

[p4guru]

Yeah i know it turned out it wasn't just keeping it consistent with the thread linked

 

[TPerry]

+1 i see alot of these in CSF firewall logs as well but CSF firewall would of never allowed 700+ failed attempts i don't think unless they're all different ips ? Of course great case for changing default SSH port 22 to something else

Yeah, and normally fail2ban will stop it also (there were over 200 from two different Chinese IP's - they are in the drop list now). Fail2ban was not reading the auth.log and banning the ip's. Got that issue resolved and that's why you see them drop to almost zero after that point.