SYN Flood Attack

[Wesker]

We've been having ongoing issues with denial service attacks the past several months but it's been increasingly bad this quarter. We're currently running through cloudflare with our host who uses Voxility. This recent attack has taken down our site for a day now as they're bypassing cloudflare and targeting the server directly again which the host has been unable to mitigate. We're in early preparations to move to Amazon Web Services this week who should be able to handle these issues but looking for a short term solution here.

Here is what our host said:

Your site is getting flooded with TIME_WAIT connections due to which you are facing this issue.

=========================
tcp 0 0 Host IP 162.158.180.14:47854 TIME_WAIT -
tcp 0 0 Host IP 162.158.76.95:47986 TIME_WAIT -
tcp 0 0 Host IP 172.68.181.125:49782 TIME_WAIT -
tcp 0 0 Host IP 108.162.214.153:22456 TIME_WAIT -
tcp 0 0 Host IP 172.69.252.78:19248 TIME_WAIT -
tcp 0 0 Host IP 108.162.214.153:23526 TIME_WAIT -
tcp 0 0 Host IP 162.158.109.239:21678 TIME_WAIT -
tcp 0 0 Host IP 172.68.56.148:53920 TIME_WAIT -
tcp 0 0 Host IP 172.69.205.29:24222 TIME_WAIT -
tcp 0 0 Host IP 162.158.77.52:28032 TIME_WAIT -
tcp 0 0 Host IP 162.158.117.135:22960 TIME_WAIT -
tcp 0 0 Host IP 108.162.214.153:57376 TIME_WAIT -
tcp 0 0 Host IP 108.162.214.4:39844 TIME_WAIT -
tcp 0 0 Host IP 108.162.214.153:47406 TIME_WAIT -
tcp 0 0 Host IP 162.158.60.40:46680 TIME_WAIT -
tcp 0 0 Host IP 108.162.214.44:12720 TIME_WAIT -
tcp 0 0 Host IP 141.101.64.27:43852 TIME_WAIT -
tcp 0 0 Host IP 162.158.105.132:43786 TIME_WAIT -
tcp 0 0 Host IP 162.158.77.138:26552 TIME_WAIT -
tcp 0 0 Host IP 172.68.1.6:9586 TIME_WAIT -
tcp 0 0 Host IP 162.158.76.95:42454 TIME_WAIT -
tcp 0 0 Host IP 172.68.33.32:24582 TIME_WAIT -
tcp 0 0 Host IP 108.162.213.81:10732 TIME_WAIT -
tcp 0 0 Host IP 172.68.14.249:19978 TIME_WAIT -
tcp 0 0 Host IP 108.162.217.111:31954 TIME_WAIT -
tcp 0 0 Host IP 162.158.76.77:56756 TIME_WAIT -
tcp 0 0 Host IP 162.158.145.29:57796 TIME_WAIT -

Their are continuous 2041 connections to this server due to which you are receiving GATEWAY error. We would suggest you to null route your domain for few hours to let the attack switch to another IP or contact nginx administrator to check if they can do any thing in this regards.

======================================

So I'm seeing a ton of SYN_RECV

I'm not sure that this is the problem but this is well outside my knowledge. I was checking for a possible SYN flood and found that. I'm going to pass this over to Advanced Support to see if they know what it may be.

======================================

We have investigated your issue and found that their is attack on yoru server due to which you are facing this issue. Below are the error_logs related to same.

=====================
2020/04/19 14:03:54 [alert] 8841#0: *403355 socket() failed (24: Too many open files) while connecting to upstream, client: 49.248.139.244, server: website.com, request: "GET / HTTP/1.1", upstream: "https://Host IP:4443/", host: "www.website.com"
2020/04/19 14:30:08 [error] 14966#0: *2120 connect() failed (111: Connection refused) while connecting to upstream, client: 49.248.139.244, server: website.com, request: "GET / HTTP/1.1", upstream: "https://Host IP:4443/", host: "www.website.com", referrer: "https://www.website.com/"
2020/04/19 14:30:08 [error] 14966#0: *2155 connect() failed (111: Connection refused) while connecting to upstream, client: 49.248.139.244, server: website.com, request: "GET / HTTP/1.1", upstream: "https://Host IP:4443/", host: "www.website.com", referrer: "https://www.website.com/"
2020/04/19 14:30:08 [error] 14966#0: *2186 connect() failed (111: Connection refused) while connecting to upstream, client: 49.248.139.244, server: website.com, request: "GET / HTTP/1.1", upstream: "https://Host IP:4443/", host: "www.website.com", referrer: "https://www.website.com/"
2020/04/19 14:30:08 [error] 14966#0: *2210 connect() failed (111: Connection refused) while connecting to upstream, client: 49.248.139.244, server: website.com, request: "GET / HTTP/1.1", upstream: "https://Host IP:4443/", host: "www.website.com", referrer: "https://www.website.com/"
2020/04/19 14:30:09 [error] 14966#0: *2232 connect() failed (111: Connection refused) while connecting to upstream, client: 49.248.139.244, server: website.com, request: "GET / HTTP/1.1", upstream: "https://Host IP:4443/", host: "www.website.com", referrer: "https://www.website.com/"
2020/04/19 15:14:01 [error] 19187#0: *21886 connect() failed (111: Connection refused) while connecting to upstream, client: 49.248.139.244, server: website.com, request: "GET / HTTP/1.1", upstream: "https://Host IP:4443/", host: "www.website.com", referrer: "https://www.google.com/"
2020/04/19 15:14:01 [error] 19187#0: *21938 connect() failed (111: Connection refused) while connecting to upstream, client: 49.248.139.244, server: website.com, request: "GET / HTTP/1.1", upstream: "https://Host IP:4443/", host: "www.website.com"
2020/04/19 15:14:23 [error] 19190#0: *25506 connect() failed (111: Connection refused) while connecting to upstream, client: 49.248.139.244, server: website.com, request: "GET / HTTP/1.1", upstream: "https://Host IP:4443/", host: "www.website.com", referrer: "https://www.google.com/"
=====================

We have found following IP's hitting your server the most.
Hits IP adresss
10597 40.77.167.53
10732 194.156.251.172
10861 176.9.1.234
11038 93.123.176.197
11288 182.161.28.128
12490 212.112.122.167
12907 79.126.114.153
13745 84.237.190.136
16000 140.0.30.136
16466 71.75.80.198
18173 5.52.241.160
18410 64.83.188.148
19151 121.200.6.43
19347 212.34.30.92
31753 95.65.92.244
33648 94.54.21.22
42715 212.34.12.255
56861 5.112.67.207

We have blocked this IP address in server firewall still their are high SYN_SENT process on your server.

tcp 0 1 :58452 Host IP:4443 SYN_SENT 19185/nginx: worker
tcp 0 1 Host IP:58284 Host IP:4443 SYN_SENT 19186/nginx: worker
tcp 0 1 Host IP:59174 Host IP:4443 SYN_SENT 19183/nginx: worker
tcp 0 1 Host IP:57338 Host IP:4443 SYN_SENT 19185/nginx: worker
tcp 0 1 Host IP:57558 Host IP:4443 SYN_SENT 19186/nginx: worker
tcp 0 1 Host IP:58332 Host IP:4443 SYN_SENT 19181/nginx: worker
tcp 0 1 Host IP:56272 Host IP:4443 SYN_SENT 19181/nginx: worker
tcp 0 1 Host IP:59916 Host IP:4443 SYN_SENT 19187/nginx: worker
tcp 0 1 Host IP:59026 Host IP:4443 SYN_SENT 19187/nginx: worker
tcp 0 1 Host IP:59124 Host IP:4443 SYN_SENT 19186/nginx: worker
tcp 0 1 Host IP:58672 Host IP:4443 SYN_SENT 19183/nginx: worker
tcp 0 1 Host IP:56098 Host IP:4443 SYN_SENT 19186/nginx: worker
tcp 0 1 Host IP:57922 Host IP:4443 SYN_SENT 19186/nginx: worker
tcp 0 1 Host IP:59678 Host IP:4443 SYN_SENT 19183/nginx: worker
tcp 0 1 Host IP:58920 Host IP:4443 SYN_SENT 19183/nginx: worker
tcp 0 1 Host IP:56048 Host IP:4443 SYN_SENT 19181/nginx: worker
tcp 0 1 Host IP:57962 Host IP:4443 SYN_SENT 19190/nginx: worker
tcp 0 1 Host IP:58210 Host IP:4443 SYN_SENT 19183/nginx: worker
tcp 0 1 Host IP:57352 Host IP:4443 SYN_SENT 19187/nginx: worker
tcp 0 1 Host IP:56816 Host IP:4443 SYN_SENT 19181/nginx: worker
tcp 0 1 Host IP:56266 Host IP:4443 SYN_SENT 19181/nginx: worker

 

[Max Taxable]

I'd see how cool life is with better hosting. And if I ditch cloudflare.

 

[Wesker]

I'd see how cool life is with better hosting. And if I ditch cloudflare.

This is a longterm solution but we can't migrate overnight. Will take days to migrate. Why ditch cloudflare?

 

[JustinHawk]

Tried enabling TCP SYN cookie protection ?

 

[Xon]

My recommendation is get a new IP provisioned by your host. Disable your webserver (aka apache/nginx), then ensure your website isn't leaking the IP. Only then you can re-open.

By "Leak IP", this includes;

• Email headers if you are sending via local host, and some smtp relays
• Setting a proxy for Xenforo; via $config['http']['proxy'] for outbound untrusted http queries. Or disabling URL unfolding, title preview, image/link proxy, etc.
  • Proxy is easier than hunting down every possible option.
  • Run tinyproxy on a disposable $5/m linode/digitial ocean vm is quite easy. Bonus to use wireguard to establish a secure VPN between your webserver and the proxy server.
• Ensure your webserver is not listening to the open internet, and is only accepting IP's from CloudFlare at the firewall level.
• Ensure your webserver simply hangs up if an unknown http host is requested.
• Do not have public DNS entries referring to the server's direct ip.

Tried enabling TCP SYN cookie protection ?

As soon as you've got conntrack referenced in your firewall rules this isn't enough

 

[Wesker]

My recommendation is get a new IP provisioned by your host. Disable your webserver (aka apache/nginx), then ensure your website isn't leaking the IP. Only then you can re-open.

Correct. Actually was quite foolish I waited this long swapping the IP likely will short term fix the issue.

 

[Wesker]

Any other recommendations?

 

[Wesker]

enabling TCP SYN cookie protection

This all I need to do? - https://www.cyberciti.biz/faq/enable-tcp-syn-cookie-protection/

 

[JustinHawk]

This all I need to do? - https://www.cyberciti.biz/faq/enable-tcp-syn-cookie-protection/

Plesk Help Center

support.plesk.com

Yeah both are almost same, but plesk offered more variables to add to file about timeout etc, you may copy that too.

# Enable TCP SYN cookie protection

net.ipv4.tcp_syncookies = 1

# Decrease the time default value for tcp_fin_timeout connection

net.ipv4.tcp_fin_timeout = 3

# Turn off the tcp_window_scaling

net.ipv4.tcp_window_scaling = 0

# Turn off the tcp_sack

net.ipv4.tcp_sack = 0

But as Xon advised, you should think about IP changes.

 

[Brad Padgett]

Keep MX entries off your site if at all possible. If you require an MX entry for email then that's not much of a solution. A lot of times attackers get your ip address that way.

Installing mod_evasive which is an apache module can help too. I would do some research first though on it and see if it's something you want to install. You can install it with YUM package manager. I'm not sure if it will stop the issue but regardless I can guarantee that it would be a good option to check out. Mod_Evasive was basically designed to help with DDoS attacks so that's why I brought it up and suggested.

You don't need an MX record to send emails, only to receive them. So if you don't need to receive any emails then you should be good removing that from the cloudflare panel. If you see in the CF panel there's a warning about using them. It should only be used if absolutely required. I would use your new hosting provider Amazon for their email servers and have a separate email server setup with a different ip address when you switch web hosts if you require being able to receive emails. This way it will make it much more difficult for an attacker to get your ip address.

I hope that helped and good luck

 

[Wesker]

But as Xon advised, you should think about IP changes.

Working on rotating IPs as a short term solution until we patch everything.

 

[Wesker]

I recommend firewalling off your origin server so that only Cloudflare IPs - https://www.cloudflare.com/ips can connect to your origin server over HTTP/S ports and all other IPs are blocked.

Cloudflare also recommending this. Do you guys recommend cloudflare or an alternative?