This is the second time I am getting DDOS'd since i've moved hosts

JoyFreak

Well-known member
Before I moved hosts I never had any issues with DDOS. I am covered through CloudFlare. The last time it happened was a couple weeks ago and I managed to look into the IP that was flooding my host and challenged it through CF but this time round someone else is doing. They even messaged me saying "this is not a threat, just please upgrade your website security, i am doing this for the purpose of showing you how bad your ddos protection is. " this is their https://twitter.com/awfulise

Is this a server level issue or a CF issue? I've put my CF under attack mode for the time being but still down!
 
Go to your mail inbox, select any mail that was sent to you from your forum and check the properties of that mail too see if there is any IP leak.
 
This is a layer 7 DOS which Cloudflare is crap at stopping and just let the traffic through

Code:
# tail -f access.log
80.187.140.26 - - [25/Feb/2020:11:43:32 +0000] "POST /?91 </dev/null &>/dev/null & HTTP/1.1" 404 9461 "-" "Opera/9.80 (Windows NT 6.0) Presto/2.12.388 Version/12.14"
80.187.140.26 - - [25/Feb/2020:11:43:32 +0000] "POST /?58 </dev/null &>/dev/null & HTTP/1.1" 404 9461 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:25.0) Gecko/20100101 Firefox/25.0"
80.187.140.26 - - [25/Feb/2020:11:43:32 +0000] "POST /?97 </dev/null &>/dev/null & HTTP/1.1" 404 9461 "-" "Mozilla/5.0 (BlackBerry; U; BlackBerry 9850; en) AppleWebKit/534.11+ (KHTML, like Gecko) Version/7.0.0.254 Mobile Safari/534.11+"
80.187.140.26 - - [25/Feb/2020:11:43:32 +0000] "POST /?98 </dev/null &>/dev/null & HTTP/1.1" 404 9461 "-" "Mozilla/5.0 (BlackBerry; U; BlackBerry 9850; en) AppleWebKit/534.11+ (KHTML, like Gecko) Version/7.0.0.254 Mobile Safari/534.11+"
80.187.140.26 - - [25/Feb/2020:11:43:32 +0000] "POST /?13 </dev/null &>/dev/null & HTTP/1.1" 404 9461 "-" "Opera/9.80 (Windows NT 5.1; U; zh-sg) Presto/2.9.181 Version/12.00"
80.187.140.26 - - [25/Feb/2020:11:43:32 +0000] "POST /?91 </dev/null &>/dev/null & HTTP/1.1" 404 9461 "-" "Mozilla/5.0 (BlackBerry; U; BlackBerry 9850; en) AppleWebKit/534.11+ (KHTML, like Gecko) Version/7.0.0.254 Mobile Safari/534.11+"
80.187.140.26 - - [25/Feb/2020:11:43:32 +0000] "POST /?88 </dev/null &>/dev/null & HTTP/1.1" 404 9488 "-" "Mozilla/5.0 (BlackBerry; U; BlackBerry 9850; en) AppleWebKit/534.11+ (KHTML, like Gecko) Version/7.0.0.254 Mobile Safari/534.11+"
80.187.140.26 - - [25/Feb/2020:11:43:32 +0000] "POST /?85 </dev/null &>/dev/null & HTTP/1.1" 404 9488 "-" "Opera/9.80 (Windows NT 6.0) Presto/2.12.388 Version/12.14"
80.187.140.26 - - [25/Feb/2020:11:43:32 +0000] "POST /?28 </dev/null &>/dev/null & HTTP/1.1" 404 9488 "-" "Opera/9.80 (Windows NT 5.1; U; zh-sg) Presto/2.9.181 Version/12.00"
80.187.140.26 - - [25/Feb/2020:11:43:32 +0000] "POST /?60 </dev/null &>/dev/null & HTTP/1.1" 404 9488 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:25.0) Gecko/20100101 Firefox/25.0"
 
Disable URL unfurling it's leaking your server IP.

 
Disable URL unfurling it's leaking your server IP.

Or, probably more importantly, make sure that only CF can communicate with it...
 
What kind of requests are they sending you?

You can enable rate limiting at the HTTPD level, or use Cloudflare's rate limiting but that'll cost extra. There's other stuff you can do like integrating fail2ban with Cloudflare's API/firewall.

Or, probably more importantly, make sure that only CF can communicate with it...
Whilst you should definitely setup a firewall so only Cloudflare can connect via 80/443, a firewall isn't going to help when your pipe is saturated with DDoS traffic. Preventing attackers from finding your server's IP in the first place is the most important thing.
 
CloudFlare is absolutely crappy at dealing with these types of attacks. They just let the attacks through while its clearly DDoS traffic. I get hit several times a day with some days on and off for the last 1.5 year. Last night I got hit by a 230 million request. CloudFlare Rate limiting is useless. Under Attack Mode doesn't touch it. Blocked IPs and ASNs just keep hitting the server. Its ridiculous. CloudFlare is not completely useless, but they simply let a lot trough which they shouldn't.
 
CloudFlare Rate limiting is useless. Under Attack Mode doesn't touch it. Blocked IPs and ASNs just keep hitting the server.
Based on the blocked IPs/ASNs being let through, I'm guessing they're hitting your IP directly. I've never had any issues with Cloudflare's firewall, if they're not hitting your IP directly then open a ticket with them.
 
They are not hitting the server directly. The server doesn't accept any connections except cloudflare. I have a ticket with dozens of replies open with them since January 19th and am getting mostly bot-type template responses or responses from staff that does not look at the attacks. Just recently I have gotten somewhat helpful responses and I am happy that they at least promised to refund the charge for 16 million 'legitimate requests' that they let through during a DDoS attack.
 
This is a layer 7 DOS which Cloudflare is crap at stopping and just let the traffic through
Yeah Cloudflare layer 7 attacks are much harder to deal with just Cloudflare alone but CF still useful for volumetric attacks which are more costly. So as you had experienced you need to also deal with it on origin server side which is expected for layer 7 attacks i.e. rate limiting at origin server level. It's a must have :)

Cloudflare alone isn't enough and no automated way to deal with it as layer 7 attacks are application level attacks and Cloudflare has no way of automatically knowing what your application is and whether it's a legit request/traffic type for your application. You'd have to tell Cloudflare what is legit or not via CF WAF/Firewall Rules or custom CF Worker based logic. But Cloudflare isn't useless, as there are other DDOS attacks at Layer 1-6 which can be even more costly to defend against where Cloudflare helps for those :)

Cloudflare even states it's difficult to defend against Layer 7 attacks https://www.cloudflare.com/en-au/learning/ddos/what-is-a-ddos-attack/
Application Layer Attacks
The Goal of the Attack:
Sometimes referred to as a layer 7 DDoS attack (in reference to the 7th layer of the OSI model), the goal of these attacks is to exhaust the resources of the target. The attacks target the layer where web pages are generated on the server and delivered in response to HTTP requests. A single HTTP request is cheap to execute on the client side, and can be expensive for the target server to respond to as the server often must load multiple files and run database queries in order to create a web page. Layer 7 attacks are difficult to defend as the traffic can be difficult to flag as malicious.
 
Last edited:
Hmm, not completely useless. Just curious: Does this mean you (finally) came to the conclusion that CF is partly/mostly useless?
Once you learn to analyze the cloudflare data and use firewall rules, zone lockdowns, managed rules you get significant improvements. But its far from bulletproof. The site still goes down from layer7.

As you keep blocking high volumes of suspect traffic chances are that legitimate users get banned in the process.

And there is no support for XenForo like for example phpBB, WordPress, Magento, Joomla, Drupal have special OWASP & XSS rule sets and integration. For example for whitelisting usergroups, seeing which user gets blocked or blocking attacks without making the site unusable. There is nothing for XenForo. @eva2000 do you foresee any improvement on this point?

rate limiting at origin server level. It's a must have
What I really liked in LiteSpeed Web Server was the security settings and the rate limiting. Its great that centminmod has a similar rate limiting feature. Is it possible to set multiple limits like in LSWS? i.e. set limits per second, per minute, per X minutes. So if the user hits any of the set limits (plural) they get IP banned?
 
@eva2000 do you foresee any improvement on this point?
Unfortunately not. My Cloudflare contact in WAF department offered to have look at Xenforo WAF rules but needed a copy of Xenforo/license and XF staff I asked, just asked my CF contact to email XF folks via https://xenforo.com/contact/ to discuss it. Never heard anything back as to how that went.
What I really liked in LiteSpeed Web Server was the security settings and the rate limiting. Its great that centminmod has a similar rate limiting feature. Is it possible to set multiple limits like in LSWS? i.e. set limits per second, per minute, per X minutes. So if the user hits any of the set limits (plural) they get IP banned?
Not Centmin Mod specific, but Nginx in general is flexible can do rate limiting per route/location path etc recently Centmin Mod member asked at https://community.centminmod.com/threads/nginx-rate-limiting-only-on-get-requests.19206/ but I haven't really tried overlapping rate limits. But for that you'd need fail2ban combined with Nginx or some Nginx lua or Nginx njs scripting if you want to ban rather than rate limit.

for rate limiting see
Centmin Mod out of box doesn't set rate limiting up but end users can do so. For example of Centmin Mod forums, I have different rate limit settings for different route paths i.e. search, find-new, profile search have a different rate limit to rest of the forums. Cloudflare firewall rules I have setup to only allow legit Xenforo search, profile search, media search query arguments as well. Non-legit will end up blocked.

Only time Centmin Mod automatically sets up rate limiting is with it's Wordpress auto installer for better out of box Wordpress security see https://community.centminmod.com/th...l-vs-centmin-sh-menu-option-22-install.15435/
 
But for that you'd need fail2ban combined with Nginx or some Nginx lua or Nginx njs scripting if you want to ban rather than rate limit.
Do you mean that for banning I'd need a custom script? I'm not familiar lua or njs. Is this possible to do in conjunction with cloudflare as the firewall sees the IP of Cloudflare?
 
Do you mean that for banning I'd need a custom script? I'm not familiar lua or njs. Is this possible to do in conjunction with cloudflare as the firewall sees the IP of Cloudflare?
There's a few options -
  • Use Cloudflare's API to block IPs at the Cloudflare level (limited to 50k IPs), this would be by far the best option
  • Use iptables' string matching to block based on the x-forwarded-for header (would have to disable HTTPS on your origin)
  • Deny the IPs with nginx or whatever httpd you're using
Any of these options can be setup with fail2ban and the actionban command.
 
Back
Top Bottom