CloudBleed HTTPS traffic leak

jOOc

Active member
Between 2016-09-22 - 2017-02-18 passwords, private messages, API keys, and other sensitive data were leaked by Cloudflare to random requesters. Data was cached by search engines, and may have been collected by random adversaries over the past few months.

https://github.com/pirate/sites-using-cloudflare (xenforo.com is on the list)
https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/

https://www.theregister.co.uk/2017/02/24/cloudbleed_buffer_overflow_bug_spaffs_personal_data/
'This leak was triggered when webpages had a particular combination of unbalanced HTML tags, which confused Cloudflare's proxy servers and caused them to spit out data belonging to other people – even if that data was protected by HTTPS.'

#Cloudbleed
 
While I've always understood the theory behind the cloud, I (and others that have a lot of experience with the internet) would never place any of a site's data on the cloud for this very reason. You are placing data in the hands of someone you have no control over.
 
Suffice it to say that I'm pretty dissatisfied with trying to mass reset passwords using XenForo. I've tried three different addons and one of them work properly.
  • DragonByte Security (Paid, $25) - Claims to reset people's passwords, but fails to iterate through all users. Reset a good hundred or so before breaking.
  • Force Password Change (Free) - Doesn't require a reset. Causes people to have to enter their previous password to change it if logged in (not what I want at all, could have been leaked). Users can still login using an old password; they're just required to change it.
  • Reset Password From ACP (Free) - Doesn't bulk reset passwords. Conspicuously built by a XenForo Developer, though, and he's explicitly said it won't bulk reset passwords (boo).
Not sure why this isn't just a button in the ACP by default, but I'm still searching for ideas if anyone has any.

Edit: Hilariously, this affects XenForo too.
 
Last edited:
This only affected certain customers. I have had an email stating we are clear and nothin to worry about
 
This only affected certain customers. I have had an email stating we are clear and nothin to worry about

That email -- posted in full below, only states that 150 sites were found in the caches. That doesn't mean that this vulnerability wasn't discovered prior and data mined to oblivion, nor does it mean you weren't affected. You should assume any CloudFlare hosted site is vulnerable if you're using their reverse proxy services. Just because you weren't found in the breach doesn't mean your data wasn't leaked somewhere and isn't being stored on some random hard drive.

Dear Cloudflare Customer:

Thursday afternoon, we published a blog post describing a memory leak caused by a serious bug that impacted Cloudflare's systems. If you haven't yet, I encourage you to read that post on the bug:

https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/

While we resolved the bug within hours of it being reported to us, there was an ongoing risk that some of our customers' sensitive information could still be available through third party caches, such as the Google search cache.

Over the last week, we've worked with these caches to discover what customers may have had sensitive information exposed and ensure that the caches are purged. We waited to disclose the bug publicly until after these caches could be cleared in order to mitigate the ability of malicious individuals to exploit any exposed data.

In our review of these third party caches, we discovered data that had been exposed from approximately 150 of Cloudflare's customers across our Free, Pro, Business, and Enterprise plans. We have reached out to these customers directly to provide them with a copy of the data that was exposed, help them understand its impact, and help them mitigate that impact.

Fortunately, your domain is not one of the domains where we have discovered exposed data in any third party caches. The bug has been patched so it is no longer leaking data. However, we continue to work with these caches to review their records and help them purge any exposed data we find. If we discover any data leaked about your domains during this search, we will reach out to you directly and provide you full details of what we have found.

To date, we have yet to find any instance of the bug being exploited, but we recommend if you are concerned that you invalidate and reissue any persistent secrets, such as long lived session identifiers, tokens or keys. Due to the nature of the bug, customer SSL keys were not exposed and do not need to be rotated.

Again, if we discover new information that impacts you, we will reach out to you directly. In the meantime, if you have any questions or concerns, please don’t hesitate to reach out.

Matthew Prince
Cloudflare, Inc.
Co-founder and CEO
 
Two of my sites were affected by this and a third closed one.

Noticed a few big names on this list including @digitalpoint (Tagged to give heads up)

Cloudflare needs to do something like maybe give free protection for a month or something. Over 4 million sites.

Bit that annoys me is the site called crimeflare saying everyone who uses the service is a pirate.
 
That email -- posted in full below, only states that 150 sites were found in the caches. That doesn't mean that this vulnerability wasn't discovered prior and data mined to oblivion, nor does it mean you weren't affected. You should assume any CloudFlare hosted site is vulnerable if you're using their reverse proxy services. Just because you weren't found in the breach doesn't mean your data wasn't leaked somewhere and isn't being stored on some random hard drive.

Dear Cloudflare Customer:

Thursday afternoon, we published a blog post describing a memory leak caused by a serious bug that impacted Cloudflare's systems. If you haven't yet, I encourage you to read that post on the bug:

https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/

While we resolved the bug within hours of it being reported to us, there was an ongoing risk that some of our customers' sensitive information could still be available through third party caches, such as the Google search cache.

Over the last week, we've worked with these caches to discover what customers may have had sensitive information exposed and ensure that the caches are purged. We waited to disclose the bug publicly until after these caches could be cleared in order to mitigate the ability of malicious individuals to exploit any exposed data.

In our review of these third party caches, we discovered data that had been exposed from approximately 150 of Cloudflare's customers across our Free, Pro, Business, and Enterprise plans. We have reached out to these customers directly to provide them with a copy of the data that was exposed, help them understand its impact, and help them mitigate that impact.

Fortunately, your domain is not one of the domains where we have discovered exposed data in any third party caches. The bug has been patched so it is no longer leaking data. However, we continue to work with these caches to review their records and help them purge any exposed data we find. If we discover any data leaked about your domains during this search, we will reach out to you directly and provide you full details of what we have found.

To date, we have yet to find any instance of the bug being exploited, but we recommend if you are concerned that you invalidate and reissue any persistent secrets, such as long lived session identifiers, tokens or keys. Due to the nature of the bug, customer SSL keys were not exposed and do not need to be rotated.

Again, if we discover new information that impacts you, we will reach out to you directly. In the meantime, if you have any questions or concerns, please don’t hesitate to reach out.

Matthew Prince
Cloudflare, Inc.
Co-founder and CEO
 
And even then you would have had to have mucked HTML source to begin with among a whole bunch of other things going on to make it a perfect storm.

People need to read the whole disclosure article.

Those 770 unique URIs covered 161 unique domains.

Your HTML source needed to be less than 4k, have specific Cloudflare features enabled and most importantly, your HTML source needed to include mucked HTML tags.

Even then Cloudflare can't leak anything from memory it doesn't have. If your site is sending passwords or API credentials to CF (to relay to end users), you have major design issues with your site to begin with. 99.99% of the stuff search caches would store from memory leak is the guest cookie given to the search engine.

Again, it affected approximately 150 Cloudflare customers and CF has already contacted every affected customer directly about it.
 
Not sure if folks posted this reveals more info at https://bugs.chromium.org/p/project-zero/issues/detail?id=1139

Cloudflare did finally send me a draft. It contains an excellent postmortem, but severely downplays the risk to customers.

Cloudflare sent a list of sites with more PII, but the list contained many well-known domains that were not using cloudflare (e.g. blogspot).

I let them know, and then tried to manually clean their list of obviously invalid requests to avoid any further delays. I filtered the remaining requests through a quick script I wrote to verify they were reasonable, which removed another 20 incorrect entries.

I'm sure this was a good faith mistake from a team rushing to protect users before publication today.
 
I'm only using CF for DNS and that's the only thing running through there. Should I be worried about this?
If your site is XenForo, no... unless you injected invalid HTML into your templates and you modified XenForo enough to be sending users private info like passwords or anything else via HTTP headers, no.

I think people are forgetting it's Cloudflare's servers with the memory issue, not your web server. Meaning nothing could possibly be exposed beyond stuff your web server sends to the web browser client normally (routes through Cloudflare). For a normal XenForo that had mucked HTML inserted into it, the worst case scenario would be the guest session cookie being sent to the search engine spider. Authentication credentials are not used (and definitely not sent to) the spider. Since your web server never sent anything of any interest through Cloudflare, a memory leak on Cloudflare's side can't expose anything you didn't send to them to begin with.

My guess is that the only sites affected were ones that have all sorts of underlying design issues... not just invalid HTML tags, but I can't think of any legitimate use case where a website should be sending credentials to an unauthenticated user (search engine spider) or sending users things like API keys within the HTTP request (again, the only way Cloudflare could get it in order to expose it via their memory leak).
 
Thanks @digitalpoint I myself haven't done anything to templates and stuff like that to my site other than what addons may edit/add but I only use addons from reputable developers. My theme was professionally designed by @ThemeHouse and I doubt they would have added anything that would expose anything. So I'm gonna say I should be good then.
 
If you were using DNS services only from what I read you would not be affected by this even with malformed html since it wasn't being run through the problematic code on their end.
 
Back
Top Bottom