Change All Your Passwords, Right Now!

by Steinthor Bjarnason, Senior ASERT Security Analyst and Roland Dobbins, ASERT Principal Engineer

CloudFlare are probably best known as a DDoS mitigation service provider, but they also operate one of the largest Content Delivery Networks (CDNs) on the Internet. Many popular Web sites, mobile apps, etc. make use of the CloudFlare CDN, which hosts content associated with more than 4 million DNS domains, worldwide.

CloudFlare have developed customized software which allows their CDN to service a huge amount of Web requests every second of every day. It’s a complex, highly-scalable system, and CloudFlare are continually working to enhance its functionality and scalability in order to meet the needs of their customers.

Unfortunately, it appears that there was a serious security vulnerability in one of the software components used in CloudFlare’s CDN system which was exposed as CloudFlare were making some alternations to the service late last year. And as a result, it appears that a great deal of information related to CloudFlare-powered Web sites and applications - potentially including usernames, passwords, chat sessions, and other forms of personally-identifying information (PII) - has inadvertently leaked from CloudFlare and has been cached by search engines such as Google, Bing, DuckDuckGo and Baidu; privately-run ‘Web scrapers’; and similar services.

That means that this heretofore-private information for millions of Internet users - again, potentially including access credentials - has been available to anyone to stumble across, and possibly exploit.

And it doesn’t really matter whether that information was encrypted via SSL/TLS or not.

How Did This Happen?

Whenever one of the CloudFlare CDN reverse-proxies receives an SSL/TLS request, it has to decrypt the request and perform some processing in order to determine how best to service the request. The results of this processing would normally be stored in the operating RAM of the reverse-proxies and then forgotten about (not deleted) as the system moved on to service subsequent requests.  A bug in a component of the CloudFlare CDN service stack resulted in some users not only receiving the results they were expecting, but also random memory contents from the CloudFlare reverse-proxies - which could contain everything from other users requests and responses to sensitive information like decrypted passwords and usernames.

Basically, if user A accessed content from server X, user B could, in addition to the expected results from server Y, see what user A got in his responses from server X.  Normally, this wouldn’t be an issue as Web browsers would ignore the additional content, and user B would only see his expected results.

However, to improve performance and for purposes of statistical analysis, almost all of the big Web search providers like Google, Bing, Baidu, et. al. will cache and store information on almost all Web servers in the world.  This means that when user B did made requests and inadvertently received potentially sensitive information related to user A, the results were often stored in the caches of these popular search engines - and possibly in privately-run Web crawlers, too.

This resulted in the CloudFlare CDN inadvertently leaking random user session information which was subsequently stored by Google and other search providers.  This information is readily searchable, and is relatively easy to locate using simple search tools.  Google has been actively working with CloudFlare to find this kind of inadvertently-leaked-and-cached sensitive content and are actively working to purge it from the Google cache. It is assumed that other prominent search engine operators are working with CloudFlare to do this on their platforms, as well.

What Does This Mean to Me?

As noted previously, many of the most popular Web sites, mobile apps, and related services make use of the CloudFlare CDN service. A programmatically-generated list of the domains for these sites and services may be found here. This handy Chrome Web browser plugin will also alert users when they’re browsing a CloudFlare-powered site, and this Web-based utility is also very useful.

Chances are, you’ve used one or more of the sites, services, and/or apps serviced by the CloudFlare CDN, whether you’ve accessed them directly, or whether they’re a dependency of some other site, service, or app you’ve used. Because of the popularity of the CloudFlare CDN service, it’s for all practical purposes impossible for most Internet users to determine whether or not their Web browsing or app traffic has been served by the CloudFlare CDN.

And that’s where the problem lies.

Change Them All, and Change Them Now!

For most of us, the only truly safe response to this large-scale information leak is to update our passwords for the Web sites and app-related services we use every day. Pretty much all of them.

Google, Apple, and Microsoft do not appear to utilize the CloudFlare CDN - but unless one has the time, energy, and available to do a detailed analysis of one’s historical Web surfing and app usage over the last several months, it’s better to do a password rotation for all other accounts, just to be on the safe side.

And keep in mind that other types of information may’ve leaked as well - everything from chat logs to email contents to financial information and Web browsing habits. So, while it’s always a good idea to keep a weather eye on one’s online information, increased vigilance in the wake of this mass information leakage is warranted.

A Rapid Response to a Big, Complex Problem

Google security researcher Tavis Ormandy of Google’s Project Zero team initially identified this issue and contacted CloudFlare in short order. CloudFlare responded rapidly, and have been working with Google and other organizations to remove as much of the leaked information as possible from Web search engine caches, and have provided a public explanation of how this incident occurred, and the steps they’ve taken to both remediate it and ensure that it doesn’t happen again.

While it’s unfortunate that events of this nature take place, both Google Project Zero and CloudFlare themselves should be commended for taking quick action to resolve the issue to the best of their ability, and in publicly discussing what took place so that other security researchers - and the Internet community at large - can make their own assessments of the potential impact. It is our hope that more organizations will learn from their example, and move to both remediate and disclose pertinent details of security incidents in a timely and forthright manner.

A Special Note for Arbor Cloud Customers

We expect to receive some inquiries as to whether it’s possible that a problem of this sort could arise for sites and services protected by our own Arbor Cloud DDoS mitigation service.

Because Arbor Cloud is focused strictly on DDoS mitigation and is not a CDN service, it doesn’t perform the same types of operation on customer content and application traffic as does a CDN service like CloudFlare’s. So, this type of issue doesn’t apply to the Arbor Cloud service, nor to Arbor Cloud customers.