How Badlock Was Discovered and Fixed

Severity analysis of vulnerabilities by experts from the information security industry is rarely based on real code review. In the ‘Badlock’ case, most read our CVE descriptions and built up a score representing a risk this CVE poses to a user. There is nothing wrong with this approach if it is done correctly.  CVEs are analyzed in isolation; as if no other issue exists. In the case of a ‘Badlock‘ there were eight CVEs. The difference is the fact that one of them was in a foundational component used by most of the code affected by the remaining seven CVEs. That very specific CVE was marked CVE-2015-5370.

CVE-2015-5370 is a set of issues in how Samba implements the DCE RPC protocol specification. DCE RPC allows a connection to be created and to associate various resources with it. Multiple remote procedure calls may be performed while accessing resources in the same association group. The specification assumes that association adds a security and presentation context negotiation to the underlying connection. If the context negotiation is not protected, a man in the middle attacker could present the same security blob over a different connection and switch to use the same association group, including resources already opened by the original connection.

The event that triggered this finding was an arrival of a single network packet from the Defensics tool produced by the Codenomicon company (acquired by Synopsys in 2015). Codenomicon’s fuzzer generated this particular packet at the Microsoft IOLab event in June 2015 where vendors tested DCE RPC-based solutions against each other. The Samba Team and Red Hat were there, testing Samba and FreeIPA against various Active Directory and Windows Server versions.  Codenomicon folks had fun poking everyone’s products to find really bizarre bugs.

CVE-2015-5370 is a Samba issue, also allowing a DCE RPC server to cause an overallocation of a memory at the client side and a subsequent crash. The conditions are hard to meet, to be fair, but not impossible. The context is what matters here. The ‘client’ here could be an smbd or winbindd daemon talking to another DCE RPC server. Such clients are typical for Samba serving as a domain member in an Active Directory domain or in a traditional NT4-style domain. A winbindd daemon would use a channel connection to talk to its Domain Controller to ask it for a user or group resolution, or other operations.  If this connection could be taken over by the man in the middle, it could cause the winbindd daemon on the domain member to crash and, perhaps, execute remotely supplied code.

The winbindd daemon runs under root privileges. A remote code execution means take over the server in question. Even if the domain member’s integrity is not compromised, a vulnerable implementation could be used to take over the secure channel on that domain member. This gives an intruder the privilege to talk to a Domain Controller with the credentials of a machine enrolled into the domain. It opens a number of possible vectors for an attack: setting up a honey pot to wait until domain administrator logs in one of them.

While CVE-2015-5370 is a Samba issue, its analysis led Jeremy Allison and Stefan Metzmacher of the Samba Team to a thought experiment that the concept of downgrading the security of the connection and accessing associated resources opened by a valid user might apply to other DCE RPC implementations. Stefan was quick in writing reproducer code to prove that the Microsoft Active Directory implementation is vulnerable to the downgrade attack as well. A security team at Microsoft was contacted and the long road to the coordinated security release had started to fix what is now known as CVE-2016-2118 for Samba and CVE-2016-0128 for Windows.

It took almost ten months to go through all the code in Samba. Stefan Metzmacher was able to find other inconsistent areas where integrity was not enforced. They resulted in a number of separate CVEs included in the ‘Badlock’ announcement.

CVE-2016-2118 is a downgrade attack to force authentication level of DCE RPC connection from SIGN or SEAL (integrity or encryption) to CONNECT (no integrity or encryption). An attacker still needs to be authenticated, and this is how CVE-2016-2118 differs from CVE-2015-5370, but authentication type (Kerberos or NTLMSSP) or application protocol do not matter here. A man in the middle can switch all of them back to a method where none of the messages would be checked for integrity.

The code was rewritten; in March 2016 the changes needed to fix all eight CVEs amounted to about 200 individual patches against a development version of Samba, with about half of those responsible for fixing CVE-2015-5370. When backported to previous stable Samba versions, they needed additional hundred patches. To oldest supported Samba version — about four hundred patches. What started as an individual snowflake became an avalanche but it wasn’t finished yet.

The Samba Team has its own program to maintain relationships with vendors who ship products based on Samba. Individual security specialists from the companies and projects are given early access to the security issues currently being addressed by the Samba Team. The process to validate bugs as security- sensitive may take a time but once most of the work is done, vendors are included in a stream of information exchange about the vulnerability and have access to the code changes required to fix it.

A vendor that ships Samba in its product has a lot of obligations towards their own customers. In many cases the version shipped lags behind the Samba upstream development. This is understandable, as vendors have own focus and subset of features they are interested in. Sometimes this leads to a point where such versions are way beyond what is supported by the Samba Team in the upstream project. Due to complexity and lack of resources to work on old versions, the Samba Team only concentrates on two to three major releases at a time. Samba 4.4 is the current major release, with Samba 4.3 and 4.2 being the two other supported branches. Anything prior to this is handled by Samba vendors, if possible.

Many vendors did not yet move to Samba 4.x. They ship Samba 3.6, 3.4, and even 3.0. Sometimes it is part of their product line with a long term support requirement. For example, Red Hat continues to support Samba 3.0 in Red Hat Enterprise Linux 4 Extended Life Cycle Support until March 31st 2017 and in Red Hat Enterprise Linux 5 Extended Life Cycle Support until November 30th 2020. A number of other vendors have similar policies for their products. If the Samba Team does not support this code anymore, it means the work needs to be done by someone else.

After an extensive analysis and discussion between the Samba Team, Red Hat, SUSE, and SerNet developers it was decided to share the work and provide a common set of patches to older versions. Backport to Samba 3.6 took several weeks, done by Andreas Schneider of Red Hat and Ralph Böhme of SerNet, with the help of Günther Deschner of Red Hat and Aurélien Aptel from SUSE. Based on their work, Simo Sorce of Red Hat did majority of the backport to Samba 3.0. In parallel, SUSE engineers continued to work in Samba 3.4 backports, while SerNet developers provided a backport to Samba 4.0 file server component. An industry-wide effort also allowed to have more rigorous review process which was very important given the complexity of the issues at hand.

These patches are available now to everyone at Samba.org. Below are statistics that show depth of the vulnerabilities:

samba-4.4.0-security-2016-04-12-final.patch
227 files changed, 14582 insertions(+), 5037 deletions(-)

samba-4.3.6-security-2016-04-12-final.patch
236 files changed, 14870 insertions(+), 5195 deletions(-)

samba-4.2.9-security-2016-04-12-final.patch
319 files changed, 17636 insertions(+), 7506 deletions(-)

samba-v4-0-security-2016-04-12-fileserver-only.patch
70 files changed, 3145 insertions(+), 540 deletions(-)

samba-v3-6-security-2016-04-12.patch
95 files changed, 4007 insertions(+), 978 deletions(-)

Backports for Samba 3.0 are complicated. Some of the CVEs cannot be fixed there at all. For example, Samba 3.0 includes its own implementation of SASL support for LDAP client operations. This implementation only supports authentication and not signing or sealing operations. Once authentication is done, no information is retained to be able to sign or encrypt (seal) outgoing LDAP packets. Thus, CVE-2016-2112 cannot be fixed in Samba 3.0 without additional non-trivial development.

In addition to that, Samba 3.0 has hand-written code to operate on network packets. It is very different from the auto-generated code in Samba 3.6 and above, so strictly speaking, CVE-2015-5370 does not apply to it. It does not mean the hand-written code is not vulnerable to similar attacks, though they would warrant a different CVE.

On the other hand, Samba 3.0 is so old that CVE-2016-2118 does not apply there. As it is an attack on SIGN or SEAL authentication level which allows to downgrade to CONNECT authentication level, Samba 3.0 RPC server code is lucky that it does not implement support for CONNECT authentication level. With “client signing = required” option set in the configuration file, the attack of CVE-2016-2115 is mitigated.

Another useful part of the industry collaboration was ability to pool quality engineering resources. Red Hat was able to find several regressions in the code that affected some of key use cases for our customers. SerNet was able to demonstrate a regression against older versions of Windows Server. All the findings were shared with the other vendors.

In addition to the coordination with Samba vendors, non-Samba using vendors who also create product implementing SMB file services which by necessity have to implement DCE RPC services also had to be notified in advance of the release so they could evaluate their own products for problems. This included Apple, EMC, NetApp, Oracle, Nexenta and Huawei. Jeremy Allison of the Samba Team and Google Inc. handled this coordination. All in all this was a large, multi-vendor collaborative effort to ensure customers were kept safe from the severity of this bug.

Red Hat’s own challenge was also to deploy changes to different support channels. Overall, 16 advisories were created, affecting four major Red Hat Enterprise Linux, Red Hat Gluster Storage, and Red Hat Enterprise Linux Atomic Host product lines. Through these advisories Samba 3.0, Samba 3.6, Samba 4.2 and associated libraries, as well as other affected components were delivered to a dozen support channels. The packages were ready for the push 10 minutes prior to the official embargo lift off time.

The story doesn’t end here either. Deployments by actual customers will take time. In busy enterprises finding a maintenance window at will isn’t easy or even possible. When SerNet made a marketing announcement for ‘Badlock’ vulnerabilities, one of their motives was to make sure a priority could be carved out for ‘Badlock’ updates. While the hype caused top management in many companies to turn their focus on the problem – this wouldn’t be needed if more serious attention to security were already in place.
 
Thoughts or questions on how ‘Badlock’ was discovered and fixed?  Feel free to reach out using the comments section (below).

  1. Thanks for writing about this, I don’t think many people realize why this was so much more complicated than your average buffer overflow CVE.

    Stefan’s last name is misspelled, it is Metzmacher rather than Metzemacher.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s