Fork me on GitHub

BetterTLS

What is BetterTLS?

BetterTLS is a collection of test suites for TLS clients. At the moment, two test suites have been implemented. One tests a client's validation of the Name Constraints certificate extension. This extension is placed on CA certificates which restrict the DNS/IP space for which the CA (or sub-CAs) can issue certificates. The other test suite evaluates a TLS client's ability to discover a valid certificate path (a certificate "chain") from an unordered collection of certificates.

Why BetterTLS?

The BetterTLS project was originally created to ensure that we could create a name-constrained CA that could not be abused. We were only able to gain this confidence by evaluating whether clients (such as popular web browsers) were correctly handling the name constraints extension.

We later added an additional test suite for certificate path building because of the recurring pains we experienced during transitions in the web PKI ecosystem, such as intermediate CA expiry and signature algorithm deprecation. The goal is to help TLS implementations identify and verify changes that could alleviate these pain points and to help application developers pick TLS implementations that won't break during these transitions.

Why "BetterTLS"?

This project is inspired in part by badssl.com. In a similar vein, we want to highlight some of the issues and difficulties we've found with HTTPS/TLS implementations so that they can be corrected. By doing so, we hope to make TLS better for everyone.

Test Results and Discussion

This site contains archived test results. These test results were from the latest version of TLS implementations at the time they were run, though some are quite old now. For discussions and interpretation of these results, you should check out our blog post about Name Constraints and the follow-up blog post about path building.

Testing Clients

Clients and browsers can be tested using the code in the BetterTLS repository. Check out the README for examples on how to do so.

Contribute

BetterTLS is open source software. We encourage you to help add more tests, fix any issues you find in the existing test suites, or even fork it for use in your own software project. Pull requests are welcome!

Update History

Dec 27, 2021

This release includes a refactor of the BetterTLS test suites and website. The latest test results were regenerated with new test executor, including execution of the Name Constraints test suite. Path building results are consistent with earlier results discussed in the blog post. Notable name constraints results include:
  • curl 7.68.0 (linked against OpenSSL 1.1.1f) has a IP name constraints bypass when IP is included as a common name in the leaf certificate.
  • A number of implementations appear to exhibit "false positive" errors; i.e. rejecting certificates that are valid. (See the recent archived results for Edge and OpenSSL). This appears to be happening when an IP is included in the subject CN and an upstream CA has a DNS name constraints extension. Presumably, to be defensive against hostname verification that uses the subject CN, the certificate verification is matching the checking the name constraint extension against the CN even if it is an IP address. That said, it seems reasonable to lean into this defensiveness since it's unlikely to cause issues in practice.

Oct 14, 2021

Release of the path building test suite. Discussion and results summary available on the Netflix Tech Blog.

Apr 10, 2017

Initial release. Discussion available on the Netflix Tech Blog.

What is BetterTLS: Name Constraints?

The Name Constraints test suite is a collection of tests for HTTPS clients implementing verification of the Name Constraints certificate extension. This extension is placed on CA certificates which restricts the DNS/IP space for which the CA (or sub-CAs) can issue certificates. One of the most common use-cases is for CAs to cross-certify another CA, but for a limited namespace. For example, Acme Corp. could sign a certificate for an intermediary CA but include a Name Constraints extension which blacklists the intermediate CA from signing certificates for acme.com.

Another use-case for the Name Constraints extension (which motivated this project) is as a way to reduce the risk exposure of trusting an internal CA. For example, we could create an internal CA which includes a Name Constraints extension with a whitelist of internal.example.com and 192.168.0.0/16. A user who imports this CA into her browser's truststore can then be confident that this certificate will only be used for internal company pages and IPs and cannot be used to intercept secure communications with external websites.

However, the Name Constraints extension only provides these assurances if we can be confident that HTTPS clients properly enforce them...

Why was this test suite created?

In order to verify that the Name Constraints extension protects users from mis-issued certificates, we developed this test suite. As we did so, it became apparent that there are many different implementations of the certificate verification algorithm, each of which has corner cases that are not properly handled. Check out the archived results tab for examples of HTTPS clients with failing tests.

These are some examples of incorrect behavior we have observed when we developed this test suite:

  • If a hostname is in the common name (CN) of the subject instead of the subject alternate name (SAN) extension, it may not be checked against the Name Constraints extension at all even when it is used as the hostname for verification.
  • Although RFCs suggest that only hostnames should appear in the common name (CN) of a certificate, many clients will still accept an IP address in the CN. However, these implementations do not always apply IP Name Constraints against the CN.
  • Some implementations show a certificate as valid when one subject name matches a Name Constraint blacklist but a second subject name does not. (The RFC specification makes it clear that validation should fail if any subject name is blacklisted.)

Although this test suite certainly isn't exhaustive, the hope is that it provides a reference for developers to use in order to help verify that Name Constraints extensions are properly enforced in their TLS implementation.

Name Constraints Documentation

The Name Constraints extension along with its verification is defined in RFC 5280. Further useful information can be found in RFC 6125 which defines how to identify a server's hostname from certificate extensions.

About the tests

This test suite looked at the typical fields implementations use to verify hostnames (i.e. including the hostname in the subject common name (CN) and using DNS and IP SANs) and the different ways of using the name constraints extension (whitelisting and blacklisting on either DNS or IP trees). The test suite generates a test case for all possible combinations of these.

Interpreting test results

The test executor first evaluates whether a given TLS implementation implements hostname validation. Some of the implementations under test only check if a certificate is valid and leave hostname verification to consuming applications.

As a TLS implementation consumer, it is important to understand if the TLS library is doing verification for you and, if not, what subject identifiers were included in name constraints checks. As noted above, many HTTPS clients have been observed to use a TLS library that checks SANs against name constraints but then also allow the subject common name (CN) to verify the required hostname.

It is therefore most useful to use this test suite against end-to-end HTTPS clients. Where that doesn't apply, make sure to pay attention to whether VALIDATE_DNS and VALIDATE_IP features are supported.

For each test definition, VALID means a field that matches the client hostname, INVALID means a field that does not match the client hostname, and NONE means the field is not present.

What is BetterTLS: Path Building?

This test suite exercises the certificate path building capabilities of TLS implementations. Path building refers to the problem of building a valid, trusted chain from an end-entity's leaf certificate to a trust anchor. For background and motivation, we highly recommend you read Ryan Sleevi's blog post on this topic. The problem stated can be succinctly summarized with this excerpt:

Often, when I talk to people who are responsible for configuring certificates on their servers, they often talk about _the_ certificate chain. ... [But] there are many chains, with different chains are needed by different clients, who have different root stores and different behaviors.

Historically the TLS specifications have not required that TLS implementations support particularly robust certificate path building. In practice many of them (such as OpenSSL at the time of writing) don't support verifying anything other than a simple, non-branching sequence of certificates. This has changed in the specification for TLS 1.3 which advises that implementations SHOULD support more robust certificate path building:

Note: Prior to TLS 1.3, "certificate_list" ordering required each certificate to certify the one immediately preceding it; however, some implementations allowed some flexibility. Servers sometimes send both a current and deprecated intermediate for transitional purposes, and others are simply configured incorrectly, but these cases can nonetheless be validated properly. For maximum compatibility, all implementations SHOULD be prepared to handle potentially extraneous certificates and arbitrary orderings from any TLS version, with the exception of the end-entity certificate which MUST be first.

This test suite can be applied to TLS implementations to not only verify whether they satisfy the above provision, but whether they are doing so correctly.

Check out our blog post for a summary of results from common TLS implementations at the time this suite was created.

Why does this matter?

The web PKI ecosystem is a shifting landscape. Features like what certificate authorities are trusted, what algorithms should be used, and what X.509 certificate extensions can (and should) be enforced have been changing every year. Service owners need to keep up with these changes while also ensuring that existing clients are able to reach their service.

For example, the Let's Encrypt R3 CA has two certificates; one signed by "DST Root CA X3" and one signed by "ISRG Root X1". Older clients only trusted the self-signed DST Root CA X3 certificate, usually since they were built before the ISRG Root X1 CA had made its way into broadly distributed truststores. However, newer clients only trust the ISRG Root X1 CA since the DST Root CA X3 self-signed certificate expired on September 30, 2021. Ideally, service owners would be able to send both the DST Root CA X3 => R3 and ISRG Root X1 => R3 certificate so that any client can verify their trust of the Let's Encrypt R3 CA and ultimately verify their trust in the end entity certificate. In practice, many clients are not able to handle getting muiltple potential paths in a TLS response and this leaves both clients and service operators subject to outages.

In general, having clients with a robust certificate path building allows the community to be more agile and make changes to the web PKI ecosystem over time while reducing risk of breaking older clients. Here are just a few examples of these sorts of changes in the past:

At a minimum, this test suite can help inform service owners about how much path building different TLS client implementations support so that they can determine how clients will be impacted by changes to their service's TLS configuration and certificates.

About the tests

Trust Graphs

All of the tests in this suite utilize a Trust Graph. A Trust Graph is a directed graph where the nodes represent potentially trusted entities (a Distinguished Name and a public key) and the edges represent a certificate. Consider the following example which defines the "TWO_ROOTS" trust graph:

+--------+
| Root 1 |======v
+--------+      +-----+         +----+
                | ICA | ======> | EE |
+--------+      +-----+         +----+
| Root 2 |======^
+--------+

There are two root CAs, both of which have created a certificate for an intermediate CA (ICA), which ultimately creates certificates for the end entity (EE). The end entity (a service owner) doesn't know in general which root is trusted by clients, so it will always send three certificates to clients: the leaf certificate for the end entity signed by ICA (ICA => EE), a certificate for the ICA signed by Root 1 (Root 1 => ICA), and a certificate for the ICA signed by Root 2 (Root 2 => ICA). Clients which support path building should be able to verify trust in EE whether they trust Root 1 or Root 2 as a trust anchor. Put another way, the presence of a certificate which leads to a non-trusted root should not prohibit the client's ability to find and verify a chain.

Invalid Edges

To verify that clients support robust path building, test cases can mark an edge in the Trust Graph as invalid. The test suite supports several reasons for an edge being invalid (see the next section) such as the certificate being expired. When there are multiple paths to a trust anchor, a robust client should be able to find a path to the trust anchor despite invalid edges. Put another way, clients should not just find any path and then verify it; they must check that each edge is valid as they build a path.

Consider the following trust graph copied from RFC 4158 figure 7:

     +---------+
     |  Trust  |
     | Anchor  |
     +---------+
      |       |
      v       v
   +---+    +---+
   | A |<-->| C |
   +---+    +---+
    |         |
    |  +---+  |
    +->| B |<-+
       +---+
         |
         v
       +----+
       | EE |
       +----+

If the certificate "Trust Anchor => A" is expired a client should still be able to find a path from "Trust Anchor" down to "EE" (and similarly if the "Trust Achor => C" certificate is expired).

Invalid Edge Reasons

This test suite supports several reasons for an edge being invalid as described below. Some of these reasons MUST be supported by any TLS implementation, such as a certificate being expired. Other reasons are not strictly required, such as the certificate using a deprecated signature algorithm (i.e. using SHA-1).

EXPIRED

The certificate's NotAfter time is before now. All TLS implementations must support this.

NAME_CONSTRAINTS

The certificate has a name constraints extension that is at odds with the end-entity certificate. This extension is marked as critical by this test suite, so all implementations should support this.

BAD_EKU

The certificate has an Extended Key Usage extension that is incompatible with the end-entity's use of the certificate for TLS server authentication. The Mozilla Certificate Policy FAQ states:

Inclusion of EKU in CA certificates is generally allowed. NSS and CryptoAPI both treat the EKU extension in intermediate certificates as a constraint on the permitted EKU OIDs in end-entity certificates. Browsers and certificate client software have been using EKU in intermediate certificates, and it has been common for enterprise subordinate CAs in Windows environments to use EKU in their intermediate certificates to constrain certificate issuance.

So while most implementations support the semantics of an incompatible EKU in CAs as a reason to treat a certificate as invalid, RFCs do not require this behavior so this might not be supported by all TLS implementations.

MISSING_BASIC_CONSTRAINTS

The certificate is missing the Basic Constraints extension. RFC 5280 requires that all CAs have this extension, so all implementations should support this.

NOT_A_CA

The certificate has a basic constraints extension, but it does not identify the certificate as being a CA. Similarly to the above, all implementations should support this.

DEPRECATED_CRYPTO

The certificate is signed with an algorithm that has been considered deprecated (i.e. using SHA-1). Enforcement of SHA-1 deprecation is not universally present in all TLS implementations.

Interpreting test results

The test executor first evaluates whether a given TLS implementation supports branching certificate chains at all (using the TWO_ROOTS trust graph described above). If it doesn't, most tests in the suite will be skipped. As noted above, according to RFC 8446 any TLS implementation supporting TLS 1.3 SHOULD support branching certificate chains.

The test executor also tests whether each Invalid Reason described above is supported by the TLS implementation. It is reasonable for some of them (such as BAD_EKU and DEPRECATED_CRYPTO) to not be supported and subsequent test cases that use those Invalid Reasons will be skipped.

This is a collection of saved test results. This test executor detects supported features (such as distrusting certificates with a deprecated signature algorithm and whether path building is supported at all) and skips tests that rely on unsupported features. To make sense of results, you should not only consider failed tests but see which were not even run.

Check out our name constraints and path building blog posts for some discussion of these results.

Summary

Implementation

Version

Date

Supported Features
Unsupported Features

Test Result Summary

Complete Test Results