BetterTLS is a collection of test suites for TLS clients. At the moment, two test suites have been implemented. One tests a client's validation of the Name Constraints certificate extension. This extension is placed on CA certificates which restrict the DNS/IP space for which the CA (or sub-CAs) can issue certificates. The other test suite evaluates a TLS client's ability to discover a valid certificate path (a certificate "chain") from an unordered collection of certificates.
The BetterTLS project was originally created to ensure that we could create a name-constrained CA that could not be abused. We were only able to gain this confidence by evaluating whether clients (such as popular web browsers) were correctly handling the name constraints extension.
We later added an additional test suite for certificate path building because of the recurring pains we experienced during transitions in the web PKI ecosystem, such as intermediate CA expiry and signature algorithm deprecation. The goal is to help TLS implementations identify and verify changes that could alleviate these pain points and to help application developers pick TLS implementations that won't break during these transitions.
This project is inspired in part by badssl.com. In a similar vein, we want to highlight some of the issues and difficulties we've found with HTTPS/TLS implementations so that they can be corrected. By doing so, we hope to make TLS better for everyone.
This site contains archived test results. These test results were from the latest version of TLS implementations at the time they were run, though some are quite old now. For discussions and interpretation of these results, you should check out our blog post about Name Constraints and the follow-up blog post about path building.
Clients and browsers can be tested using the code in the BetterTLS repository. Check out the README for examples on how to do so.
BetterTLS is open source software. We encourage you to help add more tests, fix any issues you find in the existing test suites, or even fork it for use in your own software project. Pull requests are welcome!
The Name Constraints test suite is a collection of tests for HTTPS clients implementing verification of the Name Constraints certificate extension. This extension is placed on CA certificates which restricts the DNS/IP space for which the CA (or sub-CAs) can issue certificates. One of the most common use-cases is for CAs to cross-certify another CA, but for a limited namespace. For example, Acme Corp. could sign a certificate for an intermediary CA but include a Name Constraints extension which blacklists the intermediate CA from signing certificates for acme.com.
Another use-case for the Name Constraints extension (which motivated this project) is as a way to reduce the risk exposure of trusting an internal CA. For example, we could create an internal CA which includes a Name Constraints extension with a whitelist of internal.example.com and 192.168.0.0/16. A user who imports this CA into her browser's truststore can then be confident that this certificate will only be used for internal company pages and IPs and cannot be used to intercept secure communications with external websites.
However, the Name Constraints extension only provides these assurances if we can be confident that HTTPS clients properly enforce them...
In order to verify that the Name Constraints extension protects users from mis-issued certificates, we developed this test suite. As we did so, it became apparent that there are many different implementations of the certificate verification algorithm, each of which has corner cases that are not properly handled. Check out the archived results tab for examples of HTTPS clients with failing tests.
These are some examples of incorrect behavior we have observed when we developed this test suite:
Although this test suite certainly isn't exhaustive, the hope is that it provides a reference for developers to use in order to help verify that Name Constraints extensions are properly enforced in their TLS implementation.
The Name Constraints extension along with its verification is defined in RFC 5280. Further useful information can be found in RFC 6125 which defines how to identify a server's hostname from certificate extensions.
This test suite looked at the typical fields implementations use to verify hostnames (i.e. including the hostname in the subject common name (CN) and using DNS and IP SANs) and the different ways of using the name constraints extension (whitelisting and blacklisting on either DNS or IP trees). The test suite generates a test case for all possible combinations of these.
The test executor first evaluates whether a given TLS implementation implements hostname validation. Some of the implementations under test only check if a certificate is valid and leave hostname verification to consuming applications.
As a TLS implementation consumer, it is important to understand if the TLS library is doing verification for you and, if not, what subject identifiers were included in name constraints checks. As noted above, many HTTPS clients have been observed to use a TLS library that checks SANs against name constraints but then also allow the subject common name (CN) to verify the required hostname.
It is therefore most useful to use this test suite against end-to-end HTTPS clients. Where that doesn't apply, make sure to pay attention to whether VALIDATE_DNS and VALIDATE_IP features are supported.
For each test definition, VALID means a field that matches the client hostname, INVALID means a field that does not match the client hostname, and NONE means the field is not present.
This test suite exercises the certificate path building capabilities of TLS implementations. Path building refers to the problem of building a valid, trusted chain from an end-entity's leaf certificate to a trust anchor. For background and motivation, we highly recommend you read Ryan Sleevi's blog post on this topic. The problem stated can be succinctly summarized with this excerpt:
Historically the TLS specifications have not required that TLS implementations support particularly robust certificate path building. In practice many of them (such as OpenSSL at the time of writing) don't support verifying anything other than a simple, non-branching sequence of certificates. This has changed in the specification for TLS 1.3 which advises that implementations SHOULD support more robust certificate path building:
This test suite can be applied to TLS implementations to not only verify whether they satisfy the above provision, but whether they are doing so correctly.
Check out our blog post for a summary of results from common TLS implementations at the time this suite was created.
The web PKI ecosystem is a shifting landscape. Features like what certificate authorities are trusted, what algorithms should be used, and what X.509 certificate extensions can (and should) be enforced have been changing every year. Service owners need to keep up with these changes while also ensuring that existing clients are able to reach their service.
For example, the Let's Encrypt R3 CA has two certificates; one signed by "DST Root CA X3" and one signed by "ISRG Root X1". Older clients only trusted the self-signed DST Root CA X3 certificate, usually since they were built before the ISRG Root X1 CA had made its way into broadly distributed truststores. However, newer clients only trust the ISRG Root X1 CA since the DST Root CA X3 self-signed certificate expired on September 30, 2021. Ideally, service owners would be able to send both the DST Root CA X3 => R3 and ISRG Root X1 => R3 certificate so that any client can verify their trust of the Let's Encrypt R3 CA and ultimately verify their trust in the end entity certificate. In practice, many clients are not able to handle getting muiltple potential paths in a TLS response and this leaves both clients and service operators subject to outages.
In general, having clients with a robust certificate path building allows the community to be more agile and make changes to the web PKI ecosystem over time while reducing risk of breaking older clients. Here are just a few examples of these sorts of changes in the past:
At a minimum, this test suite can help inform service owners about how much path building different TLS client implementations support so that they can determine how clients will be impacted by changes to their service's TLS configuration and certificates.
All of the tests in this suite utilize a Trust Graph. A Trust Graph is a directed graph where the nodes represent potentially trusted entities (a Distinguished Name and a public key) and the edges represent a certificate. Consider the following example which defines the "TWO_ROOTS" trust graph:
+--------+
| Root 1 |======v
+--------+ +-----+ +----+
| ICA | ======> | EE |
+--------+ +-----+ +----+
| Root 2 |======^
+--------+
There are two root CAs, both of which have created a certificate for an intermediate CA (ICA), which ultimately creates certificates for the end entity (EE). The end entity (a service owner) doesn't know in general which root is trusted by clients, so it will always send three certificates to clients: the leaf certificate for the end entity signed by ICA (ICA => EE), a certificate for the ICA signed by Root 1 (Root 1 => ICA), and a certificate for the ICA signed by Root 2 (Root 2 => ICA). Clients which support path building should be able to verify trust in EE whether they trust Root 1 or Root 2 as a trust anchor. Put another way, the presence of a certificate which leads to a non-trusted root should not prohibit the client's ability to find and verify a chain.
To verify that clients support robust path building, test cases can mark an edge in the Trust Graph as invalid. The test suite supports several reasons for an edge being invalid (see the next section) such as the certificate being expired. When there are multiple paths to a trust anchor, a robust client should be able to find a path to the trust anchor despite invalid edges. Put another way, clients should not just find any path and then verify it; they must check that each edge is valid as they build a path.
Consider the following trust graph copied from RFC 4158 figure 7:
+---------+
| Trust |
| Anchor |
+---------+
| |
v v
+---+ +---+
| A |<-->| C |
+---+ +---+
| |
| +---+ |
+->| B |<-+
+---+
|
v
+----+
| EE |
+----+
If the certificate "Trust Anchor => A" is expired a client should still be able to find a path from "Trust Anchor" down to "EE" (and similarly if the "Trust Achor => C" certificate is expired).
This test suite supports several reasons for an edge being invalid as described below. Some of these reasons MUST be supported by any TLS implementation, such as a certificate being expired. Other reasons are not strictly required, such as the certificate using a deprecated signature algorithm (i.e. using SHA-1).
The certificate's NotAfter time is before now. All TLS implementations must support this.
The certificate has a name constraints extension that is at odds with the end-entity certificate. This extension is marked as critical by this test suite, so all implementations should support this.
The certificate has an Extended Key Usage extension that is incompatible with the end-entity's use of the certificate for TLS server authentication. The Mozilla Certificate Policy FAQ states:
So while most implementations support the semantics of an incompatible EKU in CAs as a reason to treat a certificate as invalid, RFCs do not require this behavior so this might not be supported by all TLS implementations.
The certificate is missing the Basic Constraints extension. RFC 5280 requires that all CAs have this extension, so all implementations should support this.
The certificate has a basic constraints extension, but it does not identify the certificate as being a CA. Similarly to the above, all implementations should support this.
The certificate is signed with an algorithm that has been considered deprecated (i.e. using SHA-1). Enforcement of SHA-1 deprecation is not universally present in all TLS implementations.
The test executor first evaluates whether a given TLS implementation supports branching certificate chains at all (using the TWO_ROOTS trust graph described above). If it doesn't, most tests in the suite will be skipped. As noted above, according to RFC 8446 any TLS implementation supporting TLS 1.3 SHOULD support branching certificate chains.
The test executor also tests whether each Invalid Reason described above is supported by the TLS implementation. It is reasonable for some of them (such as BAD_EKU and DEPRECATED_CRYPTO) to not be supported and subsequent test cases that use those Invalid Reasons will be skipped.
This is a collection of saved test results. This test executor detects supported features (such as distrusting certificates with a deprecated signature algorithm and whether path building is supported at all) and skips tests that rely on unsupported features. To make sense of results, you should not only consider failed tests but see which were not even run.
Check out our name constraints and path building blog posts for some discussion of these results.
| Implementation | |
| Version | |
| Date | |
| Supported Features | |
| Unsupported Features |