Your Server Has More Than One Trust Store

On 21 October my main Ubuntu server installed the eMudhra certificate fix by itself and logged "done." Three days later I asked its Java runtime whether it still trusted the compromised keys. It did, on every test chain I gave it.

The operating system was patched. The machine was not. Those turned out to be different questions, and the second one has as many answers as the machine has trust stores, the sets of root certificates each program checks to decide which servers to believe. This one has more than I would have guessed.

Eight bits

On Saturday 17 October a researcher reported that eight intermediate certificate authorities run by eMudhra had private keys anyone could compute. Not stolen. Computed. Scott Helme recovered all eight on a laptop in under a second. They are P-384 keys, which should be random 384-bit numbers. One of them is the number 4. The others are 56, 66, 115, 155, 156, 230 and 237. Each fits in a single byte.

Six of the eight could issue TLS certificates for any website. All were created in a key ceremony on 30 April 2025. eMudhra's own account says the keys were made on FIPS 140-2 Level 3 hardware security modules, and that a key wrap/unwrap step "produced a second, weak set of keys alongside the original strong keys." The weak set was the one published. It went unnoticed for about 18 months, through two independent audit cycles.

eMudhra says none of the eight ever issued a certificate. That tells you nothing about tomorrow. Whoever does the arithmetic can sign a certificate for your bank, and every client that trusts the chain will believe it.

Why the browsers shrugged

The browsers fixed this before most people heard about it. Google's root program got the report at 18:05 UTC and pushed a block at 19:45 UTC, by key, so every certificate sharing those keys died at once. Mozilla's report arrived at 20:06 UTC, and its block had reached every Firefox user with standard settings by 22:59 UTC. No restart, no update.

Browsers had a second guard as well. Under their Certificate Transparency policies, Chrome, Firefox and Safari expect a publicly trusted TLS certificate to have been logged publicly, and reject one that was not. Google says no certificate chaining to the six TLS intermediates was ever logged before the block, so a browser that enforces CT had a second barrier against any forgery made with these keys. CT is not a general shield against a broken CA. It rejects certificates nobody logged, and here nobody had logged one. Browsers had a block they push themselves and a log they check against, and this time both held.

Everything else reads a file

Your services are not browsers. Helme's post puts the difference plainly: most non-browser clients with default settings, curl, Python, Node, Go, Java, git and the rest, "trust a CA bundle on disk, they don't check revocation by default, and they don't check CT." eMudhra revoking the intermediates does nothing for them. They only stop trusting the chain when the file they read stops containing a root that leads to it.

There were two such roots. Through eMudhra's own hierarchy, the intermediates lead to emSign ECC Root CA - G3, which every major root program trusted. Each was also cross-signed by Sectigo's AAA Certificate Services, which Apple and Microsoft still trust and Mozilla and Chrome had already dropped. A client is exposed if any bundle it reads contains either root. Mozilla's guidance to trust store operators offers three options. Reliable fail-closed revocation checking. Blocking the specific intermediates. Or, failing those, removing the roots.

Mozilla's NSS took the third option, and so did Ubuntu, and most distribution bundles derive from Mozilla's list. That is why the fix shipped as a package update. A package update fixes the copy of the bundle that the package owns. The question is how many copies you have.

One machine, nine answers

The researcher who found the bug runs test servers that present certificates issued from the compromised intermediates, through both roots, with and without CT logging. I pointed every TLS client I could find on haywire, the build and VM host I run most of my work on, at four of them. I also pointed each one at a known-good site as a control. A client that cannot reach the control is not telling you anything. All four test hosts use the same intermediate, DV G3B, so the live test checks one of the six. The offline check below covers all of them.

Same interpreter, same machine, two answers. Python's standard library asks OpenSSL, which reads the patched system bundle. The requests library reads its own copy from the certifi package, which had not been updated; the fixed release is 2026.10.19. Node ships its own copy of Mozilla's list inside the binary; 24.22.1 is the fixed release on that line. The Azure CLI carries a certifi from July 2024 and trusts every chain, including the ones no CT-enforcing browser would ever have accepted.

Offline, the researcher publishes all 14 bad chains in chains.zip. I ran them against every bundle file on the disk. There were 175 files with bundle-shaped names and 27 distinct bundles. That count overstates the risk, and I will come back to why. The script below needs OpenSSL 3 and the roots the chains lead to, which the researcher publishes as roots.json. It first checks that every test chain verifies against those roots, then counts a failure as a rejection only when OpenSSL says it found no trusted issuer. Its first version counted a typo in a path as a pass, and a reviewer showed that its second would have counted a damaged chain the same way.

#!/usr/bin/env bash
# Usage: ./trust-check.sh CHAINS_DIR ROOTS_PEM BUNDLE...
# CHAINS_DIR: the unzipped chains.zip from emudhra.we-have-a-problem.com.
# ROOTS_PEM:  the roots those chains lead to (the "pem" fields of roots.json).
# Each chain must first verify against ROOTS_PEM, so damaged test material
# cannot pass as a rejection. After that, a failure counts as a rejection
# only when OpenSSL says it found no trusted issuer or root.
# Each chain file must start with its leaf: openssl verify checks only the
# first certificate in a file, and -untrusted supplies the rest.
set -euo pipefail
openssl version | grep -q '^OpenSSL 3' || { echo "needs OpenSSL 3.x" >&2; exit 2; }
chains="$1"; roots="$2"; shift 2

check() {  # check BUNDLE CHAIN -> prints accepted | rejected | error: ...
  local out
  if out=$(openssl verify -no-CApath -no-CAstore -CAfile "$1" \
             -untrusted "$2" "$2" 2>&1); then
    echo accepted
  elif grep -Eq 'unable to get (local )?issuer certificate|self-signed certificate in certificate chain' \
         <<<"$out"; then
    echo rejected
  else
    echo "error: $(grep -m1 -E 'error [0-9]+ at' <<<"$out" || tail -n 1 <<<"$out")"
  fi
}

for chain in "$chains"/*.pem; do
  [ "$(check "$roots" "$chain")" = accepted ] ||
    { echo "test chain does not verify against $roots: $chain" >&2; exit 2; }
done

for bundle in "$@"; do
  if ! grep -q 'BEGIN CERTIFICATE' "$bundle" 2>/dev/null; then
    echo "SKIP $bundle: not a readable PEM bundle" >&2
    continue
  fi
  ok=0; rejected=0; other=0
  for chain in "$chains"/*.pem; do
    result=$(check "$bundle" "$chain")
    case "$result" in
      accepted) ok=$((ok + 1)) ;;
      rejected) rejected=$((rejected + 1)) ;;
      *) other=$((other + 1)); echo "  $chain: $result" >&2 ;;
    esac
  done
  echo "$ok accepted, $rejected rejected, $other errors: $bundle"
done

$ ./trust-check.sh chains roots.pem /etc/ssl/certs/ca-certificates.crt \
    ~/.venv/lib/python3.14/site-packages/certifi/cacert.pem \
    /opt/az/lib/python3.13/site-packages/certifi/cacert.pem
0 accepted, 14 rejected, 0 errors: /etc/ssl/certs/ca-certificates.crt
7 accepted, 7 rejected, 0 errors: /home/cisco/.venv/lib/python3.14/site-packages/certifi/cacert.pem
14 accepted, 0 rejected, 0 errors: /opt/az/lib/python3.13/site-packages/certifi/cacert.pem

To be fair to the package manager, most of that table is rejections. curl, Go, wget, Python's standard library and gcloud all read the system bundle, and the system bundle was fixed the day after public disclosure without anyone touching the machine. "Keep the OS patched" is good advice because, for most of the clients on this host, it was the whole job. The trouble is that you cannot tell from the outside which clients are in the other column.

The keystore nobody rewrote

Java was the one that bothered me, because Ubuntu's OpenJDK does not carry its own list. It reads a keystore that the system builds from the same certificates as everything else. If the system bundle was fixed, Java should have been.

The package log shows the update arriving on 21 October at 06:40 UTC. It shows the Java trigger running. It ends with done. The keystore on disk tells a different story.

$ ls -l --time-style=long-iso /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/java/cacerts
-rw-r--r-- 1 root root 180744 2026-10-21 06:40 /etc/ssl/certs/ca-certificates.crt
-rw-r--r-- 1 root root 171627 2025-10-13 09:49 /etc/ssl/certs/java/cacerts
# the system bundle is three days old. The keystore Java reads is a year old.
$ keytool -list -v -cacerts -storepass changeit | grep -E "^Owner: CN=(AAA|emSign ECC Root CA - G3)"
Owner: CN=AAA Certificate Services, O=Comodo CA Limited, (trimmed)
Owner: CN=emSign ECC Root CA - G3, O=eMudhra Technologies Limited, (trimmed)
# both roots that lead to the bad intermediates are still in it.

That keystore had not been rewritten in over a year. It held 152 certificates against the system's 120. Forty-four of them are certificates the system store no longer carries. Twelve certificates the system has added since are missing. Every root the system dropped since October 2025 had stopped at the PEM file and never reached Java.

One machine is an anecdote, so I reproduced it in a clean container. Install Ubuntu 26.04's original ca-certificates with Java, then upgrade to the fixed package the normal way. The upgrade reports "6 added, 30 removed." The system bundle drops from 144 certificates to 120. The Java keystore stays at 144, still holds the emSign root, and Java still accepts the compromised chain. Ubuntu 24.04 does the same: its keystore stays at 146 certificates and keeps both bad roots while the system bundle drops to 120. Ubuntu's post says AAA left its bundles from 22.04 on, but the 24.04 release package I installed, 20240203, still ships it, and the keystore built from it kept it through the upgrade. I ran the 26.04 case twice. A fresh install with the fixed package builds a clean keystore, which is why a quick test on a new box looks fine.

Ubuntu knows. Its security post, updated on 22 October, says the Java store's synchronization "is currently affected by an unresolved issue" and gives the manual fix: delete /etc/ssl/certs/java/cacerts, then run update-ca-certificates --fresh. I tested the variants in the container. Running --fresh without deleting the file first does not remove a single stale root; it added twelve and left the emSign root in place. Deleting the file and running dpkg-reconfigure ca-certificates-java leaves Java with no keystore at all, so every TLS connection fails. Ubuntu's two commands, in that order, produced a clean store of 120 certificates, and Java then refused both bad chains while still reaching the control site.

This is Ubuntu's fix for Ubuntu's Java keystore, not a general Java one. Two warnings before you run it. Ubuntu's post says any CA you added to the Java store by hand, an internal or partner CA, is gone afterwards and has to be added again, so export it first. And removing roots can break legitimate connections that chained through them, which Ubuntu also warns about. A control site proves the store works. It does not prove your own dependencies still connect, so test those too. Then restart the services. A running JVM has already loaded its trust store, and a fixed file does nothing for it until it starts again.

The same post also says unattended upgrades will apply the fix within 24 hours. That part was true. Automation applies the part of a fix that is a package. The part that is a paragraph in a blog post waits for a human to read it, and my server does not read blogs.

2008 already taught this

This is not the first time a key-generation mistake hid in plain sight. In May 2008, Debian announced that a Debian-specific change to OpenSSL had made its random number generator predictable. The broken version had been uploaded on 17 September 2006. For about twenty months, SSH, VPN and TLS keys generated with OpenSSL on Debian and its derivatives, Ubuntu among them, came out of a generator the advisory said made key material "guessable."

Fixing the generator took one package update. It did nothing for the keys already made. Debian shipped a detector for known weak keys, because the cleanup meant finding every copy: in authorized_keys files, in certificates, on machines nobody remembered owning.

Eighteen years later the shape is the same. The fix lands at the source in hours. The copies live wherever someone vendored them, cached them, baked them into an image or froze them into firmware. The source does not know where they are. Only you can look.

How this test lies

My advice is to test each client's real TLS path against the bad chains. Before writing that down, I tried to break it. Three ways turned up. The first one had already caught me.

A broken test reads as "rejected." My first table said Go, Java and git all refused the bad chains. They had refused nothing. The Go binary sat in a directory that would not execute, and the Java and git commands failed for reasons unrelated to certificates. Failure was the default output, and failure looks like safety. The control column exists because of that run. Read the error text too: "unknown authority" or "PKIX path building failed" is a rejection. A timeout is not.

The environment changes the answer. From my shell, Python's urllib rejected the Sectigo chain. With one variable set, SSL_CERT_FILE pointing at the Azure CLI's old bundle, the same binary accepted it. CURL_CA_BUNDLE did the same to curl. A service that sets either one in its unit file or container image is a different client from the one in your terminal. None of the services on this host set them, but I had to check, and so do you. Run the test as the service: same user, same unit, same image.

A file on disk is not a bundle in use. Ubuntu's pip ships an old certifi file that accepts both paths. Counted naively, it is a finding. It is not: Debian patches pip so that file is never read, and its where() function returns the system bundle. Counting files gives you a list to check. Each file on it needs a client that actually reads it before it counts as exposure.

There is plenty this does not cover. I did not test revocation checking, which would have caught some of this in Java if it were turned on. I did not test macOS or Windows, where Apple and Microsoft blocked the intermediates in their own stores on 22 and 20 October. I did not test the two code-signing intermediates, which affect Windows signing rather than TLS. And this is one machine plus a container, not a survey. Cloudflare's writeup makes the same point from the other side: "A successful browser test does not prove that a background service is protected. Neither does a curl test if the application uses a different verifier."

Find the copies

A trust store is a dependency, and it rots like one, as I argued in every dependency is technical debt. Nothing announces that a vendored bundle has fallen behind. A year-old keystore and a three-day-old one look identical until something asks the specific question. That is the gap I wrote about in your backups are a belief system: the thing exists, nobody has exercised it, and its existence passes for evidence.

It is the same failure as a green signal that answers an easier question than the one you asked. A clean audit is not a safe version was a database that had not caught up. This one is narrower. The package manager said done, and it was right about the package. Java reads a different file.

The cost asymmetry is what makes this worth an hour on Monday. Building the inventory is a few commands per host. The keys had eight bits of entropy, at most 255 tries each, and the method is now public. Anyone who can get on a network path between your service and an API can present a certificate your Java runtime will accept, until you find the copy and replace it.

"The package manager said done, and it was right about the package. Java reads a different file."

Cisco Caceres•Coding since the late 1970s. Still learning, still building.