Add support for post-quantum groups

[jtesta]

OpenSSL v3.5 (an LTS release) came out early last month. In it is support for a few standardized post-quantum groups (standardized via NIST FIPS 203), along with hybrid algorithms. By upgrading our OpenSSL dependency, I've been able to implement tests for these groups.

Along the way, I also added support for TLSv1.3-specific Brainpool curves, along with curveSM2 (an ECDHE curve mandated for use in China).

The full list of new TLS groups detected by this PR is:

• MLKEM512
• MLKEM768
• MLKEM1024
• X25519MLKEM768
• SecP256r1MLKEM768
• SecP384r1MLKEM1024
• brainpoolP256r1tls13
• brainpoolP384r1tls13
• brainpoolP512r1tls13
• curveSM2

Post-quantum algorithms are based on new cryptographic primitives, so their security is much less trusted at this time. Hence, the industry now prefers hybrid algorithms, which are new PQ algs combined with classical/well-tested systems (like X25519). For example, in the SSH world, OpenSSH has used sntrup761x25519-sha512@openssh.com as the default key exchange from v9.0 (released April 2022) to v9.9, and offered sntrup4591761x25519-sha512@tinyssh.org from v8.0 (released April 2019) to v8.4. In v10.0 (released April 2025), the default key exchange is now mlkem768x25519-sha256.

I tagged the pure MLKEM groups as yellow since they're so new. I tagged the hybrid X25519MLKEM768 as green since its at least as strong as X25519 on its own. I tagged curveSM2 as red since it was developed entirely by the Chinese government without input from the broader academic community (as far as I know).

[rbsec]

Great stuff as always, and definitely adds more weight to the argument for requiring OpenSSL 3.x going forward.

I tagged the pure MLKEM groups as yellow since they're so new.

The one area that I would query is flagging those MLKEM groups in yellow, because that opens up some interesting debates. Newer crypto is always a little bit concerning - but they've been around for a little while, and have gone through the (fairly robust) NIST processes to end up in FIPS 203. So if we're going to flag them as yellow on the basis of their newness, then I think we need to have some clearly defined basis for that (i.e. after how many years do we remove that flag?), and make sure that we're consistent with that approach going forward.

But I also wonder about what message that's sending, and how people will interpret the very crude rating system that sslscan has. Because by flagging them as yellow, we're essentially saying that we have the same level of concern about them as we do over 1024 RSA keys, TLSv1.0, and 3DES/RC4/CCM8 ciphers - all of which have significant known vulnerabilities - and actively discourage their use. But "it's a fairly new NIST approved standard" feels pretty weak as justification for that.

So I wonder if a safer middle ground is perhaps just to leave them as white (not giving a strong opinion either way), and flagging the ones that have been demonstrated to be more robust like X25519MLKEM768 as green?

I tagged curveSM2 as red since it was developed entirely by the Chinese government without input from the broader academic community (as far as I know).

We currently flag the SM4 and GOST ciphers (from the Chinese and Russian governments respectively) as yellow, so should probably be consistent with that. If we're changing the main MLKEM ones to white then SM2 should probably be yellow to match the existing ones.

Alternatively they could all go red - but in the absence of any actual evidenced weaknesses or concerns with them that could be a bit questionable, especially given that US developed crypto doesn't exactly have a spotless record *cough*Dual_EC_DRBG*cough*....

I don't really have strong feeling either way, but we should probably be consistent between them. What do you think?

[jtesta]

but they've been around for a little while, and have gone through the (fairly robust) NIST processes to end up in FIPS 203

They have indeed withstood some good challenges to make it this far, but it seems that the crypto community at large isn't ready to accept them as full replacements. As I mentioned before, OpenSSH is still relying on hybrid key exchanges to ensure no surprises come up (in fact, they just added mlkem768x25519-sha256 early last month in the 10.0 release). Speaking of surprises, a few years ago OpenSSH abruptly replaced sntrup4591761x25519-sha512@tinyssh.org with sntrup761x25519-sha512@openssh.com due to the PQ algorithm having much less security than was expected! So I'd be wary of anyone using pure PQ algorithms at this time.

So if we're going to flag them as yellow on the basis of their newness, then I think we need to have some clearly defined basis for that (i.e. after how many years do we remove that flag?), and make sure that we're consistent with that approach going forward.

Perhaps at this time, we can wait until OpenSSH feels comfortable with trusting pure PQ key exchanges. I'd trust that community's judgement.

So I wonder if a safer middle ground is perhaps just to leave them as white

I see a tool like sslscan as somewhat of an authority figure. People will more likely avoid anything it tags as a warning or problem. And so this influence should be used for the greater good. In this case, they should be warned that using pure PQ carries a higher risk at this point in time. From there, its up to them whether they want to accept that risk in exchange for a slightly faster connection setup vs a hybrid algorithm.

FYI, in ssh-audit (https://github.com/jtesta/ssh-audit), I'm now flagging all non-hybrid key exchanges with warnings, including classic DHE/ECDHE systems. I haven't gotten any pushback about that as of yet...

Because by flagging them as yellow, we're essentially saying that we have the same level of concern about them as we do over 1024 RSA keys, TLSv1.0, and 3DES/RC4/CCM8 ciphers - all of which have significant known vulnerabilities - and actively discourage their use.

Hmm... it wouldn't take much to convince me that RSA1024, 3DES, RC4, and CCM8 should all be red instead of yellow! But perhaps that's another conversation. Speaking of which, it wouldn't hurt to start another thread to re-evaluate all the color codings. I've had other ideas for quite some time about re-classifying things, so now might be a good time to sit down and focus on it, if you're interested.

We currently flag the SM4 and GOST ciphers (from the Chinese and Russian governments respectively) as yellow, so should probably be consistent with that.

I agree that they should be consistent, but I'd vote strongly in favor of marking them in red. (I suppose I'll save my rationale for the dedicated thread...)

Alternatively they could all go red - but in the absence of any actual evidenced weaknesses or concerns with them that could be a bit questionable, especially given that US developed crypto doesn't exactly have a spotless record coughDual_EC_DRBGcough....

The Dual_EC_DRBG fiasco was a disaster. But it lent believability to the suspicions that the NIST P-curves (secp256r1 / P-256, et. al.) got backdoored in the 90's. (Another topic to save for the reclassification thread...)

Its true that NIST violated the public's trust. Though since then, people have been watching their moves VERY closely. And they know this. Slipping in another backdoor at this point with this level of attention is nigh impossible. So I do have faith that MLKEM and the other PQ algs were approved in the interests of the public. Hence, right now I do trust X25519MLKEM768, and in a couple years from now once its been proven, I'd trust MLKEM on its own too.

As far as things go for this PR, I'd be happy to change the color codings to whatever you'd like. Then in a few weeks, we can have a separate PR that brings into alignment the outcome of the overall reclassification discussion.

[jtesta]

I pushed a few more commits to this PR. They do the following:

• Re-based the PR against master to make it merge-able again.
• Updated the compile-time #error to trigger when OpenSSL < 3.5 is detected (since this PR makes 3.5 a hard requirement).
• Removed the CCM8 downgrading logic (added in Flag CCM8 cipher suites as weak #321), as OpenSSL 3.5 now does it automatically.
• Moved some dynamic string operations into a static string buffer for simplicity.

[rbsec]

Great stuff - I'd forgotten about that bodge for CCM8, but nice not to have to do that any more.

Thinking about the colouring for the MLKEM groups (and ignoring the other colouring discussion as we've got a separate thread for that), I think I'm happy to keep them as yellow for now - but this is something that should be reviewed when:

• They're enabled by default in OpenSSL; or
• They're enabled by default in other significant projects like OpenSSH; or
• In a couple of years, if they've started to be more widely accepted and trusted.

Does that sound reasonable?

---

Also, from a little play around with openssl, it looks like we don't detect servers that only support the pure MLKEM groups.

If we support both the pure and hybrid groups, it works fine:

$ ./openssl s_server -key /tmp/key.pem -cert /tmp/cert.pem -tls1_3 -groups X25519MLKEM768:MLKEM768

$ ./sslscan 127.0.0.1:4433
[...]
TLSv1.3   enabled
[...]
  Server Key Exchange Group(s):
TLSv1.3  192 bits  MLKEM768
TLSv1.3  192 bits  X25519MLKEM768

But with just the MLKEM groups we don't see TLSv1.3, and no groups are detected (although openssl s_client connects fine if you explicitly give it that group).

$ ./openssl s_server -key /tmp/key.pem -cert /tmp/cert.pem -tls1_3 -groups MLKEM768

$ ./sslscan 127.0.0.1:4433
[...]
TLSv1.3   disabled

I'm not sure how likely we'd ever be to see this though? I suppose if someone wants a purely PQ server they might set it up like that.

[jtesta]

Thinking about the colouring for the MLKEM groups [...], I think I'm happy to keep them as yellow for now - but this is something that should be reviewed when: [...] Does that sound reasonable?

Yep, that sounds great!

But with just the MLKEM groups we don't see TLSv1.3

This is because we're not sending an MLKEM keyshare in the ClientHello. In this case, some servers won't respond with a ServerHello, but rather a HelloRetryRequest (see https://datatracker.ietf.org/doc/html/rfc8446#section-2.1). This effectively tells the client "please try again with another ClientHello, but this time, send a keyshare for MLKEM512/768/1024/etc." I thought this could be a good reason for us to fall back on OpenSSL's functions in #335: perhaps the built-in handshake logic can handle this for us. I never tested it, though.

OpenSSL's s_server, oddly, doesn't respond with a HelloRetryRequest; it simply raises a handshake alert and kills the connection. When the s_client connects to it, I see that the entire connection fails since the client isn't given any hints on what to try next. Hence, servers in the wild that only support non-default groups (like pure MLKEM) and don't issue HelloRetryRequests can't be enumerated unless we send ClientHellos with LOTS of keyshare data (800 bytes for MLKEM512, plus ~1KB for MLKEM768, plus ~1.5KB for MLKEM1024, plus keyshares for ALL the brainpool curves, curveSM2, SecP256r1MLKEM768, etc, etc, etc, etc!).

I suppose we could wait and see how often servers in the wild reject the default groups AND refuse to give hints on what to try next. If there are enough of them out there, we can certainly add another fallback mechanism to send a mega ClientHello.

[rbsec]

Yeah, I thought that we probably wouldn't want to bloat the ClientHello too much for such a weird edge case. Interesting that s_server is acting in a slightly weird way - but then I guess it is only a demo implementation rather than one that's meant to be fully robust.

So I think leaving things as they are is probably fine - if we start running into stuff in the wild that acts like this then we may need to rethink things a bit. But I doubt we will, and certainly not in the near future.

[rbsec]

@jtesta thanks for all your work on this - I've merged it into master and added a comment about the justification for flagging the pure MLKEM groups (mostly as a reminder to myself in future). I'll go through the stuff in #333 and look to get those changes made at some point as well, and then we can tag the whole lot as 2.2.0 and see what breaks.

Thanks again.

[jtesta]

Great, thanks!!