Post-Quantum TLS 1.3: Which Hybrid KEM Configuration Minimizes Handshake Overhead in OpenSSL?

TLS 1.3 is being migrated to post-quantum key exchange because classical Diffie-Hellman and X25519 will not resist a large quantum computer. NIST has picked ML-KEM as its post-quantum key encapsulation mechanism [9]. Post-quantum key exchange and authentication with ML-KEM and ML-DSA will affect TLS 1.3 performance in the Web and other applications [9]. Studies so far have concentrated on the overhead of these quantum-resistant algorithms on TLS time-to-first-byte, that is, handshake time [9]. That focus makes sense for connection setup cost, but it leaves code size, a second concern named directly in the research question, almost unaddressed by the measurement literature gathered here.

Deployment data shows the transition is already underway but incomplete. A 2026 measurement study found 49.3% of domains support hybrid post-quantum key exchange mechanisms such as MLKEM768 with X25519, while 50.7% of domains continue to use classical key exchange [14]. The same study found 0% adoption of hybrid post-quantum certificates, leaving the authentication layer vulnerable to quantum-enabled attacks such as certificate forgery [14]. It also found 15.70% of domains, especially in critical sectors such as banking and government, still rely on TLS 1.2 [14]. This mixed state is why builders need concrete performance numbers now, not after migration is complete.

The security motivation is quantifiable. Symmetric key material at 128-bit classical security provides approximately 64 bits of effective security against a quantum attacker [14]. That halving is the reason KEM migration is treated as urgent rather than optional, and it frames why latency and size trade-offs of specific KEM configurations matter operationally, not just academically.

The research question asks specifically about OpenSSL 3.4+ and named configurations such as X25519Kyber768 and X25519MLKEM512. The available claims describe experiments on OpenSSL 3.x integrated with liboqs [1] [8] and on cross-platform benchmarks of related PQC families [12], but none of the sources tie their measurements to the exact OpenSSL 3.4+ version or code-size metric named in the question. A builder should read the rest of this note as an answer to the latency half of the question, with an explicit gap flagged on the code-size half.

Stated plainly and once, up front: no verified claim connects ML-KEM-512's packet-size advantage, or any KEM configuration's performance, to code size. Every claim discussing ML-KEM-512's advantage speaks only to packet size and handshake latency under network stress, never to compiled or linked code size, binary footprint, or library size [1]. No claim compares code size across ML-KEM-512, ML-KEM-768, Kyber768, or any hybrid combination. A reader who needs a code-size answer should treat this note as silent on that dimension; nothing here implies that a packet-size advantage translates into a code-size advantage, because no source makes that connection.

This matters for embedded and constrained deployments, where code size is often the binding constraint rather than latency. Since the literature reviewed here does not measure it, any procurement or engineering decision that depends on code size cannot be settled by these claims alone. The rest of this note therefore treats latency as the answerable half of the question and treats code size as an open, unaddressed question.

ML-KEM is one of NIST's postquantum algorithm picks for key exchange, alongside ML-DSA for authentication [9]. The experiments most relevant to OpenSSL performance evaluate ML-KEM algorithms directly and also a hybrid configuration, X25519+ML-KEM-768, combining the classical X25519 elliptic-curve exchange with the post-quantum ML-KEM-768 mechanism [1]. This hybrid design is common in deployment: hybrid post-quantum key exchange mechanisms observed in the wild, such as MLKEM768 with X25519, follow the same pattern of pairing a classical and a post-quantum primitive in one negotiation [14].

Other work in this space studies related but distinct KEM families. One study investigates the impact of incorporating post-quantum KEMs, specifically the standards CRYSTALS-Kyber and HQC, together with the candidate standard BIKE, into the TLS 1.3 handshake [8]. Another presents what it calls the most extensive cross-platform empirical evaluation to date of NIST-selected PQC algorithms, including CRYSTALS-Kyber and NTRU for KEMs, alongside BIKE as a code-based alternative, and CRYSTALS-Dilithium and Falcon for signatures [12]. These broaden the picture beyond ML-KEM alone but do not report figures for the exact X25519MLKEM512 or X25519Kyber768 labels named in the question; a builder mapping these families onto OpenSSL group names must do so with care, since the claims here do not state that Kyber and ML-KEM are interchangeable or identical.

The signature side of TLS 1.3 is a separate mechanism from key exchange but shares the same migration pressure. Dilithium 2 and Falcon 512 outperform RSA 4096 in TLS handshake time duration [13]. This is useful context for a builder assembling a full post-quantum TLS 1.3 stack, since key exchange and signature algorithm choices both contribute to total handshake cost, but it is a separate finding from the KEM comparisons that are the focus of this note.

The most detailed method for the numbers relevant to OpenSSL is the ARM64 study. Experiments were performed using OpenSSL 3.x integrated with liboqs [1]. The hybrid X25519+ML-KEM-768 configuration was one of the configurations evaluated [1]. 100 iterations were performed for each configuration [1]. Five network scenarios were used: loopback, LAN (10 ms RTT), WAN (50 ms RTT), and packet loss rates of 1% and 5% [1]. Read as five distinct conditions, loopback, LAN, WAN, 1% loss, and 5% loss, this matches the claim's stated count of five; a builder reproducing the setup should treat it that way rather than as four scenarios.

A related but separate study on similar ground investigates Kyber, HQC, and BIKE in TLS 1.3, and conducted a comprehensive experimental evaluation measuring handshake latency under emulated network conditions with varying packet loss probabilities [8]. This is methodologically parallel to the ARM64 study's network-stress design but covers a different KEM family, so its numbers should not be merged with the ML-KEM figures without care.

A third, larger benchmarking effort measured computational latency, memory utilization, key sizes, and protocol overhead across multiple security levels, NIST Levels 1, 3, and 5, in three distinct hardware environments and various network conditions [12]. This is the only claim in the set that names key sizes and memory utilization as measured variables, which is close to but still distinct from code size; it does not report a code-size figure for any OpenSSL configuration, so it does not close the gap flagged above.

A fourth study takes a different angle, decomposing the TLS handshake into stages and measuring effect sizes rather than raw times. This layered approach lets it isolate where in the handshake a KEM choice actually matters, rather than treating the handshake as one lump measurement; its central finding is discussed once, in full, in the next section.

The headline latency numbers come from the ARM64 study. ML-KEM algorithms introduce negligible computational overhead compared to the classic X25519 under low latency conditions [1]. Base-state 1-RTT handshake times for ML-KEM algorithms range from 11.3 to 13.3 ms [1]; this is the single authoritative range for base-state timing in this note, referenced from other sections rather than restated. Under packet loss, the gap widens: ML-KEM reached 180 ms with 5% packet loss, while X25519 reached 281 ms [1], so at the highest tested loss rate ML-KEM was faster in absolute terms, not merely comparable. Session restart times consistently reduced handshake latency for all algorithms tested [1], meaning that resumed connections were faster than fresh 1-RTT handshakes across the board, independent of which KEM was in use.

Within this ARM64 study, ML-KEM-512 showed the best performance, particularly due to its small packet size [1]. This is the one claim in the set that ranks configurations against each other on a named mechanism, and it is the closest the evidence comes to answering the latency half of the research question directly.

A separate, layered analysis decomposes the TLS 1.3 handshake into stages instead of treating it as one number, and this is the single most important qualifier on the ARM64 results above. Its central finding, stated once here and referenced rather than repeated elsewhere in this note: the TLS handshake exchange, the stage from ClientHello to Finished, is effectively algorithm-neutral, with all configurations tested, classical, hybrid, and pure ML-KEM, showing negligible effect sizes, Glass's Δ below 0.2 to 0.33 [4]. Consistent with that, no practically meaningful penalty or advantage attributable to the key exchange algorithm was found in the TLS handshake exchange [4]. Instead, the algorithm-sensitive cost is isolated to ClientHello construction specifically [4], not to the round-trip exchange as a whole.

This reframes the ARM64 numbers rather than contradicting them. The base-state range and packet-loss figures cited above [1] describe raw wall-clock handshake time under network stress; the layered study's negligible-effect-size finding [4] describes the statistical size of the algorithm-attributable difference within the handshake exchange stage, under presumably different conditions and metrics. The two are not measuring the same quantity, so a builder should not read one as overturning the other; rather, the ARM64 numbers show what happens under adverse network conditions specifically, while the layered analysis shows that averaged across configurations the exchange stage itself carries little algorithm-attributable cost, with the real sensitivity sitting upstream in ClientHello construction.

Put together, a practical reading is: if the deployment target is dominated by network stress, packet loss and RTT, the ARM64 numbers are the relevant ones and ML-KEM-512 is reported as the best performer there [1]. If the deployment target is a stable, low-loss network, the layered study's finding suggests the choice among tested configurations will not, by itself, produce a practically meaningful latency difference in the exchange stage [4], though the ClientHello construction cost is worth separate attention.

Handshake time is not the only place post-quantum overhead can show up. One study argues that studies so far have focused on the overhead of quantum-resistant algorithms on TLS time-to-first-byte, that is, handshake time [9], and sets out to fill a gap: it quantifies the impact of ML-KEM and ML-DSA on typical TLS 1.3 connections that transfer a few hundred KB from server to client, studying the slowdown in time-to-last-byte [9]. Its finding is that under stable network conditions, the impact of ML-KEM and ML-DSA on TLS 1.3 time-to-last-byte is lower than the impact on time-to-first-byte, and this impact diminishes as the transferred data increases [9]. For a builder, this means the handshake-latency numbers discussed above are most relevant to short-lived or handshake-dominated connections; for larger payload transfers, the same algorithms matter proportionally less to total connection time.

The broader benchmarking landscape also includes work that is wider in scope but not focused on OpenSSL specifically. A cross-platform empirical evaluation covers CRYSTALS-Kyber, NTRU, BIKE, CRYSTALS-Dilithium and Falcon, measuring computational latency, memory utilization, key sizes, and protocol overhead across NIST security Levels 1, 3, and 5 in three distinct hardware environments and various network conditions [12]. A further study investigates CRYSTALS-Kyber, HQC and BIKE specifically inside the TLS 1.3 handshake, with a comprehensive experimental evaluation of handshake latency under emulated packet loss [8]. Neither of these two studies is reported here with figures pinned to OpenSSL 3.4+ or to the named configurations X25519Kyber768 or X25519MLKEM512, so they should be read as context for the KEM landscape rather than as direct answers to this note's question.

On the authentication side, which interacts with handshake cost even though it is a separate mechanism, Dilithium 2 and Falcon 512 outperform RSA 4096 in TLS handshake time duration [13]. A builder assembling a full post-quantum TLS 1.3 configuration in OpenSSL needs to choose both a KEM and a signature algorithm, and this finding is the relevant one for the signature side, independent of which KEM is chosen.

Any latency or size finding is only useful if it maps onto what is deployable and what is actually deployed. The 2026 measurement study aimed to identify quantum-vulnerable cryptographic primitives, detect the presence of post-quantum or hybrid algorithms, and analyze the real-world deployment of TLS across different sectors [14]. Its headline numbers, already noted above, are that 49.3% of domains support hybrid post-quantum key exchange such as MLKEM768 with X25519, and 50.7% of domains still use classical key exchange only [14]. That split means a builder targeting broad compatibility must still support classical-only negotiation as a fallback for roughly half of observed domains, at least at the time of that measurement.

More concerning for anyone reasoning about long-term security rather than just latency: 0% adoption of hybrid post-quantum certificates was observed, leaving the authentication layer vulnerable to quantum-enabled attacks such as certificate forgery [14]. This means that even where hybrid key exchange is deployed, the certificate chain authenticating that exchange is not yet migrated anywhere in the measured set. A further 15.70% of domains, especially in critical sectors such as banking and government, still rely on TLS 1.2 [14], a protocol version that predates the hybrid PQC mechanisms discussed throughout this note entirely.

These figures do not measure OpenSSL configurations directly, and they say nothing about latency or code size. But they set the practical ceiling on why the latency question is being asked in the first place: hybrid key exchange is roughly half-deployed, hybrid authentication is not deployed at all, and a meaningful fraction of the ecosystem has not moved past TLS 1.2. A builder optimizing for the fastest hybrid KEM configuration should keep in mind that the config's latency advantage will only matter for the roughly half of connections that can currently negotiate it, and that the certificate layer remains unaddressed by the same migration.

The most important limit has already been stated near the top: no claim in this set measures code size for any OpenSSL KEM configuration, so the code-size half of the research question is unanswered by this evidence [1]. This is not a minor gap; it is one of the two quantities the question asks about, and a builder needing a code-size figure must look to sources outside this note.

Second, the ARM64 study's finding that ML-KEM-512 showed the best performance due to small packet size [1] is reported from a single study on one hardware platform, ARM64, with 100 iterations per configuration across five network scenarios [1]. It is the only claim in this set that directly ranks KEM configurations against each other by measured performance; it has not been cross-checked here against an independent replication on a different platform or OpenSSL point release.

Third, the layered study's finding that the handshake exchange is effectively algorithm-neutral [4] and the ARM64 study's finding of a 180 ms versus 281 ms gap under 5% packet loss [1] were produced by different research groups, likely under different exact conditions, and this note has not been given a claim that reconciles their metrics into one comparable scale. A builder should treat them as complementary evidence at different levels of granularity, exchange-stage effect size versus raw wall-clock time under stress, rather than as a single unified number.

Fourth, several of the broader benchmarking studies, covering Kyber, HQC, BIKE, NTRU, Dilithium and Falcon [8] [12] [13], are not reported here with figures pinned to the exact configuration names in the research question, X25519Kyber768 or X25519MLKEM512, or to OpenSSL 3.4+ specifically. Their relevance is contextual, not a direct answer. Finally, the deployment figures [14] describe internet-wide adoption at the time of that study and may not reflect current adoption; they also do not measure performance, only presence or absence of support.

1. Set up OpenSSL 3.x integrated with liboqs as the base stack, matching the environment in which the reference latency numbers were produced [1]; this is a prerequisite for reproducing or extending those figures rather than guessing at them.
2. Enumerate the KEM configurations to test: at minimum, plain X25519 as the classical baseline, a pure ML-KEM configuration, and the hybrid X25519+ML-KEM-768 configuration, since this hybrid was one of the configurations directly evaluated in the source experiments [1].
3. Instrument the handshake to record 1-RTT establishment time separately from session-restart (resumption) time, because session restart times consistently reduced handshake latency for all algorithms in the reference study [1], and mixing fresh and resumed handshakes in one measurement will blur the comparison.
4. Build five network-condition test harnesses matching the reference design: loopback, LAN at 10 ms RTT, WAN at 50 ms RTT, 1% packet loss, and 5% packet loss [1]. Confirm your harness produces exactly five distinct conditions to be a faithful reproduction.
5. Run 100 iterations per configuration per network condition, matching the reference protocol [1], and record the full distribution, not just the mean, so outliers under packet loss are visible.
6. Compare each KEM configuration's base-state performance against the reference range for ML-KEM base-state 1-RTT handshakes [1] as a sanity check that the test harness is behaving consistently with published figures before drawing new conclusions.
7. Under the 5% packet-loss condition specifically, check whether ML-KEM-family configurations stay closer to the reference figures reported for ML-KEM versus X25519 under that condition [1]; a configuration performing far outside both figures signals a harness or environment difference that needs investigating before comparing configurations to each other.
8. Separately, instrument and measure ClientHello construction time on its own, distinct from full handshake exchange time, since the algorithm-sensitive cost sits in ClientHello construction rather than the exchange stage itself [4]. Skipping this separation is a common failure point: measuring only total handshake time will mask where any real algorithm-dependent cost actually lives.
9. If code size matters for the deployment target, treat it as a separate, unaddressed measurement task: compile the OpenSSL/liboqs build for each configuration and record binary or library size directly. No claim in this note provides a baseline or expected range for this, so any number produced here is new work, not a reproduction of prior findings.
10. If the deployment involves larger payload transfers rather than short handshake-dominated connections, also measure time-to-last-byte for a representative transfer of a few hundred KB, since one study found the post-quantum impact on time-to-last-byte is lower than on time-to-first-byte and diminishes as transferred data increases [9]; this changes which configuration matters most depending on typical connection size in the target deployment.
11. Report results with explicit network condition, iteration count, and hardware platform attached to every number, following the pattern of the reference studies [1] [12], so that any reader can judge whether a result generalizes beyond the tested environment.

for config in [X25519, ML-KEM-512, ML-KEM-768, X25519+ML-KEM-768]:
  for scenario in [loopback, LAN_10ms, WAN_50ms, loss_1pct, loss_5pct]:
    for i in 1..100:
      measure(ClientHello_construction_time)
      measure(full_handshake_time)
      measure(session_restart_time)
    record distribution, not just mean
compare each config's numbers against reference ranges [1]
if code_size_required: compile and measure binary size separately (no reference baseline exists)