OpenSSL server packets get fragmented into 270 bytes per packet

Question

Problem
I'm trying to analyze the DTLS 1.2 handshake (over IPv6) with OpenSSL s_server and s_client by capturing it with Wireshark. The handshake works as intended but the server packets get fragmented into an extremely small size (270 bytes) for whatever reason.

This is the flow of the handshake with the packet size in parentheses:

Client                                        | Server
=========================================================================================
Client Hello (233)                            |
                                              | Hello Verify Request (110)
Client Hello (253)                            |
                                              | Server Hello, Certificate (Fragment) (270)
                                              |
                                              | Certificate (Fragment) (270)
                                              | Certificate (Fragment) (270)
                                              | Certificate (Fragment) (270)
                                              | Certificate (Fragment) (270)
                                              | Certificate (Fragment) (270)
                                              | Certificate (Fragment) (270)
                                              | Certificate (Fragment) (270)
                                              | Certificate (Fragment) (270)
                                              |
                                              | Certificate (Reassembled),
                                              | Server Key Exchange (Fragment) (270)
                                              | 
                                              | Server Key Exchange (Reassembled),
                                              | Certificate Request (Fragment) (270)
                                              |
                                              | Certificate Request (Reassembled),
                                              | Server Hello Done (235)
Certificate, Client Key Exchange,             |
Certificate Verify, Change Cipher Spec (1764) |
                                              | Change Cipher Spec,
                                              | Encrypted Handshake Message (129)
                                              :
                                              :

It seems that the server will not send any packets longer than 270 bytes. However, it can be seen clearly that the client does not have such a limitation and sends the client certificate in one packet.

How to Replicate the Behavior
Open terminal (Ubuntu 18.04) and use this command to start the server:

openssl s_server -dtls1_2 -6 -no_ticket -cipher ECDHE-ECDSA-AES128-CCM8 -key <server_private_key>.key.pem -cert <server_certificate>.cert.pem -CAfile <ca_certificate>.cert.pem -Verify 5 -verify_return_error -accept [::1]:4444 -debug

Open another terminal and start the client:

openssl s_client -dtls1_2 -6 -cipher ECDHE-ECDSA-AES128-CCM8 -cert <client_certificate>.cert.pem -key <client_private_key>.key.pem -CAfile <ca_certificate>.cert.pem -verify 5 -verify_return_error -connect [::1]:4444

The project requires a DTLS session over IPv6 without any tickets. The ciphersuite needs to be ECDHE-ECDSA-AES128-CCM8 and the client needs to be verified by the server.
To rule out any errors originating in the network, I assigned the loopback address to the server. The MTU of the loopback interface is 65536 and should not be the culprit.

Unsuccessful Attemps
I tried setting the flags max_send_frag, split_send_frags and read_buf to 9000 on the server and the client but this also didn't change anything. However, using TLS 1.2 this works without fragmenting the server certificate. Unfortunately, DTLS is required for our project.

What openssl version are you using? – Matt Caswell Jun 16 '20 at 16:59 — Matt Caswell, Jun 16 '20 at 16:59
Version: OpenSSL 1.1.1 11 Sep 2018 – afemis Jun 16 '20 at 17:07 — afemis, Jun 16 '20 at 17:07

score 2 · Accepted Answer · answered Jun 17 '20 at 10:21

This is caused by s_server being unsuccessful in querying the underlying MTU. The reason is that, by default, s_server doesn't "connect" the underlying socket to the client and so any attempts to query the MTU fail.

The answer to this problem is to use the "-listen" option to s_server. This causes s_server to pause the handshake after the initial ClientHello has occurred, discover the IP address of the client, and "connect" the underlying socket. MTU queries then succeed and you don't see the fragmentation. This option should perhaps be the default.

Thank you so much for the quick response! This fixed it and I can finally start to use the s_server to debug my own TLS client. And yes I would also expect this to be the default. — afemis, Jun 18 '20 at 05:28

OpenSSL server packets get fragmented into 270 bytes per packet

1 Answers1