WebRTC DTLS-SRTP OpenSSL Server Handshake Failure

Question

Here is my procedure in OpenSSL Server Mode,

Initialization Part of SSL and BIO variables:

map<int, SSL> m_SSLMap;
map<int, BIO> m_BioWriteMap;
map<int, BIO> m_BioReadMap;
int InitializeServerNegotiationMode(int iFd)
{
        SSL *pServSslFd;
        BIO *pWb, *pRb;

        pServSslFd = SSL_new(m_pCtx);
        assert(pServSslFd);

        if ( SSL_version(pServSslFd) == DTLS1_VERSION)
        {
            pWb = BIO_new(BIO_s_mem());
            pRb = BIO_new(BIO_s_mem());
            assert(pWb);
            assert(pRb);
            SSL_set_bio(pServSslFd, pRb, pWb);
            SSL_set_accept_state(pServSslFd);
        }
        m_SSLMap[iFd] = *pServSslFd;
        m_BioReadMap[iFd] = *pRb;
        m_BioWriteMap[iFd] = *pWb;

        return INITIALIZATION_SUCCESS;
 }

Server Mode Negotiation Operations when DTLS data comes to the server:

int ServerModeDTLSNegotiation(int iChannel, const char *pBuff, const int iLen, int iFd)
{

    SSL *pServSslFd;
    BIO *pRbio;
    BIO *pWbio;
    pServSslFd = &m_SSLMap[iFd];
    pRbio = &m_BioReadMap[iFd];
    pWbio = &m_BioWriteMap[iFd];


    char buff[4096];
    memset(buff, 0, strlen(buff));

    BIO_write(pRbio, pBuff, iLen);

    if(!SSL_is_init_finished(pServSslFd))
    {
        int iRet = SSL_do_handshake(pServSslFd);
    }

    int iNewLen = BIO_read(pWbio, buff, 2048);
    if(iNewLen>0)
    {
        char *pNewData = new char[iNewLen+1];
        for(int i=0;i<iNewLen;i++)
        pNewData[i] = buff[i];
         m_pEventHandler->SendReply(iChannel, (unsigned char *)pNewData, iNewLen);
    }
    else
    {
         printf("[DTLS]:: HandShaking Response failed for this data, 
         return -1;
    }
    return NEGOTIATION_SUCCESS;

}

Here I am attaching Wireshark TCP-Dump for better monitoring about the issue.

https://www.dropbox.com/s/quidcs6gilnvt2o/WebRTC%20DTLS%20Handshake%20Failure.pcapng?dl=0

Now, I am confident about my initialization of SSL_CTX variable. Because, Sometimes Handshake successfully negotiate for every port. But sometimes Handshake fails for one or two port. I am working for 5 days to solve WebRTC DTLS Server Mode Negotiation for Google Chrome. But I haven't found the root cause for this problem.

Would appreciate if you can take a look at this [post](https://stackoverflow.com/questions/62627063/clarification-needed-on-openssl-api-in-implementing-dtls-1-2-server) — asinix, Jun 28 '20 at 20:25
Do you have to call SSL_do_handshake(pServSslFd); in server mode? I think that before a handshake, the server should imply, or you should check if handshake is completed, and conclude that the data is part of a handshake message? — asinix, Jun 29 '20 at 04:29

score 1 · Accepted Answer · answered Oct 16 '17 at 09:32

1

The link for TCP-Dump is not working. Anyway, it seems your solution should work.

As it's a server program, it's definitely multi threaded. But it's really dangerous to initialize SSL variables or to perform handshake procedure without locking. In that case so many things can happen if these two methods are processed by multiple thread.

My suggestion is to add locking mechanism for these methods.

answered Oct 16 '17 at 09:32

Light Yagami

26
2

Actually, I solved this a long time ago. And yes, you are right. I solved this problem adding mutex lock. – RajibTheKing Oct 16 '17 at 09:45
@Light Yagami, Can you please take a look at this [post](https://stackoverflow.com/questions/62627063/clarification-needed-on-openssl-api-in-implementing-dtls-1-2-server) – asinix Jun 28 '20 at 20:23

WebRTC DTLS-SRTP OpenSSL Server Handshake Failure

1 Answers1