API SSL_Connect fails and always returns SSL_ERROR_WANT_READ causes infinite loop in application

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

API SSL_Connect fails and always returns SSL_ERROR_WANT_READ causes infinite loop in application

mahesh gs
Hi All,

We have application that provide DTLS security for SCTP connections. During our testing we found that API "SSL_connect " fail and always returns SSL_ERROR_WANT_READ which causes infinite loop in the application.

Scenario:

1) On Server side "Client Certificate Request" is enabled by setting the SSL context as shown below

    SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER|SSL_VERIFY_FAIL_IF_NO_PEER_CERT, NULL);

2) On client side we have not configured the public certificate.


Logs:

[10/14/0117 15:05:06]         F42C2700 Link-2 (SSL_accept) Failed to accept new connection,  Socket Id 65, Return Value 1
[10/14/0117 15:05:06]         F42C2700 Link-2 SSL File : ssl/statem/statem_srvr.c , Line number : 2882 , Linux Error Code 0
[10/14/0117 15:05:06]         F26B7700 Link-1 SSL_connect() fails to connect need to retry, returned error code 2 , retry ? true
[10/14/0117 15:05:06]         F26B7700 Link-1 SSL_connect() fails to connect need to retry, returned error code 2 , retry ? true
[10/14/0117 15:05:06]         F26B7700 Link-1 SSL_connect() fails to connect need to retry, returned error code 2 , retry ? true

<<< SSL_connect() always returns error code 2 that leeds to infinite loop in application >>>

Attaching PCAP file for your reference. 

Thanks,
Mahesh G S


--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

connect.pcap (84K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: API SSL_Connect fails and always returns SSL_ERROR_WANT_READ causes infinite loop in application

Graham Leggett
On 14 Nov 2017, at 12:00 PM, mahesh gs <[hidden email]> wrote:

We have application that provide DTLS security for SCTP connections. During our testing we found that API "SSL_connect " fail and always returns SSL_ERROR_WANT_READ which causes infinite loop in the application.

Are you properly handling that SSL_ERROR_WANT_READ, or are you ignoring it?

The message isn’t an error (the symbol was misnamed), it just means openssl is asking you permission to read. If your code is saying "yes openssl you may read" when you actually aren’t ready you’ll end up in an infinite loop.

Regards,
Graham


--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: API SSL_Connect fails and always returns SSL_ERROR_WANT_READ causes infinite loop in application

mahesh gs
Hi,

As per the suggestion from openssl documentation whenever the SSL API returns SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE, The calling process then must repeat the call after taking appropriate action to satisfy the needs of SSL_connect().

I am copying the code bits here, 

  do
  {
/* Clear openssl error queue */
ERR_clear_error();

/* Initiate SSL Handshake */
aRetValue = SSL_connect(ivSSL);

if (aRetValue <= 0)
{
  aTlsError = SSL_get_error(ivSSL, aRetValue);
  
  switch(aTlsError)
  {
case SSL_ERROR_WANT_READ:  
case SSL_ERROR_WANT_WRITE:
{
  /* Select on the socket for read/write events */
  retry = pollSocketForEvents(aTlsError);            --------------> Function is copied below

  /* Nothing to do, retry to connect again*/
  LOGG_DBUG(Logger::M3UA_LOG,"Link-%d SSL_connect() fails to connect "
  "need to retry, returned error code %d , retry ? %s", ivLink->getLinkId(), aTlsError, retry?"true":"false");
}
break;

case SSL_ERROR_SYSCALL:

if (EWOULDBLOCK == errno || EAGAIN == errno)
{
  /* Nothing to do, retry to connect again */
}
else
{
  int aRet = ERR_get_error_line(&aFile, &aLine);
  
  LOGG_DBUG(Logger::M3UA_LOG,"Link-%d SSL File : %s , Line number : %d , "
"Socket Id %d, Linux Error Code %d",ivLink->getLinkId(), aFile, aLine, getFd(), errno);
  
  LOGG_DBUG(Logger::M3UA_LOG,"Link-%d SSL_connect () :: Result Code : %d ",ivLink->getLinkId(), aTlsError);

  retry = false;
}

break;

default:
{
  int aRet = ERR_get_error_line(&aFile, &aLine);
  
  LOGG_DBUG(Logger::M3UA_LOG,"Link-%d (SSL_connect) Failed to connect to server, "
" Socket Id %d, Return Value %d ", ivLink->getLinkId(), getFd(), aTlsError);
  
  LOGG_DBUG(Logger::M3UA_LOG,"Link-%d SSL File : %s , Line number : %d , Linux Error Code %d",ivLink->getLinkId(), aFile, aLine, errno);

  retry = false;
}
break;
  }    
}
  }while (aRetValue != 1 && retry != false);


bool TlsAssociation::pollSocketForEvents(long aTlsError)
{
/* This function is to implement the SSL Socket call behaviour
  
fd_set readFds, writeFds;
struct timeval timeout;
int retValue;

int nfds = getFd();

FD_ZERO (&readFds);
FD_ZERO (&writeFds);
FD_SET(nfds, &readFds);
FD_SET(nfds, &writeFds);

/* Wait for 5 Seconds */
timeout.tv_usec = 0;
timeout.tv_sec = 5;

if (SSL_ERROR_WANT_READ == aTlsError)
{
retValue = select(nfds + 1, &readFds, NULL, NULL, &timeout);
if (retValue <= 0)
{
// Timeout or error just return failure
return false;
}
}

if (SSL_ERROR_WANT_WRITE == aTlsError)
{
retValue = select(nfds + 1, NULL, &writeFds, NULL, &timeout);
if (retValue <= 0)
// Timeout or error just return failure
return false;
}
}

return true;
}



Thanks,
Mahesh G S



On Tue, Nov 14, 2017 at 4:01 PM, Graham Leggett <[hidden email]> wrote:
On 14 Nov 2017, at 12:00 PM, mahesh gs <[hidden email]> wrote:

We have application that provide DTLS security for SCTP connections. During our testing we found that API "SSL_connect " fail and always returns SSL_ERROR_WANT_READ which causes infinite loop in the application.

Are you properly handling that SSL_ERROR_WANT_READ, or are you ignoring it?

The message isn’t an error (the symbol was misnamed), it just means openssl is asking you permission to read. If your code is saying "yes openssl you may read" when you actually aren’t ready you’ll end up in an infinite loop.

Regards,
Graham


--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users



--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: API SSL_Connect fails and always returns SSL_ERROR_WANT_READ causes infinite loop in application

Matt Caswell-2


On 14/11/17 10:44, mahesh gs wrote:

> case SSL_ERROR_SYSCALL:
>
> if (EWOULDBLOCK == errno || EAGAIN == errno)
> {
>   /* Nothing to do, retry to connect again */
> }

This doesn't look right. If SSL_connect() fails due to an NBIO event
then you should get SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE back. If
you get SSL_ERROR_SYSCALL then something bad happened and you should not
retry. Could you add some logging here? I'm wondering whether you are
ending up here but missing it and looping around again.

Matt


--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: API SSL_Connect fails and always returns SSL_ERROR_WANT_READ causes infinite loop in application

mahesh gs
Hi Matt,

Thanks for the response,

I added a log as suggested by you. I don't see the call entering the above mentioned code block.

Logs on server side:

[10/15/0117 10:34:43]         803F1700 Link-2 (SSL_accept) Failed to accept new connection,  Socket Id 65, Return Value 1
[10/15/0117 10:34:43]         803F1700 Link-2 SSL File : ssl/statem/statem_srvr.c , Line number : 2882 , Linux Error Code 0

Logs on client side:

[10/15/0117 10:34:43]         7DDE1700 Link-1 SSL_connect() fails to connect need to retry, returned error code 2 , retry ? true
[10/15/0117 10:34:43]         7DDE1700 Link-1 SSL_connect() fails to connect need to retry, returned error code 2 , retry ? true
[10/15/0117 10:34:43]         7DDE1700 Link-1 SSL_connect() fails to connect need to retry, returned error code 2 , retry ? true
[10/15/0117 10:34:43]         7DDE1700 Link-1 SSL_connect() fails to connect need to retry, returned error code 2 , retry ? true
[10/15/0117 10:34:43]         7DDE1700 Link-1 SSL_connect() fails to connect need to retry, returned error code 2 , retry ? true
[10/15/0117 10:34:43]         7DDE1700 Link-1 SSL_connect() fails to connect need to retry, returned error code 2 , retry ? true


We observe from wireshark capture, client sending out the certificate with length = 0 (because we have not configured the public key on client side) and also server sends handshake failure "Alert" to client. Why does client respond with "Client key exchange" even if the the handshake failure alert is sent from server?

Openssl version used is 01.01.00g. I am also attaching the latest pcap file for your reference.


On Tue, Nov 14, 2017 at 4:35 PM, Matt Caswell <[hidden email]> wrote:


On 14/11/17 10:44, mahesh gs wrote:

> case SSL_ERROR_SYSCALL:
>
> if (EWOULDBLOCK == errno || EAGAIN == errno)
> {
>   /* Nothing to do, retry to connect again */
> }

This doesn't look right. If SSL_connect() fails due to an NBIO event
then you should get SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE back. If
you get SSL_ERROR_SYSCALL then something bad happened and you should not
retry. Could you add some logging here? I'm wondering whether you are
ending up here but missing it and looping around again.

Matt


--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users


--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

4.pcap (18K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: API SSL_Connect fails and always returns SSL_ERROR_WANT_READ causes infinite loop in application

Matt Caswell-2


On 17/11/17 06:42, mahesh gs wrote:
>  Why
> does client respond with "Client key exchange" even if the the handshake
> failure alert is sent from server?

The client will send its entire flight of messages before it attempts to
read anything from the server. So, in this case, the ClientKeyExchange
message is still sent because the client hasn't read the alert yet.

Matt

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: API SSL_Connect fails and always returns SSL_ERROR_WANT_READ causes infinite loop in application

mahesh gs
Hi Matt,

Thanks for the response.

We debugged through openssl code to get to know the reason why client is not reading "SSL Alert".

Once the "ClientKeyExchange" is sent openssl trying to send out the "ChangeCipherSpec"  message which is creating the problem. 

The pre-work function for "ChangeCipherSpec" enables SCTP dry event and wait for dry event notification.

Inline image 1


In this scenario, dry notification is never sent from SCTP. "dtls_wait_for_dry" always returns "WORK_MORE_A". Hereafter flow never enters "read_state_machine" where alert is to be red.This causes SSL_Connect to be in infinite loop.


Thanks,
Mahesh G S

On Fri, Nov 17, 2017 at 3:36 PM, Matt Caswell <[hidden email]> wrote:


On 17/11/17 06:42, mahesh gs wrote:
>  Why
> does client respond with "Client key exchange" even if the the handshake
> failure alert is sent from server?

The client will send its entire flight of messages before it attempts to
read anything from the server. So, in this case, the ClientKeyExchange
message is still sent because the client hasn't read the alert yet.

Matt

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users


--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: API SSL_Connect fails and always returns SSL_ERROR_WANT_READ causes infinite loop in application

mahesh gs
Hi,

We were able to further localize this problem and found the problem is with the function "BIO_dgram_sctp_wait_for_dry". In this function after enabling the "sctp_sender_dry_event" we are trying to do MSG_PEEK to peek to the message at SCTP layer, in this case the size of the message waiting in the lower layer is 15 which size is exactly the size of "Handshake Alert" that is received from the server and waiting to be read. Dry event is never read from the lower layer that causes the SUB_STATE_ERROR and intern causes the SSL_Connect to loop in application.

Current version of openssl we are using is 01.01.00g.

We have tested and able to reproduce this issue with the OPENSSL 01.00.02k version that is packaged with RHEL 7.4 as well.


Thanks,
Mahesh G S

On Mon, Nov 20, 2017 at 4:42 PM, mahesh gs <[hidden email]> wrote:
Hi Matt,

Thanks for the response.

We debugged through openssl code to get to know the reason why client is not reading "SSL Alert".

Once the "ClientKeyExchange" is sent openssl trying to send out the "ChangeCipherSpec"  message which is creating the problem. 

The pre-work function for "ChangeCipherSpec" enables SCTP dry event and wait for dry event notification.

Inline image 1


In this scenario, dry notification is never sent from SCTP. "dtls_wait_for_dry" always returns "WORK_MORE_A". Hereafter flow never enters "read_state_machine" where alert is to be red.This causes SSL_Connect to be in infinite loop.


Thanks,
Mahesh G S

On Fri, Nov 17, 2017 at 3:36 PM, Matt Caswell <[hidden email]> wrote:


On 17/11/17 06:42, mahesh gs wrote:
>  Why
> does client respond with "Client key exchange" even if the the handshake
> failure alert is sent from server?

The client will send its entire flight of messages before it attempts to
read anything from the server. So, in this case, the ClientKeyExchange
message is still sent because the client hasn't read the alert yet.

Matt

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users



--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: API SSL_Connect fails and always returns SSL_ERROR_WANT_READ causes infinite loop in application

Matt Caswell-2
Sounds like a bug. Can you raise this as an issue on github?

https://github.com/openssl/openssl/issues

Thanks

Matt


On 21/11/17 08:53, mahesh gs wrote:

> Hi,
>
> We were able to further localize this problem and found the problem is
> with the function "BIO_dgram_sctp_wait_for_dry". In this function after
> enabling the "sctp_sender_dry_event" we are trying to do MSG_PEEK to
> peek to the message at SCTP layer, in this case the size of the message
> waiting in the lower layer is 15 which size is exactly the size of
> "Handshake Alert" that is received from the server and waiting to be
> read. Dry event is never read from the lower layer that causes the
> SUB_STATE_ERROR and intern causes the SSL_Connect to loop in application.
>
> Current version of openssl we are using is 01.01.00g.
>
> We have tested and able to reproduce this issue with the OPENSSL
> 01.00.02k version that is packaged with RHEL 7.4 as well.
>
>
> Thanks,
> Mahesh G S
>
> On Mon, Nov 20, 2017 at 4:42 PM, mahesh gs <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hi Matt,
>
>     Thanks for the response.
>
>     We debugged through openssl code to get to know the reason why
>     client is not reading "SSL Alert".
>
>     Once the "ClientKeyExchange" is sent openssl trying to send out the
>     "ChangeCipherSpec"  message which is creating the problem. 
>
>     The pre-work function for "ChangeCipherSpec" enables SCTP dry event
>     and wait for dry event notification.
>
>     Inline image 1
>
>
>     In this scenario, dry notification is never sent from SCTP.
>     "dtls_wait_for_dry" always returns "WORK_MORE_A". Hereafter flow
>     never enters "read_state_machine" where alert is to be red.This
>     causes SSL_Connect to be in infinite loop.
>
>
>     Thanks,
>     Mahesh G S
>
>     On Fri, Nov 17, 2017 at 3:36 PM, Matt Caswell <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>
>
>         On 17/11/17 06:42, mahesh gs wrote:
>         >  Why
>         > does client respond with "Client key exchange" even if the the handshake
>         > failure alert is sent from server?
>
>         The client will send its entire flight of messages before it
>         attempts to
>         read anything from the server. So, in this case, the
>         ClientKeyExchange
>         message is still sent because the client hasn't read the alert yet.
>
>         Matt
>
>         --
>         openssl-users mailing list
>         To unsubscribe:
>         https://mta.openssl.org/mailman/listinfo/openssl-users
>         <https://mta.openssl.org/mailman/listinfo/openssl-users>
>
>
>
>
>
--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: API SSL_Connect fails and always returns SSL_ERROR_WANT_READ causes infinite loop in application

mahesh gs
Hi Matt,

I have raised bug.


Thanks,
Mahesh G S

On Tue, Nov 21, 2017 at 3:26 PM, Matt Caswell <[hidden email]> wrote:
Sounds like a bug. Can you raise this as an issue on github?

https://github.com/openssl/openssl/issues

Thanks

Matt


On 21/11/17 08:53, mahesh gs wrote:
> Hi,
>
> We were able to further localize this problem and found the problem is
> with the function "BIO_dgram_sctp_wait_for_dry". In this function after
> enabling the "sctp_sender_dry_event" we are trying to do MSG_PEEK to
> peek to the message at SCTP layer, in this case the size of the message
> waiting in the lower layer is 15 which size is exactly the size of
> "Handshake Alert" that is received from the server and waiting to be
> read. Dry event is never read from the lower layer that causes the
> SUB_STATE_ERROR and intern causes the SSL_Connect to loop in application.
>
> Current version of openssl we are using is 01.01.00g.
>
> We have tested and able to reproduce this issue with the OPENSSL
> 01.00.02k version that is packaged with RHEL 7.4 as well.
>
>
> Thanks,
> Mahesh G S
>
> On Mon, Nov 20, 2017 at 4:42 PM, mahesh gs <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hi Matt,
>
>     Thanks for the response.
>
>     We debugged through openssl code to get to know the reason why
>     client is not reading "SSL Alert".
>
>     Once the "ClientKeyExchange" is sent openssl trying to send out the
>     "ChangeCipherSpec"  message which is creating the problem. 
>
>     The pre-work function for "ChangeCipherSpec" enables SCTP dry event
>     and wait for dry event notification.
>
>     Inline image 1
>
>
>     In this scenario, dry notification is never sent from SCTP.
>     "dtls_wait_for_dry" always returns "WORK_MORE_A". Hereafter flow
>     never enters "read_state_machine" where alert is to be red.This
>     causes SSL_Connect to be in infinite loop.
>
>
>     Thanks,
>     Mahesh G S
>
>     On Fri, Nov 17, 2017 at 3:36 PM, Matt Caswell <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>
>
>         On 17/11/17 06:42, mahesh gs wrote:
>         >  Why
>         > does client respond with "Client key exchange" even if the the handshake
>         > failure alert is sent from server?
>
>         The client will send its entire flight of messages before it
>         attempts to
>         read anything from the server. So, in this case, the
>         ClientKeyExchange
>         message is still sent because the client hasn't read the alert yet.
>
>         Matt
>
>         --
>         openssl-users mailing list
>         To unsubscribe:
>         https://mta.openssl.org/mailman/listinfo/openssl-users
>         <https://mta.openssl.org/mailman/listinfo/openssl-users>
>
>
>
>
>
--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users


--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users