Determine that there is no forward progress with non blocking SSL socket

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Determine that there is no forward progress with non blocking SSL socket

Eran Borovik
Hi all,
My application is using non-blocking sockets to send and receive data. To avoid issues, my code guarantees that a specific socket is always owned by a specific thread, thus preventing any issues or races from concurrently running send and receive at the same time on the same socket.
I've read thoroughly the lists regarding the best way to use non blocking sockets with SSL. The common wisdom I gathered was:
-It is allowed to interleave SSL_read and SSL_write, meaning that SSL_read doesn't have to complete successfully before issuing SSL_write( this makes sense otherwise full-duplex doesn't work).
-SSL socket has a global state. So return value from one call negates a previous return value from other call.

My question is how to determine that forward progress isn't possible and to return to epoll.
Suppose I have a main-loop that deals with all the pending receives and sends for a specific socket. For simplicity, assume that I try to SSL_receive until I get some kind of an error (WANT_READ or WANT_WRITE). Then I try to issue SSL_send until I get error as well, but if I read correctly the group, the new error negates previous errors so prior receives must be retried.
When do I stop? what is the best way to actually determine there can be no more forward progress both on the send and the receive side, and epoll must be used?

I can think on the following "algorithm" but  I am not sure if it makes sense:
-Receive until you get an Error X.
-Send. If I receive a return value that isn't X, it means receive will have to be retried. If I immediately receive a return value that is X, it means that no more progress is possible and I need to wait with epoll.

Problem with the above algorithm, is that I am afraid that if the send buffer is full and there is nothing to receive, I will keep getting WANT_READ for receive, and WANT_WRITE for send until actual data arrives or can be sent which defeats the purpose of epoll.

I've been banging my head here for several days. Any help here will be much appreciated.

Thx,
Eran
Reply | Threaded
Open this post in threaded view
|

RE: Determine that there is no forward progress with non blocking SSL socket

Michael Wojcik
From: openssl-users [mailto:[hidden email]] On Behalf Of Eran Borovik
Sent: Monday, January 27, 2020 07:07

> When do I stop? what is the best way to actually determine there can
> be no more forward progress both on the send and the receive side, and
> epoll must be used?

...

> I am afraid that if the send buffer is full and there is nothing to receive,
> I will keep getting WANT_READ for receive, and WANT_WRITE for send until
> actual data arrives or can be sent which defeats the purpose of epoll.

The following is untested and off the top of my head.

Think of it this way. You have these possible states:

1. You don't want to do anything with the conversation. It can be closed.
2. You want to receive. There are three possible causes:
  - You'd like to receive data (or close/error indication) from the peer.
  - SSL_send returned WANT_READ.
  - Both of the above are true.
3. You want to send. There are three possible causes:
  - You have buffered outbound application data (or shutdown/close) which you
    haven't been able to send yet.
  - SSL_receive returned WANT_WRITE.
  - Both of the above are true.
4. You want to receive and to send.

So, keep track of the above using a per-conversation pair of want_to_send and want_to_receive variables and use the following algorithm:


   /* any time we buffer outbound data, set want_to_send = 1 */
   want_to_receive = 1 /* at start of conversation, we want data from peer */

   while want_to_send or want_to_receive
      {
      /* Wait until we can do something we want to do */
      do_recv = false, do_send = false
      if want_to_send and want_to_receive
         {
         epoll until socket is readable or writable
         do_recv = readable
         do_send = writable
         }
      else if want_to_send
         {
         epoll until socket is writable
         do_send = true
         }
      else if want_to_receive
         {
         epoll until socket is readable
         do_recv = true
         }

      /* Now perform I/O */
      if do_recv
         {
         SSL_read(...)
         if WANT_WRITE
            want_to_send = true
         else if WANT_READ
            want_to_receive = true /* couldn't send enough to clear WANT_WRITE */
         else if error
            ...
         else
            want_to_receive = false /* maybe, depending on application */

         process_received_data(...)
         }
      if do_send
         {
         SSL_write(...)
         if WANT_READ
            want_to_receive = true
         else if WANT_WRITE
            want_to_send = true /* couldn't send enough to clear WANT_WRITE */
         else if error
            ...
         else
            {
            update send buffer info
            if all buffered data sent
               want_to_send = false /* any pending WANT_WRITE should now be clear */
            }
         }
      }
   close conversation

Now, in practice, you probably always want to poll on readability even if you're not
expecting data from the peer, because you'll want to be notified of peer close. And
you may want to be able to break out of an epoll for readability only in order to send
data, depending on your application design and requirements. You can do that using a
timeout and polling the want_to_send state variable, or using an internal channel such
as a pipe or socket which the polling thread adds to the epoll readable-descriptor set,
and the sending thread writes a wakeup message to.

It's an open design question if you want to set want_to_receive to false in the do_recv
body, as I have it above, and then set it back to true if you decide you're not done
with the conversation in process_received_data; or leave it set to true and set it to
false only when you've received a close indication from the peer; or possibly some
other behavior that's appropriate for your application.


In brief: at the top of the loop, figure out if you want to try to send, receive, or
both. The types of I/O you want to do is the union of the application state and the
OpenSSL WANT_* state. Then poll for all the types of I/O you want, and when the socket
is available for any of them, attempt that type. Deal with any successful operation,
and loop.


In the "problem case" you described above, this code will discover that you want to
receive and to send, so it will first poll for either type of availability on the
socket. If the socket ends up readable, you'll do a receive, which may succeed or
may return WANT_WRITE or may indicate a peer close or may indicate an error. It could
also return WANT_READ if it's not able to receive enough TLS data to satisfy a pending
WANT_READ. In that case we keep want_to_receive set and go around the loop again.

And if the socket is writable (regardless of whether it's also readable), it will try
to send some of your application data; that will also try to handle any pending
WANT_WRITE condition. The result might be sending all the application data, sending
some or none of the application data, satisfying a WANT_WRITE condition, failing to
send enough to satisfy the WANT_WRITE, getting an error, etc. If we do send some
application data, we can assume any pending WANT_WRITE from the previous receive
operation was satisfied.

--
Michael Wojcik
Distinguished Engineer, Micro Focus

Reply | Threaded
Open this post in threaded view
|

RE: Determine that there is no forward progress with non blocking SSL socket

Michael Wojcik
> From: Eran Borovik [mailto:[hidden email]]
> Sent: Wednesday, January 29, 2020 07:32

Please respond to the list rather than directly to me, since the subject
may be of interest to other readers. I'm including the list in this
response.

> The only thing that still confuses me is that if I am reading the docs
> correctly,  SSL_read may return SSL_WANT_WRITE and SSL_write may return
> SSL_WANT_READ even when they don't encounter a blocking condition, but
> because of negotiation.

Right.

> Now, I use edge triggered polling with Epoll (EPOLLET), which means
> that if SSL_read/write decides to give me an WANT* status when the
> socket doesn't have a blocking condition, then epoll will never wake
> and I am stuck (unless I understand that this is the case and retry
> immediately). Is there a way to actually understand that there is a
> blocking condition in the socket from OpenSSL or do I need to use
> select with zero timeout to realize what is the correct condition of
> the socket after each time SSL gives me SSL_WANT*?

Yes, I think you need to test for readability / writability at some
point. You can do that immediately when you get a WANT_*, or you can
have your epoll time out periodically and test then if you have
pending I/O.

Personally, I'm leery of edge-triggered activation for this reason.
It's too easy to miss some race condition or other case where you
might end up blocked forever. I'd always have epoll time out so you
can do a level-poll of all sockets that have pending operations;
that turns a failure mode into one that simply has suboptimal
performance, at a small resource cost.

--
Michael Wojcik
Distinguished Engineer, Micro Focus


Reply | Threaded
Open this post in threaded view
|

Re: Determine that there is no forward progress with non blocking SSL socket

Eran Borovik
Hi,
Excellent, so I think I am finally understanding this correctly. And yes, we do have epoll timeouts to clean up all sort of missed race conditions and bugs (either in kernel or user mode).
For performance reasons, EDGE trigger is a must for my application.
What if instead of using select to understand the socket status (big hammer), I will simple retry the call?
For example, if SSL_read returns WANT_*, I do another SSL_read. On the low chance that this is a negotiation, there will be forward progress and I am fine. If not, I know that the socket is blocked.
It is too bad that I cannot differentiate between the two states without going to kernel space. I wish there were some way by just querying SSL. I see that there is a family of SSL_want* methods. Can these be of use?

Regards,
Eran.

On Wed, Jan 29, 2020 at 6:12 PM Michael Wojcik <[hidden email]> wrote:
> From: Eran Borovik [mailto:[hidden email]]
> Sent: Wednesday, January 29, 2020 07:32

Please respond to the list rather than directly to me, since the subject
may be of interest to other readers. I'm including the list in this
response.

> The only thing that still confuses me is that if I am reading the docs
> correctly,  SSL_read may return SSL_WANT_WRITE and SSL_write may return
> SSL_WANT_READ even when they don't encounter a blocking condition, but
> because of negotiation.

Right.

> Now, I use edge triggered polling with Epoll (EPOLLET), which means
> that if SSL_read/write decides to give me an WANT* status when the
> socket doesn't have a blocking condition, then epoll will never wake
> and I am stuck (unless I understand that this is the case and retry
> immediately). Is there a way to actually understand that there is a
> blocking condition in the socket from OpenSSL or do I need to use
> select with zero timeout to realize what is the correct condition of
> the socket after each time SSL gives me SSL_WANT*?

Yes, I think you need to test for readability / writability at some
point. You can do that immediately when you get a WANT_*, or you can
have your epoll time out periodically and test then if you have
pending I/O.

Personally, I'm leery of edge-triggered activation for this reason.
It's too easy to miss some race condition or other case where you
might end up blocked forever. I'd always have epoll time out so you
can do a level-poll of all sockets that have pending operations;
that turns a failure mode into one that simply has suboptimal
performance, at a small resource cost.

--
Michael Wojcik
Distinguished Engineer, Micro Focus