Close TCP socket after SSL_clear()?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Close TCP socket after SSL_clear()?

Charles Mills

On Windows, for a new session, I am issuing a Windows accept() followed by SSL_new(), SSL_set_fd() and so forth.

 

When the session sees some sort of an abnormal receive condition, I am doing

 

       int retCode = SSL_get_shutdown(sessionSSL);

       if ( retCode & SSL_RECEIVED_SHUTDOWN )

       {

              SSL_shutdown(sessionSSL);

       }

       else

       {

              SSL_clear(sessionSSL);

       }

 

Questions:

 

1.       Do I also need to do a closesocket() (equivalent to UNIX close()) on the Windows socket?

2.       Does anyone want to critique the above logic in any other way?

 

The code basically “works” but I see evidence that a Windows TCP session is still open following an SSL error.

 

Thanks,

 

Charles Mills


--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: Close TCP socket after SSL_clear()?

Karl Denninger


On 1/10/2019 17:07, Charles Mills wrote:

On Windows, for a new session, I am issuing a Windows accept() followed by SSL_new(), SSL_set_fd() and so forth.

 

When the session sees some sort of an abnormal receive condition, I am doing

 

       int retCode = SSL_get_shutdown(sessionSSL);

       if ( retCode & SSL_RECEIVED_SHUTDOWN )

       {

              SSL_shutdown(sessionSSL);

       }

       else

       {

              SSL_clear(sessionSSL);

       }

 

Questions:

 

1.       Do I also need to do a closesocket() (equivalent to UNIX close()) on the Windows socket?

2.       Does anyone want to critique the above logic in any other way?

 

The code basically “works” but I see evidence that a Windows TCP session is still open following an SSL error.

 

Thanks,

 

Charles Mills


Are you sure you want to use SSL_clear() in the first place?  It retains the session's settings which is only useful if the *exact* same peer is going to reconnect on the same SSL object.  If a *different* peer connects there's a decent shot that the connection will fail.

You also likely want to call SSL_shutdown(connection) again IF the first call returns zero; the first one sends a notification and if the other end hasn't closed yet returns zero.  The second waits for a termination, either normal notification or abnormal, from the other end.

    if (!SSL_shutdown(connection)) {
        SSL_shutdown(connection)
    }

The underlying handle is still open at the OS level after this, so on Unix anyway you want to notify the OS that the socket is invalid for further I/O and then close it.

Code snippet (took_error is a flag that says "this connection is no longer needed", it's could be either an error in the higher level code or a "we're all done, let this connection go" indication):

                if (slave_socket[x].took_error) {
                    slave_socket[x].connected = 0;  /* Connection is void */
                    if (slave_socket[x].ssl_fd != NULL) { /* If there's a valid SSL connection */
                        if (!SSL_shutdown(slave_socket[x].ssl_fd)) {
                            SSL_shutdown(slave_socket[x].ssl_fd);
                        }
                        SSL_free(slave_socket[x].ssl_fd);
                        slave_socket[x].ssl = 0; /* We are not in SSL mode */
                    }
                    shutdown(slave_socket[x].fd, SHUT_RDWR);
                    close(slave_socket[x].fd);

                    ..... Clean up the rest of the things you need to do when the connection ends

Since the next connection may come from a different peer I do not use SSL_clear but rather SSL_free.

The call to shutdown() tells the OS to send any data queued on the socket, wait for an ACK and then send FIN.

--
Karl Denninger
[hidden email]
The Market Ticker
[S/MIME encrypted email preferred]

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Close TCP socket after SSL_clear()?

Charles Mills

@Karl, thanks, I’m not sure of anything. This was my first OpenSSL project and I just hacked on it until it “worked.” It’s been working for years but now we are seeing a re-connection error.

 

So, it sounds like

 

·         Do the SSL_shutdown() a second time if it returns 0.

·         Lose the SSL_clear()

·         There is an SSL_free() in there following the snippet I pasted – leave it in there

·         Clean up the underlying socket appropriately. Looks like perhaps shutdown(socket, SD_BOTH) is the Windows equivalent of SHUT_RDWR – followed by closesocket()

 

Thanks again!

 

Charles

 

From: openssl-users [mailto:[hidden email]] On Behalf Of Karl Denninger
Sent: Friday, January 11, 2019 10:04 AM
To: [hidden email]
Subject: Re: [openssl-users] Close TCP socket after SSL_clear()?

 

 

On 1/10/2019 17:07, Charles Mills wrote:

On Windows, for a new session, I am issuing a Windows accept() followed by SSL_new(), SSL_set_fd() and so forth.

 

When the session sees some sort of an abnormal receive condition, I am doing

 

       int retCode = SSL_get_shutdown(sessionSSL);

       if ( retCode & SSL_RECEIVED_SHUTDOWN )

       {

              SSL_shutdown(sessionSSL);

       }

       else

       {

              SSL_clear(sessionSSL);

       }

 

Questions:

 

1.       Do I also need to do a closesocket() (equivalent to UNIX close()) on the Windows socket?

2.       Does anyone want to critique the above logic in any other way?

 

The code basically “works” but I see evidence that a Windows TCP session is still open following an SSL error.

 

Thanks,

 

Charles Mills


 

Are you sure you want to use SSL_clear() in the first place?  It retains the session's settings which is only useful if the *exact* same peer is going to reconnect on the same SSL object.  If a *different* peer connects there's a decent shot that the connection will fail.

You also likely want to call SSL_shutdown(connection) again IF the first call returns zero; the first one sends a notification and if the other end hasn't closed yet returns zero.  The second waits for a termination, either normal notification or abnormal, from the other end.

    if (!SSL_shutdown(connection)) {
        SSL_shutdown(connection)
    }

The underlying handle is still open at the OS level after this, so on Unix anyway you want to notify the OS that the socket is invalid for further I/O and then close it.

Code snippet (took_error is a flag that says "this connection is no longer needed", it's could be either an error in the higher level code or a "we're all done, let this connection go" indication):

                if (slave_socket[x].took_error) {
                    slave_socket[x].connected = 0;  /* Connection is void */
                    if (slave_socket[x].ssl_fd != NULL) { /* If there's a valid SSL connection */
                        if (!SSL_shutdown(slave_socket[x].ssl_fd)) {
                            SSL_shutdown(slave_socket[x].ssl_fd);
                        }
                        SSL_free(slave_socket[x].ssl_fd);
                        slave_socket[x].ssl = 0; /* We are not in SSL mode */
                    }
                    shutdown(slave_socket[x].fd, SHUT_RDWR);
                    close(slave_socket[x].fd);

                    ..... Clean up the rest of the things you need to do when the connection ends

Since the next connection may come from a different peer I do not use SSL_clear but rather SSL_free.

The call to shutdown() tells the OS to send any data queued on the socket, wait for an ACK and then send FIN.

--
Karl Denninger
[hidden email]
The Market Ticker
[S/MIME encrypted email preferred]


--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: Close TCP socket after SSL_clear()?

Michael Wojcik
In reply to this post by Karl Denninger
> From: openssl-users [mailto:[hidden email]] On Behalf Of Karl Denninger
> Sent: Friday, January 11, 2019 13:04

>    if (!SSL_shutdown(connection)) {
>        SSL_shutdown(connection)
>    }

Or if you really want to baffle future maintainers:

        SSL_shutdown(connection) || SSL_shutdown(connection);

> The underlying handle is still open at the OS level after this, so on Unix anyway you want
> to notify the OS that the socket is invalid for further I/O and then close it.
> ...
>                    shutdown(slave_socket[x].fd, SHUT_RDWR);
>                    close(slave_socket[x].fd);

Maybe I'm missing something, but I don't see much advantage to calling shutdown(SHUT_RDRW) and then immediately calling close(). close will implicitly do what shutdown does, in the normal case, including trying to send unsent data and waiting (for a while) for any remaining ACKs.

If there's unsent or un-ACK'd data, shutdown will attempt to send it until the TCP retransmit limit is reached; that's normally longer than the linger time for the socket, so shutdown could try harder, and by the same token block longer, than close. But the same effect can be achieved by setting a longer linger time for the socket and just calling close.

Similarly, if linger has been disabled (by setting the SO_LINGER option appropriately), then close will just abort the connection (i.e. send an RST, rather than a FIN, and not wait for the corresponding FIN-ACK; or if the peer sent the FIN, send an RST rather than a FIN-ACK and not wait for the last ACK). But anyone who disables linger on a TLS connection gets what they deserve.

shutdown is generally useful if:

- You only want a half-close (which is rarely used, even when it would be useful, and isn't generally appropriate for a TLS connection).

- You want a full close, but you want to be able to retrieve the error information from the socket if the close fails. In that case, use shutdown, followed by getsockopt(SO_ERROR) if shutdown returns an error, followed by close. But your code is ignoring the return value from shutdown and not using getsockopt(SO_ERROR).

The real question is: will the application do anything differently if any remaining outbound data - which there shouldn't be because at this point we've tried to do a blocking SSL_shutdown - can't be sent, and the closing FIN / FIN-ACK / ACK handshake completed, within the default linger time? And if so, will the application do anything that can't be achieved by just increasing the linger time?

I think it'd be nice if more non-trivial applications used shutdown(SHUT_RDWR) + getsockopt(SO_ERROR) + close, and reported the error (if there is one) for diagnostic purposes. But beyond that there isn't a lot most applications can do, and for most a simple close is probably going to be fine.

But as I said I may have overlooked some good reason for this particular code pattern.

--
Michael Wojcik
Distinguished Engineer, Micro Focus



--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: Close TCP socket after SSL_clear()?

Charles Mills
>        SSL_shutdown(connection) || SSL_shutdown(connection);

I like it! (Not!)

I don't pretend to be a bits and bytes expert on TCP protocol. You can't be
an expert on everything.

So I will listen to expert advice. I know 99% of you all are 'nix guys and
this is a Windows problem. I am seeing OTOH where my Windows doc says
closesocket() does an abortive termination, and OTOH a discussion of a
graceful closesocket() with SO_LINGER/SO_DONTLINGER.

(1) This code is (at the application level) purely a receiver of data and
(2) without the TLS layer in place it is hard to picture any meaningful data
transfer and (3) we are in a session cleanup situation anyway -- so it seems
to me that an abortive disconnect is perfectly fine. Am I wrong?

Thanks for all of your help.

Charles


-----Original Message-----
From: openssl-users [mailto:[hidden email]] On Behalf Of
Michael Wojcik
Sent: Friday, January 11, 2019 12:48 PM
To: [hidden email]
Subject: Re: [openssl-users] Close TCP socket after SSL_clear()?

> From: openssl-users [mailto:[hidden email]] On Behalf
Of Karl Denninger
> Sent: Friday, January 11, 2019 13:04

>    if (!SSL_shutdown(connection)) {
>        SSL_shutdown(connection)
>    }

Or if you really want to baffle future maintainers:

        SSL_shutdown(connection) || SSL_shutdown(connection);

> The underlying handle is still open at the OS level after this, so on Unix
anyway you want
> to notify the OS that the socket is invalid for further I/O and then close
it.
> ...
>                    shutdown(slave_socket[x].fd, SHUT_RDWR);
>                    close(slave_socket[x].fd);

Maybe I'm missing something, but I don't see much advantage to calling
shutdown(SHUT_RDRW) and then immediately calling close(). close will
implicitly do what shutdown does, in the normal case, including trying to
send unsent data and waiting (for a while) for any remaining ACKs.

If there's unsent or un-ACK'd data, shutdown will attempt to send it until
the TCP retransmit limit is reached; that's normally longer than the linger
time for the socket, so shutdown could try harder, and by the same token
block longer, than close. But the same effect can be achieved by setting a
longer linger time for the socket and just calling close.

Similarly, if linger has been disabled (by setting the SO_LINGER option
appropriately), then close will just abort the connection (i.e. send an RST,
rather than a FIN, and not wait for the corresponding FIN-ACK; or if the
peer sent the FIN, send an RST rather than a FIN-ACK and not wait for the
last ACK). But anyone who disables linger on a TLS connection gets what they
deserve.

shutdown is generally useful if:

- You only want a half-close (which is rarely used, even when it would be
useful, and isn't generally appropriate for a TLS connection).

- You want a full close, but you want to be able to retrieve the error
information from the socket if the close fails. In that case, use shutdown,
followed by getsockopt(SO_ERROR) if shutdown returns an error, followed by
close. But your code is ignoring the return value from shutdown and not
using getsockopt(SO_ERROR).

The real question is: will the application do anything differently if any
remaining outbound data - which there shouldn't be because at this point
we've tried to do a blocking SSL_shutdown - can't be sent, and the closing
FIN / FIN-ACK / ACK handshake completed, within the default linger time? And
if so, will the application do anything that can't be achieved by just
increasing the linger time?

I think it'd be nice if more non-trivial applications used
shutdown(SHUT_RDWR) + getsockopt(SO_ERROR) + close, and reported the error
(if there is one) for diagnostic purposes. But beyond that there isn't a lot
most applications can do, and for most a simple close is probably going to
be fine.

But as I said I may have overlooked some good reason for this particular
code pattern.

--
Michael Wojcik
Distinguished Engineer, Micro Focus



--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: Close TCP socket after SSL_clear()?

Michael Wojcik
> From: openssl-users [mailto:[hidden email]] On Behalf Of
> Charles Mills
> Sent: Friday, January 11, 2019 17:06
>
> >        SSL_shutdown(connection) || SSL_shutdown(connection);
>
> I like it! (Not!)
>
> I don't pretend to be a bits and bytes expert on TCP protocol. You can't be
> an expert on everything.
>
> So I will listen to expert advice. I know 99% of you all are 'nix guys and
> this is a Windows problem. I am seeing OTOH where my Windows doc says
> closesocket() does an abortive termination, and OTOH a discussion of a
> graceful closesocket() with SO_LINGER/SO_DONTLINGER.
>
> (1) This code is (at the application level) purely a receiver of data and
> (2) without the TLS layer in place it is hard to picture any meaningful data
> transfer and (3) we are in a session cleanup situation anyway -- so it seems
> to me that an abortive disconnect is perfectly fine. Am I wrong?

Yes, you're wrong. You don't want an abortive disconnect.

A TCP connection can be closed in four (or five) ways:

1. Normal close, which involves the FIN / FIN-ACK / ACK sequence. When the last ACK is received, both sides know that all data has been received by the peer stack, and at the point when the corresponding ACK was generated, the peer "believed" it would be able to deliver the data to the application eventually. (That is, the stack hadn't been informed that the application's identifier for the connection - the socket - had been closed.)

2. Abortive close, which involves a RST from one side to the peer, and that's it. RST is a one-way, unacknowledged flow. There are a number of reasons why it's undesirable, some of which I'll go into below.

3. Abortive close due to network management message: the stack receives an ICMP message indicating a packet could not be delivered, such as HOST_UNREACH. From the application's point of view, the result is similar to #2, except for the particular error code it sees.

4. Timeout from TCP retransmit, either for an application send or, if it's enabled, TCP keepalive.

5. Arguably a separate case: 1-3 but generated by a middlebox, such as as a router, or an application firewall. In other words, the connection is forced closed by someone spoofing the peer. From the application's point of view, that makes no difference.

Applications should almost never use an abortive close. TCP is intended to be a reliable (best-effort) stream transport, and it can only meet its (already weak) service guarantees if you let it acknowledge all application data and close the conversation cleanly.

Now, when you have a higher-level conversation protocol such as TLS, and the higher-level protocol has already negotiated end-of-conversation, that may not seem important; the peers have agreed that they're not going to send anything more. That assumes, however, that the peers are well-behaved. And it is at the very least notionally cleaner to let the conversation close normally.

Beyond that, an abortive close can cause TIME_WAIT Assassination, which is a Bad Thing. If you don't know what TIME_WAIT Assassination is, that's a sign you shouldn't be doing abortive closes. Don't invoke extraordinary behavior you don't understand.

Now, all that said: Winsock closesocket will NOT do an abortive disconnect if you have not mucked with the SO_LINGER socket option (which you should not do unless you understand TCP). I don't know what documentation you saw that claims otherwise, but it's wrong.

Calling shutdown before closesocket won't hurt anything, but (if you use the pattern that we've discussed in this thread) won't do anything useful either, in most cases.

One case I forgot in my previous discussion: It's worth remembering that close/closesocket operates on a single reference to the connection, while shutdown operates on the connection itself. That is, the logic for close/closesocket is notionally something like this:

   close the descriptor/handle
   decrement the conversation's reference count
   if the reference count is 0
      if connection is still open for sending
         shutdown(SHUT_WR)
      if connection is still open for receiving
         shutdown(SHUT_RD)

In the case where you have multiple descriptors or handles for a conversation - for example due to dup'ing a socket or forking on UNIX, or duplicating a handle (possibly into a different process) on Windows - then close/closesocket won't do the shutdown-equivalent until they have *all* been closed. An explicit shutdown, on the other hand, doesn't close any of the descriptors/handles, but it does send a FIN (for SHUT_WR) or flush inbound data and refuse to receive any more (for SHUT_RD) on the conversation, which of course affects all descriptors/handles.

So if your application creates multiple references to the conversation, then depending on your design, you might want the shutdown. Or you might not, if you want the shutdown-on-last-close semantics. Neither option is correct for all applications.

--
Michael Wojcik
Distinguished Engineer, Micro Focus



--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: Close TCP socket after SSL_clear()?

Charles Mills
Thanks @Michael. I read up on TIME_WAIT Assassination.

I think that sort of thing may have been the problem I was trying to fix.
After an "error" disconnection, the customer was reporting that their client
could not re-connect. I had trouble getting good traces out of the customer,
but I suspect the problem was that the underlying TCP connection was still
hanging.

I have never in my life touched SO_LINGER. There is no socket duplication,
fork(), or the like.

Thanks again,

Charles


-----Original Message-----
From: openssl-users [mailto:[hidden email]] On Behalf Of
Michael Wojcik
Sent: Saturday, January 12, 2019 6:20 AM
To: [hidden email]
Subject: Re: [openssl-users] Close TCP socket after SSL_clear()?

> From: openssl-users [mailto:[hidden email]] On Behalf
Of
> Charles Mills
> Sent: Friday, January 11, 2019 17:06
>
> >        SSL_shutdown(connection) || SSL_shutdown(connection);
>
> I like it! (Not!)
>
> I don't pretend to be a bits and bytes expert on TCP protocol. You can't
be
> an expert on everything.
>
> So I will listen to expert advice. I know 99% of you all are 'nix guys and
> this is a Windows problem. I am seeing OTOH where my Windows doc says
> closesocket() does an abortive termination, and OTOH a discussion of a
> graceful closesocket() with SO_LINGER/SO_DONTLINGER.
>
> (1) This code is (at the application level) purely a receiver of data and
> (2) without the TLS layer in place it is hard to picture any meaningful
data
> transfer and (3) we are in a session cleanup situation anyway -- so it
seems
> to me that an abortive disconnect is perfectly fine. Am I wrong?

Yes, you're wrong. You don't want an abortive disconnect.

A TCP connection can be closed in four (or five) ways:

1. Normal close, which involves the FIN / FIN-ACK / ACK sequence. When the
last ACK is received, both sides know that all data has been received by the
peer stack, and at the point when the corresponding ACK was generated, the
peer "believed" it would be able to deliver the data to the application
eventually. (That is, the stack hadn't been informed that the application's
identifier for the connection - the socket - had been closed.)

2. Abortive close, which involves a RST from one side to the peer, and
that's it. RST is a one-way, unacknowledged flow. There are a number of
reasons why it's undesirable, some of which I'll go into below.

3. Abortive close due to network management message: the stack receives an
ICMP message indicating a packet could not be delivered, such as
HOST_UNREACH. From the application's point of view, the result is similar to
#2, except for the particular error code it sees.

4. Timeout from TCP retransmit, either for an application send or, if it's
enabled, TCP keepalive.

5. Arguably a separate case: 1-3 but generated by a middlebox, such as as a
router, or an application firewall. In other words, the connection is forced
closed by someone spoofing the peer. From the application's point of view,
that makes no difference.

Applications should almost never use an abortive close. TCP is intended to
be a reliable (best-effort) stream transport, and it can only meet its
(already weak) service guarantees if you let it acknowledge all application
data and close the conversation cleanly.

Now, when you have a higher-level conversation protocol such as TLS, and the
higher-level protocol has already negotiated end-of-conversation, that may
not seem important; the peers have agreed that they're not going to send
anything more. That assumes, however, that the peers are well-behaved. And
it is at the very least notionally cleaner to let the conversation close
normally.

Beyond that, an abortive close can cause TIME_WAIT Assassination, which is a
Bad Thing. If you don't know what TIME_WAIT Assassination is, that's a sign
you shouldn't be doing abortive closes. Don't invoke extraordinary behavior
you don't understand.

Now, all that said: Winsock closesocket will NOT do an abortive disconnect
if you have not mucked with the SO_LINGER socket option (which you should
not do unless you understand TCP). I don't know what documentation you saw
that claims otherwise, but it's wrong.

Calling shutdown before closesocket won't hurt anything, but (if you use the
pattern that we've discussed in this thread) won't do anything useful
either, in most cases.

One case I forgot in my previous discussion: It's worth remembering that
close/closesocket operates on a single reference to the connection, while
shutdown operates on the connection itself. That is, the logic for
close/closesocket is notionally something like this:

   close the descriptor/handle
   decrement the conversation's reference count
   if the reference count is 0
      if connection is still open for sending
         shutdown(SHUT_WR)
      if connection is still open for receiving
         shutdown(SHUT_RD)

In the case where you have multiple descriptors or handles for a
conversation - for example due to dup'ing a socket or forking on UNIX, or
duplicating a handle (possibly into a different process) on Windows - then
close/closesocket won't do the shutdown-equivalent until they have *all*
been closed. An explicit shutdown, on the other hand, doesn't close any of
the descriptors/handles, but it does send a FIN (for SHUT_WR) or flush
inbound data and refuse to receive any more (for SHUT_RD) on the
conversation, which of course affects all descriptors/handles.

So if your application creates multiple references to the conversation, then
depending on your design, you might want the shutdown. Or you might not, if
you want the shutdown-on-last-close semantics. Neither option is correct for
all applications.


--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users