design strategy for clean termination of multithreaded server ?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

design strategy for clean termination of multithreaded server ?

Laurent Oget
One of the desirable features of a server is the ability to shutdown
gracefully without leaving the resources it uses in an inconsistent state.
I am looking for a manageable way to ensure this when the server is
multithreaded. to that effect i need to close all the SSL connections
opened by the servers before terminating.  The problem comes for the
connections for which there is a thread blocked on SSL_read, and the BIOs
for which there is a thread blocked on BIO_do_accept. I fear closing these
from another thread is likely to lead to a crash. On a POSIX platform it
looks like the cancellation mechanisms would be the right tool to use, but
how should I, or could I do it on the windows platform?

Thanks to all those who have some wisdom to share on that topic,

Laurent Oget



______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: design strategy for clean termination of multithreaded server ?

JoelKatz

> One of the desirable features of a server is the ability to shutdown
> gracefully without leaving the resources it uses in an inconsistent state.
> I am looking for a manageable way to ensure this when the server is
> multithreaded. to that effect i need to close all the SSL connections
> opened by the servers before terminating.  The problem comes for the
> connections for which there is a thread blocked on SSL_read, and the BIOs
> for which there is a thread blocked on BIO_do_accept. I fear closing these
> from another thread is likely to lead to a crash. On a POSIX platform it
> looks like the cancellation mechanisms would be the right tool to use, but
> how should I, or could I do it on the windows platform?

        Generally, if you value clean termination, you try not to block threads
indefinitely. To unblock threads in SSL_read, it should be safe to call
'shutdown' on the socket. I'm pretty sure you can also safely call
'shutdown' on a listening socket, but if not, you can always unblock the
thread by making a dummy connection to that socket yourself.

        DS


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: design strategy for clean termination of multithreaded server ?

Katie Lucas
On Mon, Oct 03, 2005 at 12:45:50PM -0700, David Schwartz wrote:

>
> > One of the desirable features of a server is the ability to shutdown
> > gracefully without leaving the resources it uses in an inconsistent state.
> > I am looking for a manageable way to ensure this when the server is
> > multithreaded. to that effect i need to close all the SSL connections
> > opened by the servers before terminating.  The problem comes for the
> > connections for which there is a thread blocked on SSL_read, and the BIOs
> > for which there is a thread blocked on BIO_do_accept. I fear closing these
> > from another thread is likely to lead to a crash. On a POSIX platform it
> > looks like the cancellation mechanisms would be the right tool to use, but
> > how should I, or could I do it on the windows platform?
>
> Generally, if you value clean termination, you try not to block threads
> indefinitely. To unblock threads in SSL_read, it should be safe to call
> 'shutdown' on the socket. I'm pretty sure you can also safely call
> 'shutdown' on a listening socket, but if not, you can always unblock the
> thread by making a dummy connection to that socket yourself.

We get decent terminations by doing a shutdown on the socket..

SSL_read() will return an error indicator and SSL_get_error() should
then say either SSL_ERROR_SYSCALL or SSL_ERROR_ZERO_RETURN depending
on how cleanly the protocol was shut down. You also check
ERR_get_error() and if that was a 0, the socket's been closed out from
under you. For -1, you can then check errno or WSAGetLastError() to
find out what happened to the socket.

The underlying socket will return a particular error system errorcode
( WSAESHUTDOWN ) which you can catch. On UNIX it'll probably be EINTR,
but I'm not going to swear to that.

Basically, we just look for either SSL return happening and then chuck
"my socket went away" exceptions so the stack gets unwound
properly. If there's more information in the stack or errno it hives
it away in our log system. The exception propagates to the thread
entry routine where a catch(...) handles it and the thread ends
normally.



An accepting socket returns something like WSAEINTR from the accept if
you shut the socket down, and it's fairly trivial to test after this
to see if it's still a valid socket and bail otherwise.

It requires a little bit of system dependent code to detect the
windows errors distinct from the UNIX errors.





Cancellation is a neat tool and makes things slightly easier, but
comes with some caveats. Setting the threads to asynch cancellation
makes it rather harder to do cleanups (unlocking mutexes properly and
so on).

Deferred cancellation is better, but there's a problem in that "read"
is a cancellation point on some UNIXes, but you probably want to do
clean up after that point... so you have to cope with pushing cleanup
code onto the stacks, or not use deferred cancellation either.

In the end, we're using a combination of a boolean "cancel" flag in
the threads and exceptions to break out of the loops, just to simplify
the cleanup.


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]