asynchronous read/write with multithread

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

asynchronous read/write with multithread

YAMANEKO/Mao
Hello,

I'm re-writing an application with OpenSSL.
Originally, the application uses only plain TCP socket as a TCP client, it
has two threads, receives and submits message asynchronously. Namely, the
timing of sending/receiving messages are independent on each other, and
there is no logical sequence.

How does it possible to do asynchronous read/write with multithread?

The outline of the original code is below;
------ snip ------
sock = socket( ... ) ;
connect( sock ... ) ;
pthread_create( thread_receive ) ;
pthread_create( thread_send ) ;

void *thread_receive( .. ) {
    while ( ! end_of_transaction ) {
        input_from_aux_device() ;
        write( sock ... ) ;
    }
}

void *thread_send( .. ) {
    while ( ! end_of_transaction ) {
        read( sock ... ) ;
        output_to_aux_device() ;
    }
}
------ snip ------

I found an FAQ about multithreading;

> an SSL connection may not concurrently be used by multiple threads

So, at first, I made two simple wrapper functions to replace plain
read/write functions.

------ snip ------
int read_ssl( .. ) {
    pthread_mutex_lock( &rw_lock ) ;
    SSL_read( ... ) ;
    pthread_mutex_unlock( &rw_lock ) ;
}

int write_ssl( .. ) {
    pthread_mutex_lock( rw_lock ) ;
    SSL_write( ... ) ;
    pthread_mutex_unlock( rw_lock ) ;
}
------ snip ------

Of course, it did not work. The mutex was locked during waiting messages at
SSL_read, could not send any messages.

Can you please help me?
Thank you in advance!

--
YAMANEKO / Mao
http://yamamaya.com/
http://wiki.livedoor.jp/yamamaya_com/

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: asynchronous read/write with multithread

JoelKatz

> So, at first, I made two simple wrapper functions to replace plain
> read/write functions.
>
> ------ snip ------
> int read_ssl( .. ) {
>     pthread_mutex_lock( &rw_lock ) ;
>     SSL_read( ... ) ;
>     pthread_mutex_unlock( &rw_lock ) ;
> }
>
> int write_ssl( .. ) {
>     pthread_mutex_lock( rw_lock ) ;
>     SSL_write( ... ) ;
>     pthread_mutex_unlock( rw_lock ) ;
> }
> ------ snip ------
>
> Of course, it did not work. The mutex was locked during waiting
> messages at
> SSL_read, could not send any messages.

You really only have two choices:

1) Use non-blocking I/O. You can move the I/O to a service thread if you
like, so 'SSL_write' becomes 'push a message on the send queue and alert the
service thread' and 'SSL_read' becomes 'block on the receive queue'.

2) Restructure your design so that you do all the writing you need to do
before you call read again. (This may or may not be possible depending upon
the protocol you are implementing. It is possible for a pure query/response
protocol but not possible if the protocol is fundamentally asynchronous.)

DS


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: asynchronous read/write with multithread

YAMANEKO/Mao
Yes, the protocol is asynchronous exactly, not "query/response" sequence,
and could not re-design it now.

I could not find sufficient documents or examples about non-blocking I/O for
newbie like me. By way of experiment, I tried to re-write the code again
with BIO and non-blocking I/O.
The read() wrapping function I made newlly is below;

------- snip -------
BIO_set_nbio( cbio, 1 ) ;
BIO_do_connect( cbio ) ;
BIO_do_handshake( cbio ) ;

int read_ssl( ... ) {
    while ( 1 ) {
        pthread_mutex_lock( &rw_lock ) ;
        int ret = BIO_read( cbio ... ) ;
        pthread_mutex_unlock( &rw_lock ) ;
        if ( ret > 0 ) {
            break ;
        } else if ( ! BIO_should_retry( cbio ) ) {
            do_something_ERROR() ;
            break ;
        }
        usleep( a_little ) ; // to prevent wasting CPU
    }
}
------- snip -------

Will this work correctly with multithreaded asynchronous I/O?
But I think, this way is not better than simple blocking I/O like the
original code. It wastes CPU by the loop and gets poor response time by the
sleep. Is there any way better than it?

Can you please help me again?
Thank you in advance!

--
YAMANEKO / Mao
http://yamamaya.com/
http://wiki.livedoor.jp/yamamaya_com/

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: asynchronous read/write with multithread

JoelKatz

> Yes, the protocol is asynchronous exactly, not "query/response" sequence,
> and could not re-design it now.

Many protocols are that way and should be that way. I wouldn't redesign the
protocol unless it was badly designed in the first place.

> I could not find sufficient documents or examples about
> non-blocking I/O for
> newbie like me. By way of experiment, I tried to re-write the code again
> with BIO and non-blocking I/O.
> The read() wrapping function I made newlly is below;

> ------- snip -------
> BIO_set_nbio( cbio, 1 ) ;
> BIO_do_connect( cbio ) ;
> BIO_do_handshake( cbio ) ;
>
> int read_ssl( ... ) {
>     while ( 1 ) {
>         pthread_mutex_lock( &rw_lock ) ;
>         int ret = BIO_read( cbio ... ) ;
>         pthread_mutex_unlock( &rw_lock ) ;
>         if ( ret > 0 ) {
>             break ;
>         } else if ( ! BIO_should_retry( cbio ) ) {
>             do_something_ERROR() ;
>             break ;
>         }
>         usleep( a_little ) ; // to prevent wasting CPU
>     }
> }
> ------- snip -------
>
> Will this work correctly with multithreaded asynchronous I/O?
> But I think, this way is not better than simple blocking I/O like the
> original code. It wastes CPU by the loop and gets poor response
> time by the
> sleep. Is there any way better than it?

That's correct, but very poor for precisely the reasons you explain. You
should take a look at the include s_client example. The basic idea is this:

Reading:

1) Acquire the mutex.

2) Call SSL_pending. If any bytes are already available, call SSL_read,
release the lock, and return. (Note that calls to SSL_write might wind up
reading from the socket, so data might already be waiting.)

3) Call SSL_read. If it returns a positive number, release the lock and
return the data. If zero, release the lock and return EOF.

4) Pass the return value of SSL_read to SSL_get_error.

5) If the error was not WANT_READ or WANT_WRITE, handle it as appropriate.
You can assume it's fatal.

6) If the error was WANT_READ or WANT_WRITE, release the lock, select for
read or write as asked, re-acquire the lock, and go back to step 2.

Writing:

1) Acquire the mutex.

2) Call SSL_write. If we have sent all of the data, release the lock and
returen.

3) If we sent any data, re-adjust to only send the data that remains and go
to step 2.

4) If we got a zero, release the lock and return the number of bytes
successfully sent.

5) If we got an error, pass the return value to SSL_get_error.

6) If the error is fatal error, release the lock and return.

7) Obey the WANT_READ or WANT_WRITE by releasing the lock, calling 'select',
and re-acquiring the lock. Go back to step 2.

Note that SSL_read can return WANT_WRITE and SSL_write can return WANT_READ.

DS


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: asynchronous read/write with multithread

YAMANEKO/Mao
Thank you for your advice.
Now, it got almost clear!

> Writing:
>
> 1) Acquire the mutex.
>
> 2) Call SSL_write. If we have sent all of the data, release the lock and
> returen.
>
> 3) If we sent any data, re-adjust to only send the data that remains and
go

> to step 2.
>
> 4) If we got a zero, release the lock and return the number of bytes
> successfully sent.
>
> 5) If we got an error, pass the return value to SSL_get_error.
>
> 6) If the error is fatal error, release the lock and return.
>
> 7) Obey the WANT_READ or WANT_WRITE by releasing the lock, calling
'select',
> and re-acquiring the lock. Go back to step 2.

I have looked into s_client example, and thought over your advice, but I
have one more doubt yet.
I think, the meaning of "ready for write" at select() is only "there is some
space to write on TCP stack". Actually, the result of select() is almost
always "ready for write".
So, the meanings of WANT_READ and WANT_WRITE are really same as buffer full
on TCP stack? If not so, I think there is a possibility to cause non-wait
loop at step 2-7.

Sorry for I'm poor on the uptake...

--
YAMANEKO / Mao
http://yamamaya.com/
http://wiki.livedoor.jp/yamamaya_com/

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: asynchronous read/write with multithread

JoelKatz

> > Writing:
> >
> > 1) Acquire the mutex.
> >
> > 2) Call SSL_write. If we have sent all of the data, release the lock and
> > returen.
> >
> > 3) If we sent any data, re-adjust to only send the data that remains and
> > go
> > to step 2.
> >
> > 4) If we got a zero, release the lock and return the number of bytes
> > successfully sent.
> >
> > 5) If we got an error, pass the return value to SSL_get_error.
> >
> > 6) If the error is fatal error, release the lock and return.
> >
> > 7) Obey the WANT_READ or WANT_WRITE by releasing the lock, calling
> > 'select',
> > and re-acquiring the lock. Go back to step 2.

> I have looked into s_client example, and thought over your advice, but I
> have one more doubt yet.
> I think, the meaning of "ready for write" at select() is only
> "there is some
> space to write on TCP stack". Actually, the result of select() is almost
> always "ready for write".

Correct.

> So, the meanings of WANT_READ and WANT_WRITE are really same as
> buffer full
> on TCP stack?

WANT_READ means the SSL engine cannot make further progress until it can
read some data from the socket. WANT_WRITE means the SSL engine cannot make
further progress until it can write some data from the socket.

> If not so, I think there is a possibility to cause non-wait
> loop at step 2-7.

There is one ugly situation, and I don't know a good way to stop it.
Consider:

1) The receive buffer is empty, SSL_read returns WANT_READ. You release the
lock and prepare to call 'select'.

2) Another thread acquires the lock and calls SSL_write. SSL_write calls
'read' and reads the data. SSL_pending would now return greater than zero.

3) You switch back to the first thread which blocks on 'select'.
Unfortunately, the data it is waiting for has already arrived and been
consumed.

The only workarounds I know for this are horribly ugly. For example, you can
call SSL_pending after every call to SSL_write, and if it's non-zero, you
can write a byte to a pipe that the read thread also selects on.

Maybe someone else knows a better way. I always use BIO pairs because I
think OpenSSL's I/O code is not well thought out.

DS


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]