OpenSSL + windows + asynch callbacks not being posted
found this one out the hard way. Had been having a problem with OpenSSL
"losing" callbacks, specifically FD_READ notifications.
Turned out to be fairly simple, but non-intuitive.
Many windows sockets apps rely on the OS to send an FD_READ notification
to tell the app when to read data, or the app calls select() with the
socket handle to see when it is ok to read or write data. With a normal
OS socket, Windows will post an FD_READ if you don't read all the
available data as well. Some apps rely on getting another FD_READ to
tell them there is data available.
Now enter OpenSSL and the BIO. There are a couple of gotchas with
intermediate buffering that BIO handlers do in OpenSSL. For starters,
when you call into SSL_read, it is possible that the underlying BIO
class will read more than you ask for, since it needs to get a whole TLS
record before it can decode it and give you back anything.
Take for example a POP3 client that makes an SSL connection to a
server. Say you are retrieving a message that is 7kB in size.
In your client code, you may call into SSL_read with a buffer. Say your
buffer length is 4kB. (this is what I was doing).
the call to SSL_read may return 4kb read, but if you were relying on
another FD_READ to be posted by the OS because you didn't read all the
data - BEWARE. Even though this is standard practise for normal winsock
programming, and the OS will post another FD_READ if there is more data
on the underlying transport, in this case there isn't!!
Since the BIO already read the whole TLS record, there is no more data
to be read off the socket, nor will any call to select or WSAAsyncSelect
make any difference - the data has been pulled off the socket, and is
sitting in the BIO buffer.
so, at the end of SSL_read, if you rely on FD_READs to be posted to tell
you there is more to read, you need to check yourself, by a call to
SSL_peek or something.