How to determine when data is finished on an SSL socket

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

How to determine when data is finished on an SSL socket

bored to death
Hi everybody,

I'm writing an application that creates multiple non-blocking SSL connections to an https server, in each one I send a request and read the server's response. my problem is, whatever I do, I can't determine when the response data is finished. here's part of my code responsible for sending and receiving data:
....
fd_set connectionfds;
struct timeval timeout2;
FD_ZERO(&connectionfds);
FD_SET(socket_server, &connectionfds);
timeout2.tv_usec = 0;
timeout2.tv_sec = 1;
while(1)
{
    r=BIO_read(io,buf,BUFSIZZ-1);
    if (r>0){
        //gather data
        continue;
    }
    else if (SSL_get_error(ssl, r)==SSL_ERROR_WANT_READ){
        int ret = select(socket_server + 1, &connectionfds, NULL, NULL, &timeout2);
        if (ret <= 0){
            break;
        }
        continue;
    }
    else{
        break;
    }
}
// use whole gathered data
....
my problem with above code is, if I set select timeout to a small time, I can't guarantee that all data is received (because some servers are really slow), and if I set timeout to a long time (5-10 seconds), my socket is getting stucked in wait-state for a long time and I can't use the response before that. I tried to make this work by using "BIO_should_read()" or "BIO_pending()" functions, but none of them is giving me what I want. so, Is there a way to determine when exactly there is nothing else to read on the SSL socket?

Thank you.

Reply | Threaded
Open this post in threaded view
|

Re: How to determine when data is finished on an SSL socket

Matthew Donald
Sockets (SSL or otherwise) are layer-3 objects while content is a layer-7 object.  To get an accurate end-of-content marker, you'll need to parse the output, buts since you are using the https protocol, that's going to be easy.

Essentially, parse out the "Content-Length" header and read that many bytes past the end of the http headers.




On 11 January 2014 19:46, M. V. <[hidden email]> wrote:
Hi everybody,

I'm writing an application that creates multiple non-blocking SSL connections to an https server, in each one I send a request and read the server's response. my problem is, whatever I do, I can't determine when the response data is finished. here's part of my code responsible for sending and receiving data:
....
fd_set connectionfds;
struct timeval timeout2;
FD_ZERO(&connectionfds);
FD_SET(socket_server, &connectionfds);
timeout2.tv_usec = 0;
timeout2.tv_sec = 1;
while(1)
{
    r=BIO_read(io,buf,BUFSIZZ-1);
    if (r>0){
        //gather data
        continue;
    }
    else if (SSL_get_error(ssl, r)==SSL_ERROR_WANT_READ){
        int ret = select(socket_server + 1, &connectionfds, NULL, NULL, &timeout2);
        if (ret <= 0){
            break;
        }
        continue;
    }
    else{
        break;
    }
}
// use whole gathered data
....
my problem with above code is, if I set select timeout to a small time, I can't guarantee that all data is received (because some servers are really slow), and if I set timeout to a long time (5-10 seconds), my socket is getting stucked in wait-state for a long time and I can't use the response before that. I tried to make this work by using "BIO_should_read()" or "BIO_pending()" functions, but none of them is giving me what I want. so, Is there a way to determine when exactly there is nothing else to read on the SSL socket?

Thank you.


Reply | Threaded
Open this post in threaded view
|

Re: How to determine when data is finished on an SSL socket

Chris Gray-4
In reply to this post by bored to death
Matthew Donald scripsit:

> Sockets (SSL or otherwise) are layer-3 objects while content is a
layer-7
> object.  To get an accurate end-of-content marker, you'll need to parse the
> output, buts since you are using the https protocol, that's going to be
easy.
> Essentially, parse out the "Content-Length" header and read that many bytes
> past the end of the http headers.

True, except that in various circumstances the "Content-Length" header may
be absent (e.g. special Transfer-Encoding, or a chunked response).  For
the full story see
http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4

Regards

Chris Gray

> On 11 January 2014 19:46, M. V. <[hidden email]> wrote:
>> Hi everybody,
>> I'm writing an application that creates multiple non-blocking SSL
connections to an https server, in each one I send a request and read
the
>> server's response. my problem is, whatever I do, I can't determine when
the

>> response data is finished. here's part of my code responsible for sending
>> and receiving data:
>> ....
>> fd_set connectionfds;struct timeval timeout2;
>> FD_ZERO(&connectionfds);
>> FD_SET(socket_server, &connectionfds);
>> timeout2.tv_usec = 0;
>> timeout2.tv_sec = 1;while(1){
>>     r=BIO_read(io,buf,BUFSIZZ-1);
>>     if (r>0){
>>         //gather data
>>         continue;
>>     }
>>     else if (SSL_get_error(ssl, r)==SSL_ERROR_WANT_READ){
>>         int ret = select(socket_server + 1, &connectionfds, NULL, NULL,
>> &timeout2);
>>         if (ret <= 0){
>>             break;
>>         }
>>         continue;
>>     }
>>     else{
>>         break;
>>     }}// use whole gathered data....
>> my problem with above code is, if I set select timeout to a small time, I
>> can't guarantee that all data is received (because some servers are really
>> slow), and if I set timeout to a long time (5-10 seconds), my socket is
getting stucked in wait-state for a long time and I can't use the
response
>> before that. I tried to make this work by using "BIO_should_read()" or
"BIO_pending()" functions, but none of them is giving me what I want.
so,
>> Is there a way to determine when exactly there is nothing else to read on
>> the SSL socket?
>> Thank you.




______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: How to determine when data is finished on an SSL socket

bored to death
In reply to this post by Matthew Donald
Hi Matthew,

but from what I know, not all HTTP responses have "Content-Length" field. responses with "chunked" payload are one of them. what can I do then?

Thank you.



On Saturday, January 11, 2014 1:23 PM, Matthew Donald <[hidden email]> wrote:
Sockets (SSL or otherwise) are layer-3 objects while content is a layer-7 object.  To get an accurate end-of-content marker, you'll need to parse the output, buts since you are using the https protocol, that's going to be easy.

Essentially, parse out the "Content-Length" header and read that many bytes past the end of the http headers.




On 11 January 2014 19:46, M. V. <[hidden email]> wrote:
Hi everybody,

I'm writing an application that creates multiple non-blocking SSL connections to an https server, in each one I send a request and read the server's response. my problem is, whatever I do, I can't determine when the response data is finished. here's part of my code responsible for sending and receiving data:
....
fd_set connectionfds;
struct timeval timeout2;
FD_ZERO(&connectionfds);
FD_SET(socket_server, &connectionfds);
timeout2.tv_usec = 0;
timeout2.tv_sec = 1;
while(1)
{
    r=BIO_read(io,buf,BUFSIZZ-1);
    if (r>0){
        //gather data
        continue;
    }
    else if (SSL_get_error(ssl, r)==SSL_ERROR_WANT_READ){
        int ret = select(socket_server + 1, &connectionfds, NULL, NULL, &timeout2);
        if (ret <= 0){
            break;
        }
        continue;
    }
    else{
        break;
    }
}
// use whole gathered data
....
my problem with above code is, if I set select timeout to a small time, I can't guarantee that all data is received (because some servers are really slow), and if I set timeout to a long time (5-10 seconds), my socket is getting stucked in wait-state for a long time and I can't use the response before that. I tried to make this work by using "BIO_should_read()" or "BIO_pending()" functions, but none of them is giving me what I want. so, Is there a way to determine when exactly there is nothing else to read on the SSL socket?

Thank you.




Reply | Threaded
Open this post in threaded view
|

Re: How to determine when data is finished on an SSL socket

Matthew Donald
When in doubt, use the source - or in this case RFC2616 §4.4.

TL;DR version - the length is indicated by one of three situations:

1. A Content-Length header exists
2. Transfer-Encoding is "chunked", in which case each chunk (there can be several) begins with a length in hex.  A zero length chunk terminates the grouping and the trailer is terminated with a CRLF.
3. Otherwise when the connection is closed.

Read the RFC for all the gory details.



On 11 January 2014 23:45, M. V. <[hidden email]> wrote:
Hi Matthew,

but from what I know, not all HTTP responses have "Content-Length" field. responses with "chunked" payload are one of them. what can I do then?

Thank you.




On Saturday, January 11, 2014 1:23 PM, Matthew Donald <[hidden email]> wrote:
Sockets (SSL or otherwise) are layer-3 objects while content is a layer-7 object.  To get an accurate end-of-content marker, you'll need to parse the output, buts since you are using the https protocol, that's going to be easy.

Essentially, parse out the "Content-Length" header and read that many bytes past the end of the http headers.




On 11 January 2014 19:46, M. V. <[hidden email]> wrote:
Hi everybody,

I'm writing an application that creates multiple non-blocking SSL connections to an https server, in each one I send a request and read the server's response. my problem is, whatever I do, I can't determine when the response data is finished. here's part of my code responsible for sending and receiving data:
....
fd_set connectionfds;
struct timeval timeout2;
FD_ZERO(&connectionfds);
FD_SET(socket_server, &connectionfds);
timeout2.tv_usec = 0;
timeout2.tv_sec = 1;
while(1)
{
    r=BIO_read(io,buf,BUFSIZZ-1);
    if (r>0){
        //gather data
        continue;
    }
    else if (SSL_get_error(ssl, r)==SSL_ERROR_WANT_READ){
        int ret = select(socket_server + 1, &connectionfds, NULL, NULL, &timeout2);
        if (ret <= 0){
            break;
        }
        continue;
    }
    else{
        break;
    }
}
// use whole gathered data
....
my problem with above code is, if I set select timeout to a small time, I can't guarantee that all data is received (because some servers are really slow), and if I set timeout to a long time (5-10 seconds), my socket is getting stucked in wait-state for a long time and I can't use the response before that. I tried to make this work by using "BIO_should_read()" or "BIO_pending()" functions, but none of them is giving me what I want. so, Is there a way to determine when exactly there is nothing else to read on the SSL socket?

Thank you.