Possibility to cache ca-bundle and reuse it between SSL sessions?

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Possibility to cache ca-bundle and reuse it between SSL sessions?

Jens Maus
Hello,

this is actually my first post to this list, so please apologize if it might be too lengthy or too short or might address a question already raised in the past (which I didn’t find in the list archives so far).

I am an application developer of an email client using openssl to secure POP3 and SMTP connections. Since a while I have also added functionality to check the server certificates against a certificate bundle file (ca-bundle.crt) which users can store in the resource bundle of the mail client and the certificate check mechanism (via OpenSSL’ callback mechanisms) is working fine so far.

The only thing I am currently wondering is if there is a possibility to load the ca-bundle.crt file in advance and then reuse it between individual SSL connections. The reason why I am asking this is, that on the systems I am developing this email client for the SSL_CTX_load_verify_locations() function easily takes 2 - 3 seconds and AFAIK there is no functionality in OpenSSL to provide a preloaded certificate bundle to the SSL context structure.

So what my client currently does is (pseudo code):

— cut here —
[…]
conn->ssLCtx = SSL_CTX_new(SSLv23_client_method());
SSL_CTX_set_options(conn->sslCtx, SSL_OP_ALL | SSL_OP_NO_SSLv2);
SSL_CTX_load_verify_locations(conn->sslCtx, …);
SSL_CTX_set_default_verify_paths(…);
SSL_CTX_set_verify(conn->sslCtx, …);
SSL_CTX_set_cert_verify_callback(conn->sslCtx, …);
SSL_CTX_set_cipher_list(conn->sslCtx, …);
conn->ssl = SSL_new(conn->sslCtx);
SSL_set_fd(conn->ssl, (int)conn->socket);
SSL_connect(conn->ssl);
[…]
— cut here —

Looking at that execution sequence the SSL_CTX_load_verify_locations() call easily takes 2 - 3 seconds here either if the ca-bundle file is quite large or if the system is busy doing other stuff. This is especially critical since there are unfortunately some mail servers on the net (so-called ‚Nemesis‘ mail server from gmx.de, web.de and 1und1.de) which have a rather short SSL negotiation timeout (8 - 10 seconds only) right from the initiating STARTTLS call until the SSL negotiation have to finished. Otherwise they simply drop the connection - which IMHO is another problem and another story not to be discussed here.

So is there some possibility that I can load the ca-bundle.crt file in advance and simply supply the data to SSL_CTX instead of having to use SSL_CTX_load_verify_locations() which actually loads the ca-bundle.crt file from disk every time a new connection (and thus SSL_CTX_load_verify_locations()) is initiated?

Best regards,
jens
--
Jens Maus, Dresden/Germany
http://jens-maus.de/

(Please note a real name change effective since 5.9.2013.
Former name: Jens Langner)

*** Content is authentic only with digital signature  ***


smime.p7s (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Possibility to cache ca-bundle and reuse it between SSL sessions?

Jakob Bohm-7
On 6/24/2014 7:58 PM, Jens Maus wrote:

> Hello,
>
> this is actually my first post to this list, so please apologize if it might be too lengthy or too short or might address a question already raised in the past (which I didn’t find in the list archives so far).
>
> I am an application developer of an email client using openssl to secure POP3 and SMTP connections. Since a while I have also added functionality to check the server certificates against a certificate bundle file (ca-bundle.crt) which users can store in the resource bundle of the mail client and the certificate check mechanism (via OpenSSL’ callback mechanisms) is working fine so far.
>
> The only thing I am currently wondering is if there is a possibility to load the ca-bundle.crt file in advance and then reuse it between individual SSL connections. The reason why I am asking this is, that on the systems I am developing this email client for the SSL_CTX_load_verify_locations() function easily takes 2 - 3 seconds and AFAIK there is no functionality in OpenSSL to provide a preloaded certificate bundle to the SSL context structure.
>
> So what my client currently does is (pseudo code):
>
> — cut here —
> […]
> conn->ssLCtx = SSL_CTX_new(SSLv23_client_method());
> SSL_CTX_set_options(conn->sslCtx, SSL_OP_ALL | SSL_OP_NO_SSLv2);
> SSL_CTX_load_verify_locations(conn->sslCtx, …);
> SSL_CTX_set_default_verify_paths(…);
> SSL_CTX_set_verify(conn->sslCtx, …);
> SSL_CTX_set_cert_verify_callback(conn->sslCtx, …);
> SSL_CTX_set_cipher_list(conn->sslCtx, …);
> conn->ssl = SSL_new(conn->sslCtx);
> SSL_set_fd(conn->ssl, (int)conn->socket);
> SSL_connect(conn->ssl);
> […]
> — cut here —
>
> Looking at that execution sequence the SSL_CTX_load_verify_locations() call easily takes 2 - 3 seconds here either if the ca-bundle file is quite large or if the system is busy doing other stuff. This is especially critical since there are unfortunately some mail servers on the net (so-called ‚Nemesis‘ mail server from gmx.de, web.de and 1und1.de) which have a rather short SSL negotiation timeout (8 - 10 seconds only) right from the initiating STARTTLS call until the SSL negotiation have to finished. Otherwise they simply drop the connection - which IMHO is another problem and another story not to be discussed here.
>
> So is there some possibility that I can load the ca-bundle.crt file in advance and simply supply the data to SSL_CTX instead of having to use SSL_CTX_load_verify_locations() which actually loads the ca-bundle.crt file from disk every time a new connection (and thus SSL_CTX_load_verify_locations()) is initiated?
>

Use SSL_CTX_get_cert_store() directly, this returns the X509_STORE
object, which you can then configure to lookup the CA certificates
from an in-memory structure of your own.

Unfortunately, the X509_STORE object is mostly undocumented, however
it seems you can simply call X509_STORE_add_cert() and
X509_STORE_add_crl() with X509 and X509_CRL objects for each of
the certificates and crls in your in-memory cache.

It seems undocumented if there is sufficient reference counting of
X509/X509_CRL objects to share them (read-only) amongst threads, or if
you will have to duplicate them before adding them to the X509_STORE.

If duplication is needed, the easiest would be to hold the ca-bundle
in memory as a single large (read only) byte array, then for each new
SSL session, loop over d2i_X509() until you reach the end of your array
or it fails.  Use a second array for the concatenated CRLs.  Note that
the arrays should be in DER format, not PEM format.


Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  http://www.wisemo.com
Transformervej 29, 2730 Herlev, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Possibility to cache ca-bundle and reuse it between SSL sessions?

Viktor Dukhovni
In reply to this post by Jens Maus
On Tue, Jun 24, 2014 at 07:58:27PM +0200, Jens Maus wrote:

> conn->ssLCtx = SSL_CTX_new(SSLv23_client_method());
> SSL_CTX_set_options(conn->sslCtx, SSL_OP_ALL | SSL_OP_NO_SSLv2);
> SSL_CTX_load_verify_locations(conn->sslCtx, ?);
> SSL_CTX_set_default_verify_paths(?);
> SSL_CTX_set_verify(conn->sslCtx, ?);
> SSL_CTX_set_cert_verify_callback(conn->sslCtx, ?);
> SSL_CTX_set_cipher_list(conn->sslCtx, ?);

The above should be done ONCE on application startup.  A single
SSL_CTX can and should typically be used for multiple connections.

> conn->ssl = SSL_new(conn->sslCtx);
> SSL_set_fd(conn->ssl, (int)conn->socket);
> SSL_connect(conn->ssl);

Then for each connection just perform the above.

> So is there some possibility that I can load the ca-bundle.crt file in advance and simply supply the data to SSL_CTX instead of having to use SSL_CTX_load_verify_locations() which actually loads the ca-bundle.crt file from disk every time a new connection (and thus SSL_CTX_load_verify_locations()) is initiated?

Use a single SSL_CTX to make multiple connections.

--
        Viktor.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Possibility to cache ca-bundle and reuse it between SSL sessions?

Jens Maus
Hello Viktor,

On 2014-06-25 at 00:49, Viktor Dukhovni <[hidden email]> wrote:

> On Tue, Jun 24, 2014 at 07:58:27PM +0200, Jens Maus wrote:
>
>> conn->ssLCtx = SSL_CTX_new(SSLv23_client_method());
>> SSL_CTX_set_options(conn->sslCtx, SSL_OP_ALL | SSL_OP_NO_SSLv2);
>> SSL_CTX_load_verify_locations(conn->sslCtx, ?);
>> SSL_CTX_set_default_verify_paths(?);
>> SSL_CTX_set_verify(conn->sslCtx, ?);
>> SSL_CTX_set_cert_verify_callback(conn->sslCtx, ?);
>> SSL_CTX_set_cipher_list(conn->sslCtx, ?);
>
> The above should be done ONCE on application startup.  A single
> SSL_CTX can and should typically be used for multiple connections.
Is this also true for multithreaded (parallel) connections? That means, can I simply create the SSL_CTX object upon starting my application and keep it throughout the whole lifetime of the application and just assign it to a new SSL connection via SSL_new()? Or is there some timeout involved or limitation with initiating parallel connections using the sam SSL_CTX?


>> conn->ssl = SSL_new(conn->sslCtx);
>> SSL_set_fd(conn->ssl, (int)conn->socket);
>> SSL_connect(conn->ssl);
>
> Then for each connection just perform the above.

This sounds reasonable and would be perfect for my purpose. However, I am still wondering if there are any things to consider when using a single SSL_CTX for multiple parallel connections to different servers.

>
>> So is there some possibility that I can load the ca-bundle.crt file in advance and simply supply the data to SSL_CTX instead of having to use SSL_CTX_load_verify_locations() which actually loads the ca-bundle.crt file from disk every time a new connection (and thus SSL_CTX_load_verify_locations()) is initiated?
>
> Use a single SSL_CTX to make multiple connections.

As asked above. Is this also true for applications doing multiple connections in parallel to different servers?

best regards,
jens
--
Jens Maus, Dresden/Germany
http://jens-maus.de/

(Please note a real name change effective since 5.9.2013.
Former name: Jens Langner)

*** Content is authentic only with digital signature  ***



smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Possibility to cache ca-bundle and reuse it between SSL sessions?

michel-60
In reply to this post by Viktor Dukhovni
Excerpt from the book : "Network Security with OpenSSL" :

"An SSL_CTX object will be a factory for producing SSL connection objects.
This context allows us to set connection configuration parameters before
the connection is made, such as protocol version, certificate
information, and verification requirements.
It is easiest to think of SSL_CTX objects as the containers for default
values for the SSL connections to be made by a program.
...
In general, an application will create just one SSL_CTX object for all
of the connections it makes."

And Yes, this is also true for multithreaded connections, as long as we
are aware of :
https://www.openssl.org/docs/crypto/threads.html



______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Possibility to cache ca-bundle and reuse it between SSL sessions?

Jens Maus
On 2014-06-25 at 15:06, Michel <[hidden email]> wrote:

> Excerpt from the book : "Network Security with OpenSSL" :
>
> "An SSL_CTX object will be a factory for producing SSL connection objects.
> This context allows us to set connection configuration parameters before the connection is made, such as protocol version, certificate information, and verification requirements.
> It is easiest to think of SSL_CTX objects as the containers for default values for the SSL connections to be made by a program.
> …

Thanks for the reminder. But I read the OpenSSL manual pages already, of course - but as the documentation of OpenSSL is (to be honest) really bad, I wanted to make this absolutely clear.

> In general, an application will create just one SSL_CTX object for all of the connections it makes."
>
> And Yes, this is also true for multithreaded connections, as long as we are aware of :
> https://www.openssl.org/docs/crypto/threads.html

Ok, but then please allow the question how I should deal with

SSL_CTX_set_cert_verify_callback(sslCtx, func, conn);

in that context? Because currently we use this function to define an own verify callback function and we supply ‘conn’ here as an application specific pointer argument (and extracting it via X509_STORE_CTX_get_app_data(x509_ctx) within the callback function) for filling in the individual results of the certificate verify process of a specific SSL connection. The problem that arises here is that this ‘conn’ pointer is connection specific in our case. That means I want to be able to use a connection specific ‘conn’ argument with SSL_CTX_set_cert_verify_callback(), but if I call this function once at the very beginning of my application I can only specify it once and calling SSL_CTX_set_cert_verify_callback() on the same sslCtx pointer for every parallel connection will of course overwrite the old setting.

So how can I specify an own app_data for every connection? IMHO there should be something like SSL_set_cert_app_data() so that I can specify different app_data for different SSL connections.

regards,
jens
--
Jens Maus, Dresden/Germany
http://jens-maus.de/

(Please note a real name change effective since 5.9.2013.
Former name: Jens Langner)

*** Content is authentic only with digital signature  ***


smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Possibility to cache ca-bundle and reuse it between SSL sessions?

Viktor Dukhovni
On Wed, Jun 25, 2014 at 03:23:27PM +0200, Jens Maus wrote:

> Ok, but then please allow the question how I should deal with
>
> SSL_CTX_set_cert_verify_callback(sslCtx, func, conn);

Set this callback once, with a "conn" value of 0.  Use

    ssl_idx = SSL_get_ex_data_X509_STORE_CTX_idx();
    conn = X509_STORE_CTX_get_ex_data(store_ctx, ssl_idx);

to retrieve the connection handle.

> So how can I specify an own app_data for every connection? IMHO
> there should be something like SSL_set_cert_app_data() so that I
> can specify different app_data for different SSL connections.

Rarely needed, could be added in the future, but that won't help
you now.

--
        Viktor.

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Possibility to cache ca-bundle and reuse it between SSL sessions?

Jakob Bohm-7
In reply to this post by Jens Maus
On 6/25/2014 3:23 PM, Jens Maus wrote:

> On 2014-06-25 at 15:06, Michel <[hidden email]> wrote:
>
>> Excerpt from the book : "Network Security with OpenSSL" :
>>
>> "An SSL_CTX object will be a factory for producing SSL connection objects.
>> This context allows us to set connection configuration parameters before the connection is made, such as protocol version, certificate information, and verification requirements.
>> It is easiest to think of SSL_CTX objects as the containers for default values for the SSL connections to be made by a program.
>> …
>
> Thanks for the reminder. But I read the OpenSSL manual pages already, of course - but as the documentation of OpenSSL is (to be honest) really bad, I wanted to make this absolutely clear.
>
>> In general, an application will create just one SSL_CTX object for all of the connections it makes."
>>
>> And Yes, this is also true for multithreaded connections, as long as we are aware of :
>> https://www.openssl.org/docs/crypto/threads.html
>
> Ok, but then please allow the question how I should deal with
>
> SSL_CTX_set_cert_verify_callback(sslCtx, func, conn);
>
> in that context? Because currently we use this function to define an own verify callback function and we supply ‘conn’ here as an application specific pointer argument (and extracting it via X509_STORE_CTX_get_app_data(x509_ctx) within the callback function) for filling in the individual results of the certificate verify process of a specific SSL connection. The problem that arises here is that this ‘conn’ pointer is connection specific in our case. That means I want to be able to use a connection specific ‘conn’ argument with SSL_CTX_set_cert_verify_callback(), but if I call this function once at the very beginning of my application I can only specify it once and calling SSL_CTX_set_cert_verify_callback() on the same sslCtx pointer for every parallel connection will of course overwrite the old setting.
>
> So how can I specify an own app_data for every connection? IMHO there should be something like SSL_set_cert_app_data() so that I can specify different app_data for different SSL connections.
>

After calling ssl_ctx = SSL_new(master_ssl_ctx) to get the new context,
call

   X509_STORE store_obj = SSL_CTX_get_cert_store(ssl_ctx),

then set your pointer in the CRYPTO_EX_DATA of
store_obj->ex_data.

When your callback receives an X509_STORE_CTX store_ctx, you can access
that same X509_STORE as store_ctx->ctx and get your pointer from the
CRYPTO_EX_DATA at store_ctx->ctx->ex_data.

At least that is what it looks like to me.

(Figuring out how to use the generic CRYPTO_EX_DATA API is left as an
exercise for the reader).



Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  http://www.wisemo.com
Transformervej 29, 2730 Herlev, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Possibility to cache ca-bundle and reuse it between SSL sessions?

Jens Maus
In reply to this post by Viktor Dukhovni
On 2014-06-25 at 16:28, Viktor Dukhovni <[hidden email]> wrote:

> On Wed, Jun 25, 2014 at 03:23:27PM +0200, Jens Maus wrote:
>
>> Ok, but then please allow the question how I should deal with
>>
>> SSL_CTX_set_cert_verify_callback(sslCtx, func, conn);
>
> Set this callback once, with a "conn" value of 0.  Use
>
>    ssl_idx = SSL_get_ex_data_X509_STORE_CTX_idx();
>    conn = X509_STORE_CTX_get_ex_data(store_ctx, ssl_idx);
>
> to retrieve the connection handle.
Ok, thank you for that hint. But I still have problems understanding it completely or I haven’t made myself clear enough.

How exactly can I set the ‘conn’ pointer so that within my verify callback I can use your suggested solution?

My current pseudo-code looks like:

— cut here —
verify_callback(..)
{
  [..]
  conn = X509_STORE_CTX_get_app_data(x509_ctx);

  /* storing verify failures as flags in sslCertFailures */
  setFlag(conn->sslCertFailures, ..)
  [..]
}

InitSSL()
{
  ssLCtx = SSL_CTX_new(SSLv23_client_method()); // global variable
  [..]
  SSL_CTX_set_verify(sslCtx, ..);
  SSL_CTX_set_cert_verify_callback(sslCtx, verify_callback, conn);
}

/* then within an own thread when creating a new connection */
NewSSLConnection()
{
  conn = CreateNewConnection(); // local variable
  conn->ssl = SSL_new(sslCtx);
  SSL_set_fd(conn->ssl, (int)conn->socket);
  SSL_connect(conn->ssl);
  [..]
  /* evaluate conn->sslCertFailues set by verify_callback() */
}
— cut here —

Hope that makes it clear now. The problem is that ‘conn’ is created as soon as a new connection is created and thus does not exist at the time I have to call SSL_CTX_set_cert_verify_callback() when I use only one SSL_CTX throughout my whole application. So I need to have some possibility that after “CreateNewConnection” I can tell either SSL_CTX or SSL to use ‘conn’ so that as soon as the verify_callback is executed I can get the correct app_data via X509_STORE_CTX_get_app_data().

Hope that makes it more clear.

Thanks in advance.

best regards,
jens
--
Jens Maus, Dresden/Germany
http://jens-maus.de/

(Please note a real name change effective since 5.9.2013.
Former name: Jens Langner)

*** Content is authentic only with digital signature  ***



smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Possibility to cache ca-bundle and reuse it between SSL sessions?

Jens Maus
In reply to this post by Jakob Bohm-7
On 2014-06-25 at 16:58, Jakob Bohm <[hidden email]> wrote:

> On 6/25/2014 3:23 PM, Jens Maus wrote:

>> So how can I specify an own app_data for every connection? IMHO there should be something like SSL_set_cert_app_data() so that I can specify different app_data for different SSL connections.
>>
>
> After calling ssl_ctx = SSL_new(master_ssl_ctx) to get the new context,
> call
>
>  X509_STORE store_obj = SSL_CTX_get_cert_store(ssl_ctx),

I am sorry, but isn’t SSL_new() returning a pointer of SSL* and not SSL_CTX*? Thus, I cannot simply pass that to SSL_CTX_get_cert_store() as that expects a pointer of type SSL_CTX*. Right?

> then set your pointer in the CRYPTO_EX_DATA of
> store_obj->ex_data.
>
> When your callback receives an X509_STORE_CTX store_ctx, you can access
> that same X509_STORE as store_ctx->ctx and get your pointer from the
> CRYPTO_EX_DATA at store_ctx->ctx->ex_data.
>
> At least that is what it looks like to me.
>
> (Figuring out how to use the generic CRYPTO_EX_DATA API is left as an
> exercise for the reader).
Well, that’s actually very hard for me since I am mainly an application developer and didn’t dive that deep into the various hundreds of possibilities of the complete OpenSSL API yet. So I would really enjoy seeing some pseudo code being posted here which solves the top-level problem I posted here: I want to be able to load my ca-bundle only once at application start and distribute it to all sub connections not having to call SSL_CTX_load_verify_locations() right before each SSL_new().

Actually, I now understand that I can keep a single SSL_CTX throughout the whole lifetime of my application and just assign it to the SSL_new() call. However, as I outlined in my last posts, if I change my code like this I run into the verify_callback problem and that I have to be able to specify different app_data pointers per SSL connection or otherwise I cannot store the certificate check results for evaluation by my application.

regards,
jens
--
Jens Maus, Dresden/Germany
http://jens-maus.de/

(Please note a real name change effective since 5.9.2013.
Former name: Jens Langner)

*** Content is authentic only with digital signature  ***



smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Possibility to cache ca-bundle and reuse it between SSL sessions?

Jeffrey Walton-3
On Wed, Jun 25, 2014 at 11:15 AM, Jens Maus <[hidden email]> wrote:
> ...
>
> Actually, I now understand that I can keep a single SSL_CTX throughout the whole lifetime of my application and just assign it to the SSL_new() call. However, as I outlined in my last posts, if I change my code like this I run into the verify_callback problem and that I have to be able to specify different app_data pointers per SSL connection or otherwise I cannot store the certificate check results for evaluation by my application.
>
The SSL* should be unique per connection. Is it not possible to keep a
data structure somewhere that uses the SSL* as a key? Then stuff the
data as the value of the pair.

I know there's a difference between connections and sessions (one
connection can have multiple sessions; sessions can be resumed, etc).
So I'm not clear if it will meet all your needs.

Jeff
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Possibility to cache ca-bundle and reuse it between SSL sessions?

Jens Maus
On 2014-06-25 at 17:33, Jeffrey Walton <[hidden email]> wrote:

> On Wed, Jun 25, 2014 at 11:15 AM, Jens Maus <[hidden email]> wrote:
>> ...
>>
>> Actually, I now understand that I can keep a single SSL_CTX throughout the whole lifetime of my application and just assign it to the SSL_new() call. However, as I outlined in my last posts, if I change my code like this I run into the verify_callback problem and that I have to be able to specify different app_data pointers per SSL connection or otherwise I cannot store the certificate check results for evaluation by my application.
>>
> The SSL* should be unique per connection. Is it not possible to keep a
> data structure somewhere that uses the SSL* as a key? Then stuff the
> data as the value of the pair.

Of course, everything is possible. But if I create a global data structure now that uses the SSL* as a key so that I can search for that one within the verify_callback this would certainly involve that I need to do an own thread locking (e.g. via a mutex). And that’s IMHO very inconvenient if OpenSSL should be in principle able to provide me that functionality so that I can access the correct application specific data even on the SSL* level.

> I know there's a difference between connections and sessions (one
> connection can have multiple sessions; sessions can be resumed, etc).
> So I'm not clear if it will meet all your needs.

Here, I am not dealing with sessions at all. Here I simply generate unique connections and each connection get its own SSL*. And now I need to be able to supply my application specific data to the verify callback I specify myself. If I use a unique SSL_CTX* per connection I can perfectly do that, but how to do that if I want to have one global SSL_CTX* throughout my application and just create unique SSL* per connection? How can I then access connection specific data from within the verify callback? That’s the one million dollar question. ;)

regards,
jens
--
Jens Maus, Dresden/Germany
http://jens-maus.de/

(Please note a real name change effective since 5.9.2013.
Former name: Jens Langner)

*** Content is authentic only with digital signature  ***


smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: Possibility to cache ca-bundle and reuse it between SSL sessions?

Michael Wojcik
In reply to this post by Jens Maus
> From: [hidden email] [mailto:owner-openssl-
> [hidden email]] On Behalf Of Jens Maus
> Sent: Wednesday, 25 June, 2014 11:05
>
> On 2014-06-25 at 16:28, Viktor Dukhovni <[hidden email]> wrote:
>
> > On Wed, Jun 25, 2014 at 03:23:27PM +0200, Jens Maus wrote:
> >
> >> Ok, but then please allow the question how I should deal with
> >>
> >> SSL_CTX_set_cert_verify_callback(sslCtx, func, conn);
> >
> > Set this callback once, with a "conn" value of 0.  Use
> >
> >    ssl_idx = SSL_get_ex_data_X509_STORE_CTX_idx();
> >    conn = X509_STORE_CTX_get_ex_data(store_ctx, ssl_idx);
> >
> > to retrieve the connection handle.
>
> Ok, thank you for that hint. But I still have problems understanding it
> completely or I haven't made myself clear enough.
>
> How exactly can I set the 'conn' pointer so that within my verify callback I
> can use your suggested solution?

You don't. As Viktor wrote, you set the "conn" parameter to 0, and you don't use it for anything.

Viktor gave you the solution, but it probably wasn't clear if you're not familiar with this particular corner of the OpenSSL API.

OpenSSL *already passes the SSL* connection "handle" to the verify callback*. You don't have to do anything to tell OpenSSL to do that. It's just not passed as one of the parameters; instead it's tucked down inside the X.509 container (X509_STORE_CTX).

To retrieve it in the callback, you use the code Viktor provided. First you get the index of the SSL* object in the X509_STORE_CTX's "external data" collection:

        int ssl_idx = SSL_get_ex_X509_STORE_CTX_idx();

Then you use that index to get the SSL* itself:

        SSL *conn = X509_STORE_CTX_get_ex_data(store_ctx, ssl_idx);

(Where "store_ctx" is the second parameter passed to the verify callback).

Of course you can do this in a single line:

        SSL *conn = X509_STORE_CTX_get_ex_data(store_ctx, SSL_get_ex_X509_STORE_CTX_idx());

That's it.


Now, if you need additional application-specific information in the callback, the best thing is to add it as external data in the SSL object:

        /* Create an index for our data in the SSL object */
        int get_index(void) {
                /* Serialize as necessary */
                static int index = -1;
                if (index < 0) index = SSL_get_ex_new_index(...);
                return index;
        }

        /* After creating the SSL object */
        SSL_set_ex_data(conn, get_index(), my_data_ptr);

        ...

        /* In the verify callback, or wherever */
        my_data_ptr = SSL_get_ex_data(conn, get_index());

But if all you need in the callback is the SSL object, you needn't worry about all that.

--
Michael Wojcik
Technology Specialist, Micro Focus



This message has been scanned for malware by Websense. www.websense.com
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Possibility to cache ca-bundle and reuse it between SSL sessions?

Dr. Stephen Henson
In reply to this post by Viktor Dukhovni
On Wed, Jun 25, 2014, Viktor Dukhovni wrote:

> On Wed, Jun 25, 2014 at 03:23:27PM +0200, Jens Maus wrote:
>
> > Ok, but then please allow the question how I should deal with
> >
> > SSL_CTX_set_cert_verify_callback(sslCtx, func, conn);
>
> Set this callback once, with a "conn" value of 0.  Use
>
>     ssl_idx = SSL_get_ex_data_X509_STORE_CTX_idx();
>     conn = X509_STORE_CTX_get_ex_data(store_ctx, ssl_idx);
>
> to retrieve the connection handle.
>
> > So how can I specify an own app_data for every connection? IMHO
> > there should be something like SSL_set_cert_app_data() so that I
> > can specify different app_data for different SSL connections.
>
> Rarely needed, could be added in the future, but that won't help
> you now.
>

Though SSL structures support ex_data and that could be used. See:

https://www.openssl.org/docs/ssl/SSL_get_ex_new_index.html`

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Possibility to cache ca-bundle and reuse it between SSL sessions?

Jens Maus
In reply to this post by Jens Maus
Am 25.06.2014 um 18:09 schrieb Jeffrey Walton <[hidden email]>:

> On Wed, Jun 25, 2014 at 11:45 AM, Jens Maus <[hidden email]> wrote:
>> Of course, everything is possible. But if I create a global data structure now that uses the SSL* as a key so that I can search for that one within the verify_callback this would certainly involve that I need to do an own thread locking (e.g. via a mutex). And that’s IMHO very inconvenient if OpenSSL should be in principle able to provide me that functionality so that I can access the correct application specific data even on the SSL* level.
>
> There's nothing anyone can do about it (except the OpenSSL devs). Take
> it or leave it.

Sorry, but I don’t understand your offensive attitude. I am here simply to discuss possibilities and so far no OpenSSL developer had specifically stated that the functionality I wanted is not possible with the current API. I think it is just a matter of discussion finding the right track for the solution I require due to the fact that the OpenSSL API comes with so many possibilities one single person can hardly keep in mind.

regards,
jens
--
Jens Maus, Dresden/Germany
http://jens-maus.de/

(Please note a real name change effective since 5.9.2013.
Former name: Jens Langner)

*** Content is authentic only with digital signature  ***


smime.p7s (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Possibility to cache ca-bundle and reuse it between SSL sessions?

Jens Maus
In reply to this post by Michael Wojcik
Am 25.06.2014 um 18:22 schrieb Michael Wojcik <[hidden email]>:

> […]
> To retrieve it in the callback, you use the code Viktor provided. First you get the index of the SSL* object in the X509_STORE_CTX's "external data" collection:
>
> int ssl_idx = SSL_get_ex_X509_STORE_CTX_idx();
>
> Then you use that index to get the SSL* itself:
>
> SSL *conn = X509_STORE_CTX_get_ex_data(store_ctx, ssl_idx);
>
> (Where "store_ctx" is the second parameter passed to the verify callback).
>
> Of course you can do this in a single line:
>
> SSL *conn = X509_STORE_CTX_get_ex_data(store_ctx, SSL_get_ex_X509_STORE_CTX_idx());
>
> That's it.
Thank you Michael for this very nice description. When Steven sent the link to the documentation around I already had a look at the API and found out about X509_STORE_CTX_get_ex_data() myself. However, there is still one open point which I feel needs to be addressed before I can actually implement it that way:

> Now, if you need additional application-specific information in the callback, the best thing is to add it as external data in the SSL object:
>
> /* Create an index for our data in the SSL object */
> int get_index(void) {
> /* Serialize as necessary */
> static int index = -1;
> if (index < 0) index = SSL_get_ex_new_index(...);
> return index;
> }
>
> /* After creating the SSL object */
> SSL_set_ex_data(conn, get_index(), my_data_ptr);
>
> ...
>
> /* In the verify callback, or wherever */
> my_data_ptr = SSL_get_ex_data(conn, get_index());
>
> But if all you need in the callback is the SSL object, you needn't worry about all that.
Exactly, I need more than just the plain SSL object within the verify callback because I want to store the individual verify callback results in a connection specific variable as pointed out in my earlier post.

Now that you have provided the above pseudo-code and due to the point that I read the documentation about SSL_set_ex_data() I perfectly understand what Viktor meant. The only thing that still worries me is that particular get_index() function and the point that you use a static int variable inside there. The reason for that is, that in multithreaded environments that actually would involve locking access to „index“ via a separate mutex which IMHO is sub-optimal, to say the least. In addition, when doing parallel connections with unique SSL* pointers per connection the above would fail since get_index() will only return the index number of the first get_index() call in the first SSL connection. But if two or more parallel SSL connections are initiated you would AFAICS require a unique index variable per running SSL*. So somehow this looks to me like a hen vs. egg problem since I need to store the index number to access the external data outside the verify callback function. Or is there anything else I can do about that using OpenSSL functionality?

best regards,
jens
--
Jens Maus, Dresden/Germany
http://jens-maus.de/

(Please note a real name change effective since 5.9.2013.
Former name: Jens Langner)

*** Content is authentic only with digital signature  ***



smime.p7s (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: Possibility to cache ca-bundle and reuse it between SSL sessions?

Michael Wojcik
> From: [hidden email] [mailto:owner-openssl-
> [hidden email]] On Behalf Of Jens Maus
> Sent: Wednesday, 25 June, 2014 14:07
>
> Am 25.06.2014 um 18:22 schrieb Michael Wojcik
> <[hidden email]>:
>
> > [...]
> > Now, if you need additional application-specific information in the
> callback, the best thing is to add it as external data in the SSL object:
> >
> > /* Create an index for our data in the SSL object */
> > int get_index(void) {
> > /* Serialize as necessary */
> > static int index = -1;
> > if (index < 0) index = SSL_get_ex_new_index(...);
> > return index;
> > }
> >
> > /* After creating the SSL object */
> > SSL_set_ex_data(conn, get_index(), my_data_ptr);
> >
> > ...
> >
> > /* In the verify callback, or wherever */
> > my_data_ptr = SSL_get_ex_data(conn, get_index());
> >
> > But if all you need in the callback is the SSL object, you needn't worry
> about all that.
>
> Exactly, I need more than just the plain SSL object within the verify
> callback because I want to store the individual verify callback results in a
> connection specific variable as pointed out in my earlier post.
>
> Now that you have provided the above pseudo-code and due to the point that I
> read the documentation about SSL_set_ex_data() I perfectly understand what
> Viktor meant. The only thing that still worries me is that particular
> get_index() function and the point that you use a static int variable inside
> there. The reason for that is, that in multithreaded environments that
> actually would involve locking access to "index" via a separate mutex which
> IMHO is sub-optimal, to say the least.

No serialization is necessary around the use of "index". Serialization is necessary in get_index(), which is invoked relatively rarely: when a conversation is created and when the verify callback is called. Neither of those are in the inner loop, and worrying about serialization there is definitely premature optimization.

get_index() serializes only to ensure that SSL_get_ex_new_index is called only once during the process lifetime. You can omit serialization if, for example, you can ensure that you invoke SSL_get_ex_new_index only from one thread (typically the initial thread, before any others are created).

> In addition, when doing parallel
> connections with unique SSL* pointers per connection the above would fail
> since get_index() will only return the index number of the first get_index()
> call in the first SSL connection.

Yes, that's what it should do. It should always return the same value.

> But if two or more parallel SSL connections
> are initiated you would AFAICS require a unique index variable per running
> SSL*.

No, that's not how it works. You need one index value per item to be stored in a given SSL object. You have one item to store - a pointer to your application data - in each SSL object. You'll use the same index value for that item in each SSL object.


--
Michael Wojcik
Technology Specialist, Micro Focus




This message has been scanned for malware by Websense. www.websense.com
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Possibility to cache ca-bundle and reuse it between SSL sessions?

Jens Maus
On 2014-06-25 at 22:22, Michael Wojcik <[hidden email]> wrote:

[…]
>> But if two or more parallel SSL connections
>> are initiated you would AFAICS require a unique index variable per running
>> SSL*.
>
> No, that's not how it works. You need one index value per item to be stored in a given SSL object. You have one item to store - a pointer to your application data - in each SSL object. You'll use the same index value for that item in each SSL object.

Thank you very much Michael. That was the final comment that made me fully understand where you want to pointed me at. I really first thought I need a unique index for each SSL* object, but as you said, I only need to call SSL_get_ex_new_index() once upon application startup and just save that single index in a global variable and then use it with the SSL_set_ex_data() call I execute when initiating a new SSL negotiation.

To finalize this mail thread and make sure if someone stumbles over it via google, allow me to add an URL to the sources of the mail client I have now implemented the SSL negotiation as you have suggested:

http://yam.ch/browser/trunk/src/tcp/ssl.c?rev=8113#L1053

And for the ones not wanting to read foreign source code, I will summarize our findings in some pseudo code:

— cut here —
int globalAppDataIndex;
SSL_CTX *globalSSLctx;

int verify_callback(…)
{
  SSL *ssl = X509_STORE_CTX_get_ex_data(x509_ctx, SSL_get_ex_data_X509_STORE_CTX_idx());
  void *app_data = SSL_get_ex_data(ssl, globalAppDataIndex);

  /* do something on app_data */
  […]
}

void makeSSLconnection()
{
  void *app_data = malloc();
 
  /* fill app_data with connection specific data */
  […]

  SSL *ssl = SSL_new(globalSSLctx);
  SSL_set_ex_data(ssl, globalAppDataIndex, app_data);
  SSL_set_fd(ssl, socket);
  SSL_connect(ssl);
 
  /* perform SSL secured connection */
  […]
}

void InitSSL()
{
  […]
  globalSSLctx = SSL_CTX_new(…);
  globalAppDataIndex = SSL_get_ex_new_index(0, NULL, NULL, NULL, NULL);
  SSL_CTX_load_verify_locations(globalSSLctx, …);
  SSL_CTX_set_default_verify_paths(globalSSLctx);
  SSL_CTX_set_verify(globalSSLctx, SSL_VERIFY_PEER, verify_callback);
  […]
}

void main(…)
{
  /* run InitSSL() only once */
  InitSSL();

  /* create multiple, parallel (multithreaded) connections */
  for(int i=0; i < numConnections; i++)
  {
    /* create socket and perform tcp connection */
    […]

    /* init the SSL negotiation */
    makeSSLConnection();
  }

  […]
}
— cut here —

This pseudo code should allow to load a ca-bundle or all types of certificates via SSL_CTX_load_verify_locations() once at application startup via keeping a single SSL_CTX* object throughout the whole lifetime of the application. At the same time it comes with a verify_callback and shows how to forward own application specific data using SSL_set_ex_data() to the verify callback function.

So thanks to anyone involved in this thread. The final solution is really now more optimized and allows to perform SSL connections way faster.

best regards,
jens
--
Jens Maus, Dresden/Germany
http://jens-maus.de/

(Please note a real name change effective since 5.9.2013.
Former name: Jens Langner)

*** Content is authentic only with digital signature  ***



smime.p7s (5K) Download Attachment