mutual-TLS / mTLS Example with certificate problem

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

mutual-TLS / mTLS Example with certificate problem

Andreas Tengicki
Hello,

I can not find a working mutual-TLS server/client example on github or
the whole internet. Only some example for pieces of code. Communication
via socket without and with encryption (openSSL) is working, but with
mTLS not. I believe that I theoretical understand mTLS, but the practice
will not work.

The whole (small) project is here:
https://github.com/deckard-rick/mTLS-example

Server Side
=========

I initialize the SSL-context without errors with (sample, error handling
is not in this email)

    SSL_CTX_set_ecdh_auto(srvCtx->ctx, 1);
    SSL_CTX_set_verify(srvCtx->ctx, SSL_VERIFY_PEER or
SSL_VERIFY_FAIL_IF_NO_PEER_CERT, NULL);
    SSL_CTX_load_verify_locations(srvCtx->ctx,NULL,"../certs"); //????
    SSL_CTX_use_certificate_chain_file(srvCtx->ctx,
"../certs/server/ca.crt");
    SSL_CTX_use_certificate_file(srvCtx->ctx,
"../certs/server/server.crt", SSL_FILETYPE_PEM);
    SSL_CTX_use_PrivateKey_file(srvCtx->ctx,
"../certs/server/server.key", SSL_FILETYPE_PEM);
    SSL_CTX_check_private_key(srvCtx->ctx);

the certificates are:

ca.crt:  Version: 3 (0x2)
    Serial Number:
5a:fc:74:e6:28:28:0e:df:5b:7a:50:9e:a8:18:e6:04:42:f0:fd:8d
    Signature Algorithm: sha256WithRSAEncryption
    Issuer: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = 42CA
     Validity  Not Before: May  6 09:21:23 2020 GMT  Not After : May  6
09:21:23 2022 GMT
     Subject: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN
= 42CA

server.crt: Version: 1 (0x0)
    Serial Number:
5f:6f:44:b5:27:47:f2:d2:fe:2b:21:5b:38:7d:e5:f6:e5:d9:c1:23
    Signature Algorithm: sha256WithRSAEncryption
    Issuer: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = 42CA
    Validity  Not Before: May  6 09:30:23 2020 GMT   Not After : May  6
09:30:23 2021 GMT
    Subject: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN =
debiandevdesktop01.sdctec.lokal

debiandevdesktop01.sdctec.lokal is the FQDN of the development server

Client Side
=========

    SSL_CTX_set_ecdh_auto(ctx, 1);
    SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER, NULL);
    SSL_CTX_use_certificate_chain_file(ctx, "../certs/client/ca.crt");
    SSL_CTX_use_certificate_file(ctx, "../certs/client/client.crt",
SSL_FILETYPE_PEM);
    SSL_CTX_use_PrivateKey_file(ctx, "../certs/client/client.key",
SSL_FILETYPE_PEM);

ca.crt:  (see server)

client.crt: Version: 1 (0x0)
   Serial Number: 
5f:6f:44:b5:27:47:f2:d2:fe:2b:21:5b:38:7d:e5:f6:e5:d9:c1:24
   Signature Algorithm: sha256WithRSAEncryption
   Issuer: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = 42CA
   Validity  Not Before: May  6 09:35:51 2020 GMT   Not After : May  6
09:35:51 2021 GMT
   Subject: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN =
CLIENT001

Error:
=====

If the client connects the server there are the following errors:

server:
139918902234240:error:1416F086:SSL
routines:tls_process_server_certificate:certificate verify
failed:../ssl/statem/statem_clnt.c:1915:

client:
139918902234240:error:1416F086:SSL
routines:tls_process_server_certificate:certificate verify
failed:../ssl/statem/statem_clnt.c:1915:

I think, there is a problem with the certificates. But where is the
problem and why?

The statement to create the certificates are in the project ./certs/read.me

Thanks for any help, I'm looking since days for a solution and I believe
it is only a small bug.

Best regards

  Andreas


Reply | Threaded
Open this post in threaded view
|

Re: mutual-TLS / mTLS Example with certificate problem

Viktor Dukhovni
On Wed, May 06, 2020 at 08:44:57PM +0200, Andreas Tengicki wrote:

> I can not find a working mutual-TLS server/client example on github or
> the whole internet. Only some example for pieces of code. Communication
> via socket without and with encryption (openSSL) is working, but with
> mTLS not. I believe that I theoretical understand mTLS, but the practice
> will not work.

Postfix uses an "ask_ccert" configuration boolean to solicit client
certificates.  The associated server-side code (with the SNI ctx
side-effects elided) is:

    if (props->ask_ccert)
        verify_flags = SSL_VERIFY_PEER | SSL_VERIFY_CLIENT_ONCE;
    SSL_CTX_set_verify(server_ctx, verify_flags,
                       tls_verify_certificate_callback);
    if (props->ask_ccert && *props->CAfile) {
        STACK_OF(X509_NAME) *calist = SSL_load_client_CA_file(props->CAfile);

        if (calist == 0) {
            /* Not generally critical */
            msg_warn("error loading client CA names from: %s",
                     props->CAfile);
            tls_print_errors();
        }
        SSL_CTX_set_client_CA_list(server_ctx, calist);
    }

Some clients will not send a certificate unless the server-side client
CA list is non-empty and includes the root CA that issued the client's
cert.


>     SSL_CTX_set_ecdh_auto(ctx, 1);
>     SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER, NULL);
>     SSL_CTX_use_certificate_chain_file(ctx, "../certs/client/ca.crt");
>     SSL_CTX_use_certificate_file(ctx, "../certs/client/client.crt", SSL_FILETYPE_PEM);
>     SSL_CTX_use_PrivateKey_file(ctx, "../certs/client/client.key", SSL_FILETYPE_PEM);

You SHOULD NOT specify both a certificate chain file and certificate
file.  The ..._chain_file() function loads the leaf cert, and then the
rest of the chain.

>
> server:
> 139918902234240:error:1416F086:SSL
> routines:tls_process_server_certificate:certificate verify
> failed:../ssl/statem/statem_clnt.c:1915:

Your trust stores don't contain the requisite CAs and/or the chain files
are missing required intermediate certs.

--
    Viktor.
Reply | Threaded
Open this post in threaded view
|

Re: mutual-TLS / mTLS Example with certificate problem

Viktor Dukhovni
In reply to this post by Andreas Tengicki
On Wed, May 06, 2020 at 08:44:57PM +0200, Andreas Tengicki wrote:

>     SSL_CTX_load_verify_locations(srvCtx->ctx,NULL,"../certs"); //????

Have you run "c_rehash" on "../certs" (not keen on relative file names
here myself).


> Client Side
> =========
>
>     SSL_CTX_set_ecdh_auto(ctx, 1);
>     SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER, NULL);
>     SSL_CTX_use_certificate_chain_file(ctx, "../certs/client/ca.crt");
>     SSL_CTX_use_certificate_file(ctx, "../certs/client/client.crt", SSL_FILETYPE_PEM);
>     SSL_CTX_use_PrivateKey_file(ctx, "../certs/client/client.key", SSL_FILETYPE_PEM);

What is the client doing for "verify_locations"?

--
    Viktor.
Reply | Threaded
Open this post in threaded view
|

RE: mutual-TLS / mTLS Example with certificate problem

Michael Wojcik
In reply to this post by Andreas Tengicki
> From: openssl-users [mailto:[hidden email]] On Behalf Of
> Andreas Tengicki
> Sent: Wednesday, May 06, 2020 12:45
> To: [hidden email]
> Subject: mutual-TLS / mTLS Example with certificate problem
>
> I can not find a working mutual-TLS server/client example on github or
> the whole internet.

By "mutual-TLS" I assume you mean "TLS with mutual authentication".

I don't know about open-source examples off the top of my head, but all the products I work on support mutual authentication.

Oh, wait, of course I know of an open-source example. It's OpenSSL, which supports mutual authentication in the s_server and s_client apps.

>     SSL_CTX_use_certificate_chain_file(srvCtx->ctx,
> "../certs/server/ca.crt");
>     SSL_CTX_use_certificate_file(srvCtx->ctx,
> "../certs/server/server.crt", SSL_FILETYPE_PEM);

This is very likely wrong. SSL(_CTX)_use_certificate_chain_file sets the entity certificate and its (partial) chain. So when you call SSL_CTX_use_certificate_file you're overwriting the entity certificate set by use_certificate_chain_file.

Get rid of the call to use_certificate_file and put everything the server should be sending into the chain file, in the order described in the OpenSSL documentation: entity certificate, certificate for its issuer, and so on up to and including the root. (I've just noticed the docs don't say whether use_certificate_chain_file specifies SSL_BUILD_CHAIN_FLAG_NO_ROOT when it calls add1_chain_cert, so offhand I don't know whether this will cause the root to be included in the chain the server sends. But that shouldn't really matter.)

> ca.crt:  Version: 3 (0x2)
>     Serial Number:
> 5a:fc:74:e6:28:28:0e:df:5b:7a:50:9e:a8:18:e6:04:42:f0:fd:8d
>     Signature Algorithm: sha256WithRSAEncryption
>     Issuer: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = 42CA
>      Validity  Not Before: May  6 09:21:23 2020 GMT  Not After : May  6
> 09:21:23 2022 GMT
>      Subject: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN
> = 42CA

Not enough information. We don't know what the Basic Constraints are for this certificate, or EKU, or whether it's actually the certificate that signed your server's entity certificate.

More importantly, if this is the only certificate in ca.crt, which was what you passed to use_certificate_chain_file, then this was stored in the context as the entity cert, and no certificates were added to the chain. Then you overwrote the entity cert with use_certificate_file, and you still have no chain.

(At least I believe that's what will happen; I haven't actually tried it.)

> server.crt: Version: 1 (0x0)

X.509v1? PKIX moved to v3 in, what, 2002 (with RFC 3280)?

I mean, X.509v1 ought to still work, but it's hardly good practice.

>     Subject: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN =
> debiandevdesktop01.sdctec.lokal
>
> debiandevdesktop01.sdctec.lokal is the FQDN of the development server

And is that exactly what the client is specifying when it tries to verify the server's certificate?

>     SSL_CTX_use_certificate_chain_file(ctx, "../certs/client/ca.crt");
>     SSL_CTX_use_certificate_file(ctx, "../certs/client/client.crt",
> SSL_FILETYPE_PEM);

Same problem as above.

> If the client connects the server there are the following errors:
>
> server:
> 139918902234240:error:1416F086:SSL
> routines:tls_process_server_certificate:certificate verify
> failed:../ssl/statem/statem_clnt.c:1915:

Is that the whole OpenSSL error stack? When reporting an OpenSSL error (from your application), you should always make sure to dump the whole stack.

Also, a piece of advice: A good place to start when diagnosing issues like this is to swap out the server for openssl s_server, or the client for openssl s_client. s_client can give you a whole bunch of information about what the server is sending, and would have shown the chain is just the entity certificate in this case.

--
Michael Wojcik
Distinguished Engineer, Micro Focus


Reply | Threaded
Open this post in threaded view
|

Re: mutual-TLS / mTLS Example with certificate problem

Matt Caswell-2
In reply to this post by Andreas Tengicki
On 06/05/2020 19:44, Andreas Tengicki wrote:
>     SSL_CTX_set_ecdh_auto(srvCtx->ctx, 1);

Viktor and Michael have already provided some excellent advice on this
so I won't cover the same ground. Just one note on this line though -
this is actually unnecessary in modern versions of OpenSSL (anything
from OpenSSL 1.1.0 and above). It's a no-op in modern versions - "auto"
ecdh is the default.

Matt

Reply | Threaded
Open this post in threaded view
|

Re: mutual-TLS / mTLS Example with certificate problem

Andreas Tengicki
In reply to this post by Michael Wojcik
Hello Michael,

thank you and Viktor for your fast help. Viktor annotations I don't
fully understand. Sure there is the openssl test server and client, but
the source code is complex for everyone who is new in this encryption tasks.

But testing with openssl was a valuable notice:

1) openssl vs openssl

openssl s_server -port 8080 -CAfile certs/server/ca.crt -cert
certs/server/server.crt -key certs/server/server.key -Verify 99

Is this the right call for "TLS with mutual authentication", I cannot
find an other parameter for SSL_VERIFY_PEER?

with

openssl s_client -connect localhost:8080 -CAfile certs/client/ca.crt
-cert certs/client/client.crt -key certs/client/client.key

it is working, and the server sees the COMMON NAME of the client (the
whole certificate)

2) openssl vs myclient

is working

3) myserver vs openssl (and my client) is not working

In the openssl server command line I set only CAfile, cert, key and verify

I believe I do this also in my source code:

    int verify_flags = SSL_VERIFY_PEER
    SSL_CTX_set_verify(srvCtx->ctx, verify_flags, NULL);

    char cafile[] =
"/home/debdev/Projects/clearing/server/certs/client/ca.crt";

    Variant 1:
    SSL_CTX_use_certificate_chain_file(srvCtx->ctx, cafile)

    Variant 2:
    STACK_OF(X509_NAME) *calist = SSL_load_client_CA_file(cafile);
    SSL_CTX_set_client_CA_list(srvCtx->ctx, calist);

    if (SSL_CTX_use_certificate_file(srvCtx->ctx, srvCtx->cert,
SSL_FILETYPE_PEM) <= 0)
    if (SSL_CTX_use_PrivateKey_file(srvCtx->ctx, srvCtx->key,
SSL_FILETYPE_PEM) <= 0 )

But there is still the error: tls_process_client_certificate:certificate
verify failed

Why?

4) I have looked into apps/s_server.c there is a
  ctx_set_verify_locations with the cafile information

in apps.c this calls:  SSL_CTX_load_verify_file, but the compiler can't
find this function.

Why? Thanks and sorry if I do not understand all details in the first try.

Best regards

  Andreas

Am 06.05.2020 um 21:59 schrieb Michael Wojcik:

>> From: openssl-users [mailto:[hidden email]] On Behalf Of
>> Andreas Tengicki
>> Sent: Wednesday, May 06, 2020 12:45
>> To: [hidden email]
>> Subject: mutual-TLS / mTLS Example with certificate problem
>>
>> I can not find a working mutual-TLS server/client example on github or
>> the whole internet.
> By "mutual-TLS" I assume you mean "TLS with mutual authentication".
>
> I don't know about open-source examples off the top of my head, but all the products I work on support mutual authentication.
>
> Oh, wait, of course I know of an open-source example. It's OpenSSL, which supports mutual authentication in the s_server and s_client apps.
>
>>     SSL_CTX_use_certificate_chain_file(srvCtx->ctx,
>> "../certs/server/ca.crt");
>>     SSL_CTX_use_certificate_file(srvCtx->ctx,
>> "../certs/server/server.crt", SSL_FILETYPE_PEM);
> This is very likely wrong. SSL(_CTX)_use_certificate_chain_file sets the entity certificate and its (partial) chain. So when you call SSL_CTX_use_certificate_file you're overwriting the entity certificate set by use_certificate_chain_file.
>
> Get rid of the call to use_certificate_file and put everything the server should be sending into the chain file, in the order described in the OpenSSL documentation: entity certificate, certificate for its issuer, and so on up to and including the root. (I've just noticed the docs don't say whether use_certificate_chain_file specifies SSL_BUILD_CHAIN_FLAG_NO_ROOT when it calls add1_chain_cert, so offhand I don't know whether this will cause the root to be included in the chain the server sends. But that shouldn't really matter.)
>
>> ca.crt:  Version: 3 (0x2)
>>     Serial Number:
>> 5a:fc:74:e6:28:28:0e:df:5b:7a:50:9e:a8:18:e6:04:42:f0:fd:8d
>>     Signature Algorithm: sha256WithRSAEncryption
>>     Issuer: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = 42CA
>>      Validity  Not Before: May  6 09:21:23 2020 GMT  Not After : May  6
>> 09:21:23 2022 GMT
>>      Subject: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN
>> = 42CA
> Not enough information. We don't know what the Basic Constraints are for this certificate, or EKU, or whether it's actually the certificate that signed your server's entity certificate.
>
> More importantly, if this is the only certificate in ca.crt, which was what you passed to use_certificate_chain_file, then this was stored in the context as the entity cert, and no certificates were added to the chain. Then you overwrote the entity cert with use_certificate_file, and you still have no chain.
>
> (At least I believe that's what will happen; I haven't actually tried it.)
>
>> server.crt: Version: 1 (0x0)
> X.509v1? PKIX moved to v3 in, what, 2002 (with RFC 3280)?
>
> I mean, X.509v1 ought to still work, but it's hardly good practice.
>
>>     Subject: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN =
>> debiandevdesktop01.sdctec.lokal
>>
>> debiandevdesktop01.sdctec.lokal is the FQDN of the development server
> And is that exactly what the client is specifying when it tries to verify the server's certificate?
>
>>     SSL_CTX_use_certificate_chain_file(ctx, "../certs/client/ca.crt");
>>     SSL_CTX_use_certificate_file(ctx, "../certs/client/client.crt",
>> SSL_FILETYPE_PEM);
> Same problem as above.
>
>> If the client connects the server there are the following errors:
>>
>> server:
>> 139918902234240:error:1416F086:SSL
>> routines:tls_process_server_certificate:certificate verify
>> failed:../ssl/statem/statem_clnt.c:1915:
> Is that the whole OpenSSL error stack? When reporting an OpenSSL error (from your application), you should always make sure to dump the whole stack.
>
> Also, a piece of advice: A good place to start when diagnosing issues like this is to swap out the server for openssl s_server, or the client for openssl s_client. s_client can give you a whole bunch of information about what the server is sending, and would have shown the chain is just the entity certificate in this case.
>
> --
> Michael Wojcik
> Distinguished Engineer, Micro Focus
>
>
Reply | Threaded
Open this post in threaded view
|

Re: mutual-TLS / mTLS Example with certificate problem

Raja Ashok-2
In reply to this post by Andreas Tengicki
Hi Andreas,

Below repo has examples to use OpenSSL for mTLS (mutual certificate authentication) with sample certificates. You can refer this.


On Thu, May 7, 2020 at 12:36 AM Andreas Tengicki <[hidden email]> wrote:
Hello,

I can not find a working mutual-TLS server/client example on github or
the whole internet. Only some example for pieces of code. Communication
via socket without and with encryption (openSSL) is working, but with
mTLS not. I believe that I theoretical understand mTLS, but the practice
will not work.

The whole (small) project is here:
https://github.com/deckard-rick/mTLS-example

Server Side
=========

I initialize the SSL-context without errors with (sample, error handling
is not in this email)

    SSL_CTX_set_ecdh_auto(srvCtx->ctx, 1);
    SSL_CTX_set_verify(srvCtx->ctx, SSL_VERIFY_PEER or
SSL_VERIFY_FAIL_IF_NO_PEER_CERT, NULL);
    SSL_CTX_load_verify_locations(srvCtx->ctx,NULL,"../certs"); //????
    SSL_CTX_use_certificate_chain_file(srvCtx->ctx,
"../certs/server/ca.crt");
    SSL_CTX_use_certificate_file(srvCtx->ctx,
"../certs/server/server.crt", SSL_FILETYPE_PEM);
    SSL_CTX_use_PrivateKey_file(srvCtx->ctx,
"../certs/server/server.key", SSL_FILETYPE_PEM);
    SSL_CTX_check_private_key(srvCtx->ctx);

the certificates are:

ca.crt:  Version: 3 (0x2)
    Serial Number:
5a:fc:74:e6:28:28:0e:df:5b:7a:50:9e:a8:18:e6:04:42:f0:fd:8d
    Signature Algorithm: sha256WithRSAEncryption
    Issuer: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = 42CA
     Validity  Not Before: May  6 09:21:23 2020 GMT  Not After : May  6
09:21:23 2022 GMT
     Subject: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN
= 42CA

server.crt: Version: 1 (0x0)
    Serial Number:
5f:6f:44:b5:27:47:f2:d2:fe:2b:21:5b:38:7d:e5:f6:e5:d9:c1:23
    Signature Algorithm: sha256WithRSAEncryption
    Issuer: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = 42CA
    Validity  Not Before: May  6 09:30:23 2020 GMT   Not After : May  6
09:30:23 2021 GMT
    Subject: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN =
debiandevdesktop01.sdctec.lokal

debiandevdesktop01.sdctec.lokal is the FQDN of the development server

Client Side
=========

    SSL_CTX_set_ecdh_auto(ctx, 1);
    SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER, NULL);
    SSL_CTX_use_certificate_chain_file(ctx, "../certs/client/ca.crt");
    SSL_CTX_use_certificate_file(ctx, "../certs/client/client.crt",
SSL_FILETYPE_PEM);
    SSL_CTX_use_PrivateKey_file(ctx, "../certs/client/client.key",
SSL_FILETYPE_PEM);

ca.crt:  (see server)

client.crt: Version: 1 (0x0)
   Serial Number: 
5f:6f:44:b5:27:47:f2:d2:fe:2b:21:5b:38:7d:e5:f6:e5:d9:c1:24
   Signature Algorithm: sha256WithRSAEncryption
   Issuer: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = 42CA
   Validity  Not Before: May  6 09:35:51 2020 GMT   Not After : May  6
09:35:51 2021 GMT
   Subject: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN =
CLIENT001

Error:
=====

If the client connects the server there are the following errors:

server:
139918902234240:error:1416F086:SSL
routines:tls_process_server_certificate:certificate verify
failed:../ssl/statem/statem_clnt.c:1915:

client:
139918902234240:error:1416F086:SSL
routines:tls_process_server_certificate:certificate verify
failed:../ssl/statem/statem_clnt.c:1915:

I think, there is a problem with the certificates. But where is the
problem and why?

The statement to create the certificates are in the project ./certs/read.me

Thanks for any help, I'm looking since days for a solution and I believe
it is only a small bug.

Best regards

  Andreas


Reply | Threaded
Open this post in threaded view
|

Re: mutual-TLS / mTLS Example with certificate problem

Kyle Hamilton
In reply to this post by Michael Wojcik
On a tangent, this file format (and order) was actually finally
standardized as "application/pem-certificate-chain" by RFC 8555
section 9.1 (the Automatic Certificate Management Environment
protocol, or ACME).

On Wed, May 6, 2020 at 2:59 PM Michael Wojcik
<[hidden email]> wrote:
> Get rid of the call to use_certificate_file and put everything the server should be sending into the chain file, in the order described in the OpenSSL documentation: entity certificate, certificate for its issuer, and so on up to and including the root. (I've just noticed the docs don't say whether use_certificate_chain_file specifies SSL_BUILD_CHAIN_FLAG_NO_ROOT when it calls add1_chain_cert, so offhand I don't know whether this will cause the root to be included in the chain the server sends. But that shouldn't really matter.)
Reply | Threaded
Open this post in threaded view
|

RE: mutual-TLS / mTLS Example with certificate problem

Michael Wojcik
In reply to this post by Andreas Tengicki
> From: openssl-users [mailto:[hidden email]] On Behalf Of
> Andreas Tengicki
> Sent: Thursday, May 07, 2020 03:23
>
> 3) myserver vs openssl (and my client) is not working

Did you do what I told you to do in my previous message? That is:

> > Get rid of the call to use_certificate_file and put everything the server
> > should be sending into the chain file, in the order described in the OpenSSL
> > documentation: entity certificate, certificate for its issuer, and so on up
> > to and including the root.

--
Michael Wojcik
Distinguished Engineer, Micro Focus