OpenSSL not accepting a certificate, whilst curl does.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

OpenSSL not accepting a certificate, whilst curl does.

OpenSSL - User mailing list
Hi,

I'm really struggling to get my head around a specific scenario that isn't behaving as I expect.  Hopefully someone with more experience/knowledge can set me on the right path.

Note - my attempts to reproduce this in a lab have been unsuccessful, although I don't have access to the server private key, so the attempts have been with a completely independent CA chain.


I have a private CA, which has signed an intermediate certificate which has signed a server certificate for an internal web server which is used by various automated systems (all linux based).

The webserver (Apache) has the server cert and key, defined and in use as well as the intermediate certificate defined as the chain certificate - this all shows as expected.

I have then added the root certificate to the trusted certs for an automated system (populated `/etc/pki/ca-trust/source/anchors/` run `update-ca-trust extract`).

After this curl no longer complains about the certificate from the web server (expected).
However OpenSSL still does (unexpected), and I presume that for the same reason(s) urllib in Python also doesn't accept the certificate.
If I manually feed `openssl verify` the certificates and chain then they all come back "OK".

I've set up these systems a number of times with both self signed and CA signed certs and never seen this behaviour.

I'm slightly at a loss as to what diagnostics I even need at this point... so I've dropped a summary of relevant(?) diagnostics at this point below.

Thanks,


John

-- 

# Check that the root is installed into the trusted bundle:
# awk -v cmd='openssl x509 -noout -subject -serial -fingerprint; echo' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-bundle.trust.crt | grep -A1 CAROOT
subject= /CN=CAROOT/O=org/C=XX
serial=4D4C00241A7A17D0

# Check that the pem file I have is correct (serial matches above):
# openssl x509 -in CAROOT.pem -text | grep erial
        Serial Number: 5569826994213492688 (0x4d4c00241a7a17d0)

# Check that the chain is contiguous:
# openssl x509 -text -noout -in CAROOT.pem | grep -A1 -e Ident -e erial
        Serial Number: 5569826994213492688 (0x4d4c00241a7a17d0)
    Signature Algorithm: sha256WithRSAEncryption
            X509v3 Subject Key Identifier:
                2A:3E:33:88:7E:19:35:7C:6E:9D:7C:63:90:80:B8:DF:96:5A:A8:9D
            X509v3 Authority Key Identifier:
                keyid:2A:3E:33:88:7E:19:35:7C:6E:9D:7C:63:90:80:B8:DF:96:5A:A8:9D
# openssl x509 -text -noout -in CAINTER.pem | grep -A1 -e Ident
            X509v3 Subject Key Identifier:
                FB:17:C5:BB:BD:AD:84:65:4F:16:A7:E8:FA:95:1D:C7:D9:29:45:6A
            X509v3 Authority Key Identifier:
                keyid:2A:3E:33:88:7E:19:35:7C:6E:9D:7C:63:90:80:B8:DF:96:5A:A8:9D
# openssl x509 -text -noout -in SERVER.pem | grep -A1 -e Ident
            X509v3 Subject Key Identifier:
                F5:26:E2:09:A4:41:EC:EE:75:E2:4E:E4:02:90:B7:CD:EB:FC:4E:EC
            X509v3 Authority Key Identifier:
                keyid:FB:17:C5:BB:BD:AD:84:65:4F:16:A7:E8:FA:95:1D:C7:D9:29:45:6A

To my eye those all look lined up, and the serial on the root still agrees.
 
 
CURL:
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="https://server.fqdn:443/path/">here</a>.</p>
</body></html>

OpenSSL:
# openssl s_client -connect server.fqdn:443
CONNECTED(00000007)
depth=1 CN = CAINTER, O = org, C = XX
verify error:num=2:unable to get issuer certificate
issuer= CN = CAROOT, O = org, C = XX
 --8<--
Verify return code: 2 (unable to get issuer certificate)

Reply | Threaded
Open this post in threaded view
|

Re: OpenSSL not accepting a certificate, whilst curl does.

Tomas Mraz-2
On Mon, 2020-09-28 at 22:35 +0100, John Robson via openssl-users wrote:

> Hi,
>
> I'm really struggling to get my head around a specific scenario that
> isn't behaving as I expect.  Hopefully someone with more
> experience/knowledge can set me on the right path.
>
> Note - my attempts to reproduce this in a lab have been unsuccessful,
> although I don't have access to the server private key, so the
> attempts have been with a completely independent CA chain.
>
>
> I have a private CA, which has signed an intermediate certificate
> which has signed a server certificate for an internal web server
> which is used by various automated systems (all linux based).
>
> The webserver (Apache) has the server cert and key, defined and in
> use as well as the intermediate certificate defined as the chain
> certificate - this all shows as expected.
>
> I have then added the root certificate to the trusted certs for an
> automated system (populated `/etc/pki/ca-trust/source/anchors/` run
> `update-ca-trust extract`).
>
> After this curl no longer complains about the certificate from the
> web server (expected).
> However OpenSSL still does (unexpected), and I presume that for the
> same reason(s) urllib in Python also doesn't accept the certificate.
> If I manually feed `openssl verify` the certificates and chain then
> they all come back "OK".
>
> I've set up these systems a number of times with both self signed and
> CA signed certs and never seen this behaviour.

...

>  
>  
> CURL:
> # curl https://server.fqdn
> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
> <html><head>
> <title>302 Found</title>
> </head><body>
> <h1>Found</h1>
> <p>The document has moved <a href="
> https://server.fqdn:443/path/">here</a>.</p>
> </body></html>
>
> OpenSSL:
> # openssl s_client -connect server.fqdn:443
> CONNECTED(00000007)
> depth=1 CN = CAINTER, O = org, C = XX
> verify error:num=2:unable to get issuer certificate
> issuer= CN = CAROOT, O = org, C = XX
>  --8<--
> Verify return code: 2 (unable to get issuer certificate)

What is the curl library linked to? Is it using OpenSSL or something
else, for example NSS, as the TLS library? What exact system are you
testing on?

Are you sure you've put your CAROOT certificate to the system
certificate trust store? And/or is the trust store properly set up to
be loaded by OpenSSL by default?

--
Tomáš Mráz
No matter how far down the wrong road you've gone, turn back.
                                              Turkish proverb
[You'll know whether the road is wrong if you carefully listen to your
conscience.]


Reply | Threaded
Open this post in threaded view
|

Re: OpenSSL not accepting a certificate, whilst curl does.

Christian Heimes
In reply to this post by OpenSSL - User mailing list
On 28/09/2020 23.35, John Robson via openssl-users wrote:

> # curl https://server.fqdn
> <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
> <html><head>
> <title>302 Found</title>
> </head><body>
> <h1>Found</h1>
> <p>The document has moved <a
> href="https://server.fqdn:443/path/">here</a>.</p>
> </body></html>
>
> OpenSSL:
> # openssl s_client -connect server.fqdn:443
> CONNECTED(00000007)
> depth=1 CN = CAINTER, O = org, C = XX
> verify error:num=2:unable to get issuer certificate
> issuer= CN = CAROOT, O = org, C = XX
>  --8<--
> Verify return code: 2 (unable to get issuer certificate)

Does your server use named based virtual hosting? curl sends a TLS SNI
header automatically. With s_client you have to add -servername
server.fqdn to send a SNI header.

Christian

Reply | Threaded
Open this post in threaded view
|

Re: OpenSSL not accepting a certificate, whilst curl does.

OpenSSL - User mailing list
In reply to this post by Tomas Mraz-2
tl;dr: Found an issue with update-ca-trust extract
OpenSSL doing what it should, but update-ca-trust is only pushing the cert into some of the trust stores.

Thanks Tomas

On Tue, 29 Sep 2020 at 07:06, Tomas Mraz <[hidden email]> wrote:

>
> On Mon, 2020-09-28 at 22:35 +0100, John Robson via openssl-users wrote:
> > Hi,
> >
> > I'm really struggling to get my head around a specific scenario that
> > isn't behaving as I expect.  Hopefully someone with more
> > experience/knowledge can set me on the right path.
> >
> > Note - my attempts to reproduce this in a lab have been unsuccessful,
> > although I don't have access to the server private key, so the
> > attempts have been with a completely independent CA chain.
> >
> >
> > I have a private CA, which has signed an intermediate certificate
> > which has signed a server certificate for an internal web server
> > which is used by various automated systems (all linux based).
> >
> > The webserver (Apache) has the server cert and key, defined and in
> > use as well as the intermediate certificate defined as the chain
> > certificate - this all shows as expected.
> >
> > I have then added the root certificate to the trusted certs for an
> > automated system (populated `/etc/pki/ca-trust/source/anchors/` run
> > `update-ca-trust extract`).
> >
> > After this curl no longer complains about the certificate from the
> > web server (expected).
> > However OpenSSL still does (unexpected), and I presume that for the
> > same reason(s) urllib in Python also doesn't accept the certificate.
> > If I manually feed `openssl verify` the certificates and chain then
> > they all come back "OK".
> >
> > I've set up these systems a number of times with both self signed and
> > CA signed certs and never seen this behaviour.
>
> ...
> >
> > OpenSSL:
> > # openssl s_client -connect server.fqdn:443
> > CONNECTED(00000007)
> > depth=1 CN = CAINTER, O = org, C = XX
> > verify error:num=2:unable to get issuer certificate
> > issuer= CN = CAROOT, O = org, C = XX
> >  --8<--
> > Verify return code: 2 (unable to get issuer certificate)


> What is the curl library linked to? Is it using OpenSSL or something
> else, for example NSS, as the TLS library? What exact system are you
> testing on?

Running curl -v on a lab box it shows that curl is using NSS, which at least gives some reason for the different behaviour.
The system is a dockerised monitoring utility (clue is in the email address) with the containers based on CentOS.
 
> Are you sure you've put your CAROOT certificate to the system
> certificate trust store? And/or is the trust store properly set up to
> be loaded by OpenSSL by default?

Curl complains about the certificate until I add the cert to the ca-trust, so I have a good degree of confidence that the cert has been installed into the trust store - and on all the systems I've installed thus far this has also been picked up by OpenSSL.

openssl connect does negotiate a key and cipher, and there is a connection created, so I'm confident there isn't a key/cipher capability mismatch, it's just the certificate verification that is failing.

Checking where OpenSSL is using for it's CA trust store:
# openssl version -d
OPENSSLDIR: "/etc/pki/tls"

In that directory cert.pem is a symlink to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem, which is subtly different from the /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt I had been searching earlier - however I still see the ROOTCA entry in the pem bundle (and they have identical timestamps, indicating that both were updated by the update-ca-trust extract command).

I'm getting a closer look at those files to make sure that the cert has been correctly pushed into them. I can't imagine why it wouldn't have been, but it is the next logical thing to check.

OK - The issue is that update-ca-trust extract command i was using doesn't do what I would expect from the docs (and previous experience)

Even with the Root CA in the anchors directory it doesn't add it to the TLS ca store, it does add it elsewhere though.
If I manually add it to the ca-bundle.trust.crt then OpenSSL does verify the cert correctly, but the cert gets pulled if I then run the update again, so that's not a good long term solution.

Since I have the Root CA I can test adding it to a few different containers and see whether there is something about that cert that is causing the odd behaviour.

Thanks to Tomas for reminding me there are various different cert trust stores, and for ensuring that I followed the correct path deep enough to figure out what was going on.

Now to try and work out ca-certificate behaviour,

Cheers,

John