How to rotate cert when only first matching cert been verified

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

How to rotate cert when only first matching cert been verified

定平袁
Hello everyone,

Recently I am trying to rotate a cert, and the client uses python requests lib, which leverages openssl. Here is my steps:

1. Generate a new cert, and append it to the cert file(at this point, there are 2 certs in the file, first is old cert, second is new, they have the same Subject), restart client side process, (no problem here, because first cert matching server side cert, and it verifies successfully)
2. Replace server side with new cert.

As soon as I issue step #2, the client side process starts to show error “certificate verify failed”. This would cause downtime to my apps. I am new to this, not sure if there is anything wrong regarding my usage or understanding. But I found this page https://www.openssl.org/docs/man1.0.2/man3/SSL_CTX_load_verify_locations.html, it says the exact behavior like my test:

If several CA certificates matching the name, key identifier, and serial number condition are available, only the first one will be examined. This may lead to unexpected results if the same CA certificate is available with different expiration dates. If a "certificate expired" verification error occurs, no other certificate will be searched. Make sure to not have expired certificates mixed with valid ones.

So I am wondering how to rotate cert in such a case? It would be very helpful if anyone could help on this. Thanks.

BTW, I tested the same cert file with CURL (compiled with gnutls), it works fine.

Regards
Dingping
Reply | Threaded
Open this post in threaded view
|

Re: How to rotate cert when only first matching cert been verified

定平袁
the exact behavior:

When looking up CA certificates, the OpenSSL library will first search the certificates in CAfile, then those in CApath. Certificate matching is done based on the subject name, the key identifier (if present), and the serial number as taken from the certificate to be verified. If these data do not match, the next certificate will be tried. If a first certificate matching the parameters is found, the verification process will be performed; no other certificates for the same parameters will be searched in case of failure.

why no other certificates for the same parameters will be searched?

定平袁 <[hidden email]> 于2020年12月20日周日 上午8:59写道:
Hello everyone,

Recently I am trying to rotate a cert, and the client uses python requests lib, which leverages openssl. Here is my steps:

1. Generate a new cert, and append it to the cert file(at this point, there are 2 certs in the file, first is old cert, second is new, they have the same Subject), restart client side process, (no problem here, because first cert matching server side cert, and it verifies successfully)
2. Replace server side with new cert.

As soon as I issue step #2, the client side process starts to show error “certificate verify failed”. This would cause downtime to my apps. I am new to this, not sure if there is anything wrong regarding my usage or understanding. But I found this page https://www.openssl.org/docs/man1.0.2/man3/SSL_CTX_load_verify_locations.html, it says the exact behavior like my test:

If several CA certificates matching the name, key identifier, and serial number condition are available, only the first one will be examined. This may lead to unexpected results if the same CA certificate is available with different expiration dates. If a "certificate expired" verification error occurs, no other certificate will be searched. Make sure to not have expired certificates mixed with valid ones.

So I am wondering how to rotate cert in such a case? It would be very helpful if anyone could help on this. Thanks.

BTW, I tested the same cert file with CURL (compiled with gnutls), it works fine.

Regards
Dingping
Reply | Threaded
Open this post in threaded view
|

RE: How to rotate cert when only first matching cert been verified

Michael Wojcik
In reply to this post by 定平袁
> From: openssl-users <[hidden email]> On Behalf Of ???
> Sent: Saturday, 19 December, 2020 17:59

> 1. Generate a new cert, and append it to the cert file

Why are you appending it to the file containing the existing certificate?

> (at this point, there are 2 certs in the file, first is old cert, second is
> new, they have the same Subject), restart client side process, (no problem
> here, because first cert matching server side cert, and it verifies
> successfully)

> 2. Replace server side with new cert.

It sounds like you're updating the server's entity certificate.

> As soon as I issue step #2, the client side process starts to show error
> “certificate verify failed”.

There are many possible reasons for verification to fail.

> https://www.openssl.org/docs/man1.0.2/man3/SSL_CTX_load_verify_locations.html,
> it says the exact behavior like my test:

Similar symptoms, perhaps. But this page discusses "CA certificates" - that is,
intermediate and root certificates that have been configured to be trust anchors
or contributors to the trust chain. It has nothing to do with entity certificates,
which is what you're changing here.

You haven't given us enough information to guess why the new certificate is
failing client verification. You need to get detailed failure information from
the client program, or use a different client that gives you detailed information,
or use a utility such as "openssl verify" to test the certificate chain locally.

--
Michael Wojcik
Reply | Threaded
Open this post in threaded view
|

RE: How to rotate cert when only first matching cert been verified

Michael Wojcik
> From: 定平袁 <[hidden email]>
> Sent: Tuesday, 22 December, 2020 20:08
> To: Michael Wojcik <[hidden email]>

Please do not send messages regarding OpenSSL to me directly. Send them to the openss-users list. That is where the discussion belongs.

> > Why are you appending it to the file containing the existing certificate?

> I am rotating certificate, before the server side cert been replaced, the client
> side cert need to be valid, so when rotating, need both old and new cert exist.

I'm afraid it still isn't clear to me what you're doing. Both the server's entity certificate and the client's entity certificate are in the same file? What does this file contain before you append the new certificate?

> > It sounds like you're updating the server's entity certificate.

> I guess it's entity certificate (still trying to understand different cert
> concept...)

Does it identify the server, in the Subject DN and/or one or more Subject Alternative Name extensions?

> Below is the error message:

I'm afraid that message doesn't appear to contain any useful information.

> All the 3 clients used the same ca.crt file, which has an old cert in
> first, then a new cert behind. Only Python (used OpenSSL) failed.

So *this* sounds like what you're changing in this particular file is the set of trust anchors, not the entity certificates.

Where did your "CA" certificates come from? A commercial CA or some personal or organizational CA? From your description it sounds like the problem may be that the CA certificates were not generated correctly. Without the certificates to examine, we can't say.

Can you post the old and new certificates in PEM form in your next message?

Please note that due to the holidays I will not be reading email for several days, and it's likely that some other regular list members will be similarly unavailable.

--
Michael Wojcik
Reply | Threaded
Open this post in threaded view
|

Re: How to rotate cert when only first matching cert been verified

David von Oheimb-2
In reply to this post by 定平袁

定平袁 you are welcome.

The OpenSSL version you are using is way too old!
Do not use version 1.1.0, 1.0.x, and anything older - those versions are unsupported and must be considered insecure.

Yet since both your old and new server cert are not expired and have the same subject, keyIdentifier, and serial number,
and you appended the new server cert to your list, it is no surprise that the certificate chain building algorithm will pick up the old one.
For efficiency reasons, no other (equally applicable) certificates will be tried.
I've just clarified this and some further details in https://github.com/openssl/openssl/pull/13735.

I think Michael Wojcik already gave the right hint to solve your problem two days before:

Why are you appending it to the file containing the existing certificate?

So I suggest you better prepend the new certificate to that file rather than appending it,
or even better, remove the old (non-matching) certificate from that file.

Hope this helps,

    David


P.S.: I will be unavailable for several days, too.

On 23.12.20 04:15, 定平袁 wrote:
@David Thanks for you help!
This is my openssl version, and the self compiled curl backend
```
$ openssl version
OpenSSL 1.0.2g  1 Mar 2016

$ ldd /usr/bin/openssl  |grep ssl
libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f3099799000)

$ ldd ./lib/.libs/libcurl.so |grep ssl
libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f8720fd4000)
```
the system built-in curl binary:
```
$ ldd /usr/bin/curl  |grep tls
libcurl-gnutls.so.4 => /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 (0x00007f4b7fa07000)
libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f4b7e851000)
```
Actually, the old cert and new cert both are not expired yet, just the old cert is not consistent with server side. The new cert has the same content with server side imported cert(after replaced).

David von Oheimb <[hidden email]> 于2020年12月22日周二 下午10:27写道:

@定平袁, which version of OpenSSL are you using?

I've just checked: since OpenSSL 1.1.0, expired certificates are effectively not used for chain building.

    David

On 20.12.20 02:02, 定平袁 wrote:
the exact behavior:

When looking up CA certificates, the OpenSSL library will first search the certificates in CAfile, then those in CApath. Certificate matching is done based on the subject name, the key identifier (if present), and the serial number as taken from the certificate to be verified. If these data do not match, the next certificate will be tried. If a first certificate matching the parameters is found, the verification process will be performed; no other certificates for the same parameters will be searched in case of failure.

why no other certificates for the same parameters will be searched?

定平袁 <[hidden email]> 于2020年12月20日周日 上午8:59写道:
Hello everyone,

Recently I am trying to rotate a cert, and the client uses python requests lib, which leverages openssl. Here is my steps:

1. Generate a new cert, and append it to the cert file(at this point, there are 2 certs in the file, first is old cert, second is new, they have the same Subject), restart client side process, (no problem here, because first cert matching server side cert, and it verifies successfully)
2. Replace server side with new cert.

As soon as I issue step #2, the client side process starts to show error “certificate verify failed”. This would cause downtime to my apps. I am new to this, not sure if there is anything wrong regarding my usage or understanding. But I found this page https://www.openssl.org/docs/man1.0.2/man3/SSL_CTX_load_verify_locations.html, it says the exact behavior like my test:

If several CA certificates matching the name, key identifier, and serial number condition are available, only the first one will be examined. This may lead to unexpected results if the same CA certificate is available with different expiration dates. If a "certificate expired" verification error occurs, no other certificate will be searched. Make sure to not have expired certificates mixed with valid ones.

So I am wondering how to rotate cert in such a case? It would be very helpful if anyone could help on this. Thanks.

BTW, I tested the same cert file with CURL (compiled with gnutls), it works fine.

Regards
Dingping
Reply | Threaded
Open this post in threaded view
|

Re: How to rotate cert when only first matching cert been verified

Jochen Bern
In reply to this post by 定平袁
On 23.12.20 23:56, [hidden email] digested:
> Message: 4
> Date: Wed, 23 Dec 2020 23:56:44 +0100
> From: David von Oheimb <[hidden email]>
[...]
> Yet since both your old and new server cert are not expired and have the
> same subject, keyIdentifier, and serial number,
> and you appended the new server cert to your list, it is no surprise
> that the certificate chain building algorithm will pick up the old one.
> For efficiency reasons, no other (equally applicable) certificates will
> be tried.

To expand on the "*should* you actually do it like this" angle: I do not
see any reason why the new server cert (SC) should have *the same serial
number* (SN) as the old one.

At least in the general case - where the CA and the server are run by
different entities -, the CA wants(*) to be able to revoke old and new
SC separately, and CRLs identify revoked certs exclusively by the
issuing CA Cert (CC) and the revoked cert's SN.

So, what *is* the rationale to reuse the SN? Do you have a
"verification" mechanism somewhere that (cannot be updated in a timely
manner for the new SC and) would protest a changed SN, but *not* the
changed validity period (or, for that matter, fingerprint or CA
signature)? Note that the mere thought already makes me put quote marks
around "verification" ...

Disclaimer: I'm *not* saying that merely using different SNs will make
the problem you're currently experiencing disappear. In fact, I consider
that rather unlikely, but it might be one contributing factor.

(*) Scenario 1: Before the old SC expires, the CA finds out that it
issued a new SC to an imposter, so they now want to revoke the new but
not the old. Scenario 2: The old SC is found to have been leaked after
the new one was already issued, so at least the server admin would
prefer to have the old SC revoked but *not* the new one.

Kind regards,
--
Jochen Bern
Systemingenieur

Binect GmbH


smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: How to rotate cert when only first matching cert been verified

定平袁
Thanks a lot for your reply! Merry Christmas!

[hidden email]   Apologies. I clicked the wrong reply button.

[hidden email] I will update to a new version and try again. To append cert is to make sure new cert and old cert both exist in trust store, thus when server switches cert, it can be trusted by client.

@Jochen actually, the certs have different SN, which indeed is not consistent with the man doc. The thing that confuses me is that CURL (compiled with gnutls) and Golang works.
below is my ca.crt file, I am not sure where it went wrong, maybe just my wrong behavior?

```

-----BEGIN CERTIFICATE-----
MIIFdzCCA1+gAwIBAgIJAJcvKUQ0Bz4tMA0GCSqGSIb3DQEBCwUAMGUxCzAJBgNV
BAYTAlVTMQswCQYDVQQIDAJDQTESMBAGA1UEBwwJUGFsbyBBbHRvMQ8wDQYDVQQK
DAZWTXdhcmUxJDAiBgNVBAMMG25zeG1hbmFnZXIucGtzLnZtd2FyZS5sb2NhbDAe
Fw0yMDEyMTcwMDM2MjBaFw0zMDEyMTUwMDM2MjBaMGUxCzAJBgNVBAYTAlVTMQsw
CQYDVQQIDAJDQTESMBAGA1UEBwwJUGFsbyBBbHRvMQ8wDQYDVQQKDAZWTXdhcmUx
JDAiBgNVBAMMG25zeG1hbmFnZXIucGtzLnZtd2FyZS5sb2NhbDCCAiIwDQYJKoZI
hvcNAQEBBQADggIPADCCAgoCggIBAMC4EFsukdnrj26EYSaCCrvUtEhbi33wXHKi
6utmOe9r+M17Q0MArjJeEzklmrTkj+qKJCB4TgWFY2djJ+hA0a5I2eOn/0OjJ0c2
67FcqX7pq1JwYMSkwN4dQUbAN82xjQOcmj03PVjgLQSFXLfNxcfym0G2KtwkIg8K
V4JwC0L048BBu/EynAXA2kYHXiJ6uSjeMOuTyogmVilzUOjfJztaNj2jpq3D8sek
qtRNBYRcgSwx1wq7uWSe6qjHVDmom4nlUQznOZfJYodFWZll6Wv8Itk28ovhIhgk
G9wJv3QJp6Gef1GN22Q7KU09/ZG61PRPVgoPTuRxHKn75aKl6FJcztvz/X4egt9K
yGxsxEtWrLW52U1EUVg0zVUO/VAbtm1NLsEGv1L19vYjg6gpU4zQjP7enuSFqvKo
rLLDvSzUWRzXIDwWSWGNBoAkry8jZmKWnjHqSW2EVbCaFTXcIQ6kPQGYvH3cFUyG
fW06NlCL+AYGNaOVJkL7J3RYH+5cstGTpCNpyAmYNsEs1G+yXwCH5aDcP/0qbU2W
WXO0Jh/+2KhmZ1Op1o6x69FLQ+g/0m705nGhx8NQWC3V+BC/mUdyXlom7yZde+uT
qZS/0K7z/O8FpNwAddLmhgNHq2cHRjQFH6WeAhw3tBLGS5OFAP23SG/OItEaWp7h
nXgRedMVAgMBAAGjKjAoMCYGA1UdEQQfMB2CG25zeG1hbmFnZXIucGtzLnZtd2Fy
ZS5sb2NhbDANBgkqhkiG9w0BAQsFAAOCAgEAlfMDgcI6DiRH7eRJfg0SrtkRSAIe
0icQ8RH6Z8SBYIbPnzR2qeAm0V7BV7qGSOHGb1ezghCXQAjL2JF1pHw9aKZ0ST49
vZSlkp6tKojk1HZqa3OSfji+o8ROSvpfBW+qYqgsTkSD0VqZ4xkGUnXaRbQ3H+2V
CV/MsXn/lgJ1pXDhNifUBtTa4OQx3WsA74lh7pddtbEWQJbFPwDvwzKo62P8b6zq
MDhccVBmV5QZDwGH3v9Dy6QHq91b1grMkIQb67e1E6VQia6++Sq8b8ZCOJ1VUOjt
I7KTIco57dLyIJPO+wvTKKpLraFIGUxNBwVOnI6wekUlhhhMcXvL/dNbD8htO/SQ
VtiB8BL8SJ8HlRy2REDwvNMj0ChWeFjimb6k/40vKet3lmmAwewjy4OWBkkfrv3Y
/I+RQ8Ua3vsz8KZywZvXAYWTTnsFbsHQBv9TgI0crKajVgm06stz7X+RHmhVyckV
54nSQhzZPagxfwJNzcKNb+HMr57D6SNl8xYLK1V5lmDjtAFeII3fnCJpCszNptKy
cHY8Jq1eb5no5cAK7WfvepVQD0CGR6JhEuNpYNa0bp6uGTYv9EqYYqrNq8cx/41v
jaNI9N6oqi3Qqt+MARXXLgMjl1CYZQ7mNT0pOXPC6gEFoyKhTnDmACAV82WB1ClR
ZlY/eRzAK/iXECs=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFdzCCA1+gAwIBAgIJAITnARyY8iCRMA0GCSqGSIb3DQEBCwUAMGUxCzAJBgNV
BAYTAlVTMQswCQYDVQQIDAJDQTESMBAGA1UEBwwJUGFsbyBBbHRvMQ8wDQYDVQQK
DAZWTXdhcmUxJDAiBgNVBAMMG25zeG1hbmFnZXIucGtzLnZtd2FyZS5sb2NhbDAe
Fw0yMDEyMTUwOTIzMjVaFw0zMDEyMTMwOTIzMjVaMGUxCzAJBgNVBAYTAlVTMQsw
CQYDVQQIDAJDQTESMBAGA1UEBwwJUGFsbyBBbHRvMQ8wDQYDVQQKDAZWTXdhcmUx
JDAiBgNVBAMMG25zeG1hbmFnZXIucGtzLnZtd2FyZS5sb2NhbDCCAiIwDQYJKoZI
hvcNAQEBBQADggIPADCCAgoCggIBAJ1vvdCHRdGFvleEAGANFv9ttVAa4DdewpKK
M+DCyOnRfnsfJWxtTSpzu+nDQg9/wvFs2RQBu+Yh1iF40KVc6aYMDjxb+4uAC2nR
/0g8ANGXYE1BoDShJwwTosWrQ6RaPTLw3rK4U6+OW//g7EHUR9LRHNRRdItbyXkT
ULQac4x/k8ApwXQvFZ6Vb/L+nNBUBJQapWoi361v7Z2fxzmJwB9D+KfGU4pMKKL7
/VuMvDaZuxzeAnPdkaYrmF8XlnUr5ZoW85xWLVLPPRjDqcNiKcXBhUHWUB3+RzEc
1leLcX9yrtiJjO91hTzsTPvd4Tqi8ojyY+SILJiqJRDNcrVtrW+leVlxOGcLgnT3
gR3EB5zAaT8z+RBMn+SPJSUKslh1P/bAyOaPLg3NQwTpk/gDoShGva01y/7/kBnk
nvkz6mTl+UZIWCj5cI7a3+zkR6ptNZDArn2JpFW1ePmnQjz+Bt7y6tueJxnj8Q5M
cUbEOhcqfzadpJort0/70STtR0LSvLe0Q+8r1sTDuO9RXjqqdbveyp9w+dUFW1et
SF/w+ak3f4nZZCjI0FU68HtzNmmqPdgKJuE197J4XNVyCHQW1h0X2zURyvGOYp5D
UHsdQYfm6G0aw3VppiT71t5BeBQi2Z6jyVVqGGBf36rhbp8BsP5FxTQI7apXR/u3
jhTblAvHAgMBAAGjKjAoMCYGA1UdEQQfMB2CG25zeG1hbmFnZXIucGtzLnZtd2Fy
ZS5sb2NhbDANBgkqhkiG9w0BAQsFAAOCAgEAlvoh8fFQpAzElmkIVLBr739cscLz
ALXnBgAFPhR/leoZjdEdHfq7Pm80dtEaluCrm81MX1wKiCJKgA6oAzAf7vK1seu5
Mx4yu9hwpNE9xXheea5cASzvR355JPjvUdFohChuvnVcPV0yZdVzEOhtmyrYPCHd
OYcEA0xyV2sqKZRil39dHRi1VRoALZL8n2UHZa1EN0wTHfRKdmx9QOAxhsxhNSMg
kiCMGe9OoYfcU98dlXNclvkIqkVl8RN6W4A8z/7VFB/Aq3NQBfGeTR3l/+dZH+e0
boioZDkpGRVCtfYyjvfPRUeMJXgqUfdMIsQGm0YbtQ0PWhIhjdxiuLUJ4jEqen8G
5ssz0/V4vlJ0wgkhliQcybxRhCWayKr95kuV6yiHKZgpTX9ovOhE+Ew208Y6Poh3
vR7YAWfyI7QxPAhSuLMQFKtRbD2cbAQ/CD+CsFVquiGj8J6DUS+pWPr5JHNz8rzA
Ba29dMTPeKmbbW3aHZ4pA2aJNT5lmA6RQ85cR7oNU48HAhwSqpw23NZQb2MF7Qqp
cTey+etb2kVR83fp47g2hfgzCBKoTYdqC5G5kVarvO1+BsdKwApz+iElUqKfkRZo
NwHJp5KUauGKGrN2WY5yAMUq9iEsVlTBt+rsixtnRlP1yhGhc9DrLsKquOw03myL
hDISqFnOh+zVz10=
-----END CERTIFICATE-----

```

Jochen Bern <[hidden email]> 于2020年12月24日周四 下午7:44写道:
On 23.12.20 23:56, [hidden email] digested:
> Message: 4
> Date: Wed, 23 Dec 2020 23:56:44 +0100
> From: David von Oheimb <[hidden email]>
[...]
> Yet since both your old and new server cert are not expired and have the
> same subject, keyIdentifier, and serial number,
> and you appended the new server cert to your list, it is no surprise
> that the certificate chain building algorithm will pick up the old one.
> For efficiency reasons, no other (equally applicable) certificates will
> be tried.

To expand on the "*should* you actually do it like this" angle: I do not
see any reason why the new server cert (SC) should have *the same serial
number* (SN) as the old one.

At least in the general case - where the CA and the server are run by
different entities -, the CA wants(*) to be able to revoke old and new
SC separately, and CRLs identify revoked certs exclusively by the
issuing CA Cert (CC) and the revoked cert's SN.

So, what *is* the rationale to reuse the SN? Do you have a
"verification" mechanism somewhere that (cannot be updated in a timely
manner for the new SC and) would protest a changed SN, but *not* the
changed validity period (or, for that matter, fingerprint or CA
signature)? Note that the mere thought already makes me put quote marks
around "verification" ...

Disclaimer: I'm *not* saying that merely using different SNs will make
the problem you're currently experiencing disappear. In fact, I consider
that rather unlikely, but it might be one contributing factor.

(*) Scenario 1: Before the old SC expires, the CA finds out that it
issued a new SC to an imposter, so they now want to revoke the new but
not the old. Scenario 2: The old SC is found to have been leaked after
the new one was already issued, so at least the server admin would
prefer to have the old SC revoked but *not* the new one.

Kind regards,
--
Jochen Bern
Systemingenieur

Binect GmbH

Reply | Threaded
Open this post in threaded view
|

Re: How to rotate cert when only first matching cert been verified

定平袁
Re post my code here, since I send it alone to Michael.

Below is the error message:

```
Traceback (most recent call last):
  File "test.py", line 6, in <module>
    r = s.get('https://nsxmanager.pks.vmware.local/api/v1/spec/vmware/types/Tag', verify='./ca.pem')
  File "/home/kubo/.local/lib/python2.7/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/home/kubo/.local/lib/python2.7/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/kubo/.local/lib/python2.7/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/kubo/.local/lib/python2.7/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='nsxmanager.pks.vmware.local', port=443): Max retries exceeded with url: /api/v1/spec/vmware/types/Tag (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)'),))
```

but the CURL command and Golang code I used to access client, it shows CA cert.

My python code(which report above error):

```
import requests

s = requests.Session()
s.auth = ('admin', 'Admin!23Admin')
r = s.get('https://nsxmanager.pks.vmware.local/api/v1/spec/vmware/types/Tag', verify='./ca.crt')
print(r.status_code)
```

CURL:
```
curl -I -u admin:'Admin!23Admin' https://nsxmanager.pks.vmware.local/api/v1/spec/vmware/types/Tag --cacert ./ca.crt
```

Golang:
```
package main

import (
    "crypto/tls"
    "io/ioutil"
    "log"
    "fmt"
    "net/http"
    "crypto/x509"
)

func main() {
    caCert, err := ioutil.ReadFile("./ca.crt")
    if err != nil {
        log.Fatal(err)
    }
    caCertPool := x509.NewCertPool()
    caCertPool.AppendCertsFromPEM(caCert)

    client := &http.Client{
        Transport: &http.Transport{
            TLSClientConfig: &tls.Config{
                RootCAs:      caCertPool,
            },
        },
    }
    req, err := http.NewRequest("GET", "https://nsxmanager.pks.vmware.local/api/v1/spec/vmware/types/Tag", nil)
    req.SetBasicAuth("admin", "Admin!23Admin")
    r, err := client.Do(req)
    if err != nil {
        panic(err)
    }
    fmt.Println(r.Status)
}
```

All the 3 clients used the same ca.crt file, which has an old cert in the first, then a new cert behind. Only Python (used OpenSSL) failed.
After I compile curl with openssl backend, the new binary failed too.

```
./curl.openssl -vvvv -u admin:'Admin!23Admin' https://nsxmanager.pks.vmware.local/api/v1/spec/vmware/types/Tag --cacert ./ca.crt
*   Trying 192.168.111.4:443...
* Connected to nsxmanager.pks.vmware.local (192.168.111.4) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*  CAfile: ./ca.crt
*  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: self signed certificate
* Closing connection 0
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
```
Is above information enough?

定平袁 <[hidden email]> 于2020年12月25日周五 上午7:35写道:
Thanks a lot for your reply! Merry Christmas!

[hidden email]   Apologies. I clicked the wrong reply button.

[hidden email] I will update to a new version and try again. To append cert is to make sure new cert and old cert both exist in trust store, thus when server switches cert, it can be trusted by client.

@Jochen actually, the certs have different SN, which indeed is not consistent with the man doc. The thing that confuses me is that CURL (compiled with gnutls) and Golang works.
below is my ca.crt file, I am not sure where it went wrong, maybe just my wrong behavior?

```

-----BEGIN CERTIFICATE-----
MIIFdzCCA1+gAwIBAgIJAJcvKUQ0Bz4tMA0GCSqGSIb3DQEBCwUAMGUxCzAJBgNV
BAYTAlVTMQswCQYDVQQIDAJDQTESMBAGA1UEBwwJUGFsbyBBbHRvMQ8wDQYDVQQK
DAZWTXdhcmUxJDAiBgNVBAMMG25zeG1hbmFnZXIucGtzLnZtd2FyZS5sb2NhbDAe
Fw0yMDEyMTcwMDM2MjBaFw0zMDEyMTUwMDM2MjBaMGUxCzAJBgNVBAYTAlVTMQsw
CQYDVQQIDAJDQTESMBAGA1UEBwwJUGFsbyBBbHRvMQ8wDQYDVQQKDAZWTXdhcmUx
JDAiBgNVBAMMG25zeG1hbmFnZXIucGtzLnZtd2FyZS5sb2NhbDCCAiIwDQYJKoZI
hvcNAQEBBQADggIPADCCAgoCggIBAMC4EFsukdnrj26EYSaCCrvUtEhbi33wXHKi
6utmOe9r+M17Q0MArjJeEzklmrTkj+qKJCB4TgWFY2djJ+hA0a5I2eOn/0OjJ0c2
67FcqX7pq1JwYMSkwN4dQUbAN82xjQOcmj03PVjgLQSFXLfNxcfym0G2KtwkIg8K
V4JwC0L048BBu/EynAXA2kYHXiJ6uSjeMOuTyogmVilzUOjfJztaNj2jpq3D8sek
qtRNBYRcgSwx1wq7uWSe6qjHVDmom4nlUQznOZfJYodFWZll6Wv8Itk28ovhIhgk
G9wJv3QJp6Gef1GN22Q7KU09/ZG61PRPVgoPTuRxHKn75aKl6FJcztvz/X4egt9K
yGxsxEtWrLW52U1EUVg0zVUO/VAbtm1NLsEGv1L19vYjg6gpU4zQjP7enuSFqvKo
rLLDvSzUWRzXIDwWSWGNBoAkry8jZmKWnjHqSW2EVbCaFTXcIQ6kPQGYvH3cFUyG
fW06NlCL+AYGNaOVJkL7J3RYH+5cstGTpCNpyAmYNsEs1G+yXwCH5aDcP/0qbU2W
WXO0Jh/+2KhmZ1Op1o6x69FLQ+g/0m705nGhx8NQWC3V+BC/mUdyXlom7yZde+uT
qZS/0K7z/O8FpNwAddLmhgNHq2cHRjQFH6WeAhw3tBLGS5OFAP23SG/OItEaWp7h
nXgRedMVAgMBAAGjKjAoMCYGA1UdEQQfMB2CG25zeG1hbmFnZXIucGtzLnZtd2Fy
ZS5sb2NhbDANBgkqhkiG9w0BAQsFAAOCAgEAlfMDgcI6DiRH7eRJfg0SrtkRSAIe
0icQ8RH6Z8SBYIbPnzR2qeAm0V7BV7qGSOHGb1ezghCXQAjL2JF1pHw9aKZ0ST49
vZSlkp6tKojk1HZqa3OSfji+o8ROSvpfBW+qYqgsTkSD0VqZ4xkGUnXaRbQ3H+2V
CV/MsXn/lgJ1pXDhNifUBtTa4OQx3WsA74lh7pddtbEWQJbFPwDvwzKo62P8b6zq
MDhccVBmV5QZDwGH3v9Dy6QHq91b1grMkIQb67e1E6VQia6++Sq8b8ZCOJ1VUOjt
I7KTIco57dLyIJPO+wvTKKpLraFIGUxNBwVOnI6wekUlhhhMcXvL/dNbD8htO/SQ
VtiB8BL8SJ8HlRy2REDwvNMj0ChWeFjimb6k/40vKet3lmmAwewjy4OWBkkfrv3Y
/I+RQ8Ua3vsz8KZywZvXAYWTTnsFbsHQBv9TgI0crKajVgm06stz7X+RHmhVyckV
54nSQhzZPagxfwJNzcKNb+HMr57D6SNl8xYLK1V5lmDjtAFeII3fnCJpCszNptKy
cHY8Jq1eb5no5cAK7WfvepVQD0CGR6JhEuNpYNa0bp6uGTYv9EqYYqrNq8cx/41v
jaNI9N6oqi3Qqt+MARXXLgMjl1CYZQ7mNT0pOXPC6gEFoyKhTnDmACAV82WB1ClR
ZlY/eRzAK/iXECs=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFdzCCA1+gAwIBAgIJAITnARyY8iCRMA0GCSqGSIb3DQEBCwUAMGUxCzAJBgNV
BAYTAlVTMQswCQYDVQQIDAJDQTESMBAGA1UEBwwJUGFsbyBBbHRvMQ8wDQYDVQQK
DAZWTXdhcmUxJDAiBgNVBAMMG25zeG1hbmFnZXIucGtzLnZtd2FyZS5sb2NhbDAe
Fw0yMDEyMTUwOTIzMjVaFw0zMDEyMTMwOTIzMjVaMGUxCzAJBgNVBAYTAlVTMQsw
CQYDVQQIDAJDQTESMBAGA1UEBwwJUGFsbyBBbHRvMQ8wDQYDVQQKDAZWTXdhcmUx
JDAiBgNVBAMMG25zeG1hbmFnZXIucGtzLnZtd2FyZS5sb2NhbDCCAiIwDQYJKoZI
hvcNAQEBBQADggIPADCCAgoCggIBAJ1vvdCHRdGFvleEAGANFv9ttVAa4DdewpKK
M+DCyOnRfnsfJWxtTSpzu+nDQg9/wvFs2RQBu+Yh1iF40KVc6aYMDjxb+4uAC2nR
/0g8ANGXYE1BoDShJwwTosWrQ6RaPTLw3rK4U6+OW//g7EHUR9LRHNRRdItbyXkT
ULQac4x/k8ApwXQvFZ6Vb/L+nNBUBJQapWoi361v7Z2fxzmJwB9D+KfGU4pMKKL7
/VuMvDaZuxzeAnPdkaYrmF8XlnUr5ZoW85xWLVLPPRjDqcNiKcXBhUHWUB3+RzEc
1leLcX9yrtiJjO91hTzsTPvd4Tqi8ojyY+SILJiqJRDNcrVtrW+leVlxOGcLgnT3
gR3EB5zAaT8z+RBMn+SPJSUKslh1P/bAyOaPLg3NQwTpk/gDoShGva01y/7/kBnk
nvkz6mTl+UZIWCj5cI7a3+zkR6ptNZDArn2JpFW1ePmnQjz+Bt7y6tueJxnj8Q5M
cUbEOhcqfzadpJort0/70STtR0LSvLe0Q+8r1sTDuO9RXjqqdbveyp9w+dUFW1et
SF/w+ak3f4nZZCjI0FU68HtzNmmqPdgKJuE197J4XNVyCHQW1h0X2zURyvGOYp5D
UHsdQYfm6G0aw3VppiT71t5BeBQi2Z6jyVVqGGBf36rhbp8BsP5FxTQI7apXR/u3
jhTblAvHAgMBAAGjKjAoMCYGA1UdEQQfMB2CG25zeG1hbmFnZXIucGtzLnZtd2Fy
ZS5sb2NhbDANBgkqhkiG9w0BAQsFAAOCAgEAlvoh8fFQpAzElmkIVLBr739cscLz
ALXnBgAFPhR/leoZjdEdHfq7Pm80dtEaluCrm81MX1wKiCJKgA6oAzAf7vK1seu5
Mx4yu9hwpNE9xXheea5cASzvR355JPjvUdFohChuvnVcPV0yZdVzEOhtmyrYPCHd
OYcEA0xyV2sqKZRil39dHRi1VRoALZL8n2UHZa1EN0wTHfRKdmx9QOAxhsxhNSMg
kiCMGe9OoYfcU98dlXNclvkIqkVl8RN6W4A8z/7VFB/Aq3NQBfGeTR3l/+dZH+e0
boioZDkpGRVCtfYyjvfPRUeMJXgqUfdMIsQGm0YbtQ0PWhIhjdxiuLUJ4jEqen8G
5ssz0/V4vlJ0wgkhliQcybxRhCWayKr95kuV6yiHKZgpTX9ovOhE+Ew208Y6Poh3
vR7YAWfyI7QxPAhSuLMQFKtRbD2cbAQ/CD+CsFVquiGj8J6DUS+pWPr5JHNz8rzA
Ba29dMTPeKmbbW3aHZ4pA2aJNT5lmA6RQ85cR7oNU48HAhwSqpw23NZQb2MF7Qqp
cTey+etb2kVR83fp47g2hfgzCBKoTYdqC5G5kVarvO1+BsdKwApz+iElUqKfkRZo
NwHJp5KUauGKGrN2WY5yAMUq9iEsVlTBt+rsixtnRlP1yhGhc9DrLsKquOw03myL
hDISqFnOh+zVz10=
-----END CERTIFICATE-----

```

Jochen Bern <[hidden email]> 于2020年12月24日周四 下午7:44写道:
On 23.12.20 23:56, [hidden email] digested:
> Message: 4
> Date: Wed, 23 Dec 2020 23:56:44 +0100
> From: David von Oheimb <[hidden email]>
[...]
> Yet since both your old and new server cert are not expired and have the
> same subject, keyIdentifier, and serial number,
> and you appended the new server cert to your list, it is no surprise
> that the certificate chain building algorithm will pick up the old one.
> For efficiency reasons, no other (equally applicable) certificates will
> be tried.

To expand on the "*should* you actually do it like this" angle: I do not
see any reason why the new server cert (SC) should have *the same serial
number* (SN) as the old one.

At least in the general case - where the CA and the server are run by
different entities -, the CA wants(*) to be able to revoke old and new
SC separately, and CRLs identify revoked certs exclusively by the
issuing CA Cert (CC) and the revoked cert's SN.

So, what *is* the rationale to reuse the SN? Do you have a
"verification" mechanism somewhere that (cannot be updated in a timely
manner for the new SC and) would protest a changed SN, but *not* the
changed validity period (or, for that matter, fingerprint or CA
signature)? Note that the mere thought already makes me put quote marks
around "verification" ...

Disclaimer: I'm *not* saying that merely using different SNs will make
the problem you're currently experiencing disappear. In fact, I consider
that rather unlikely, but it might be one contributing factor.

(*) Scenario 1: Before the old SC expires, the CA finds out that it
issued a new SC to an imposter, so they now want to revoke the new but
not the old. Scenario 2: The old SC is found to have been leaked after
the new one was already issued, so at least the server admin would
prefer to have the old SC revoked but *not* the new one.

Kind regards,
--
Jochen Bern
Systemingenieur

Binect GmbH

Reply | Threaded
Open this post in threaded view
|

Directly trusted self-issued end-entity certs - Re: How to rotate cert when only first matching cert been verified

David von Oheimb-2
In reply to this post by 定平袁
On 25.12.20 00:35, 定平袁 wrote:
[hidden email] I will update to a new version and try again.

Good. Ideally try also a current 3.0.0 alpha release because there have been some changes to cert chain building and verification recently.

To append cert is to make sure new cert and old cert both exist in trust store, thus when server switches cert, it can be trusted by client.
Understood, but my point was on a different aspect:
The chain building will take the first matching cert, so if you want to prefer the new cert, it must be in the list *before* the old one -
in other words, prepend the new cert to the list rather than appending to it.

@Jochen actually, the certs have different SN, which indeed is not consistent with the man doc

Different certs with the same issuer indeed must have different SNs (except in the special case I mention below).
See also RFC 5280 section 4.1.2.2 https://tools.ietf.org/html/rfc5280#section-4.1.2.2:

  It MUST be unique for each certificate issued by a given CA
     (i.e., the issuer name and serial number identify a unique certificate). 


Yet there is a different inconsistency in what you write:

The thing that confuses me is that CURL (compiled with gnutls) and Golang works.
below is my ca.crt file, I am not sure where it went wrong, maybe just my wrong behavior?
You refer to them as CA certs, but they are not: they do no have a basicConstraints field with the cA bit set.
And as far as I understand your scenario, they are not used to issue other certs but by some (TLS) server,
so they really are end-entity (EE) certs, not CA certs, and it looks like this is correct in your application scenario.

Directly trusted self-issued EE certs (which may be self-signed or not) are a special situation.
This has been clarified in RFC 6818 (which updates RFC 5280) https://tools.ietf.org/html/rfc6818#section-2:
| Consistent with Section 3.4.61 of X.509 (11/2008) [X.509], we note
| that use of self-issued certificates and self-signed certificates
| issued by entities other than CAs are outside the scope of this
| specification.  Thus, for example, a web server or client might
| generate a self-signed certificate to identify itself.  These
| certificates and how a relying party uses them to authenticate
| asserted identities are both outside the scope of RFC 5280.
So the path building and verification, as well as other checks defined RFC 5280, does not apply to them at all!
They are essentially just a convenient container for a public key, where it is optional to check expiration etc.


Unfortunately, when using such certs for TLS connections etc., still verification is done on them, which may fail.
After renaming your ca.crt file to ee.crt for clarity and extracting the first cert in ee1.crt and the second one in ee2.crt,
when verifying these directly trusted certs one gets the problem you reported:

openssl verify -x509_strict -trusted ee.crt ee1.crt
ee1.crt: OK

openssl verify -x509_strict -trusted ee.crt ee2.crt
C = US, ST = CA, L = Palo Alto, O = VMware, CN = nsxmanager.pks.vmware.local
error 18 at 0 depth lookup: self signed certificate
error ee2.crt: verification failed

So as I wrote before, unfortunately the path building picks up the first matching cert from ee.crt,
which is the one in ee1.crt (i.e., your old one), and does not try the second one (i.e., your new one).
This happens also with the latest OpenSSL pre-3.0.0 master.


A solution is to add both the subjectKeyIdentifier and authorityKeyIdentifier extensions to your certs,
for instance like this:

echo >ee.cnf "
prompt = no
distinguished_name = my_server
x509_extensions = my_exts
[my_server]
commonName = test
[my_exts]
basicConstraints = CA:false
subjectKeyIdentifier=hash
authorityKeyIdentifier = keyid"

openssl req -config ee.cnf -new -x509 -out ee1.crt -nodes -keyout ee1.pem
openssl req -config ee.cnf -new -x509 -out ee2.crt -nodes -keyout ee2.pem
cat ee1.crt ee2.crt >ee.crt

The subjectKeyIdentifier and authorityKeyIdentifier extensions are generally recommend
(and actually required to add for certs that are RFC 5280 compliant)
because they help for correct chain building, and indeed also in this case they do:

openssl verify -x509_strict -trusted ee.crt ee1.crt
ee1.crt: OK
openssl verify -x509_strict -trusted ee.crt ee2.crt
ee2.crt: OK

Regards,

    David


Reply | Threaded
Open this post in threaded view
|

Re: Directly trusted self-issued end-entity certs - Re: How to rotate cert when only first matching cert been verified

定平袁
[hidden email]
Thank you so much for your deep investigation! 
With subjectKeyIdentifier and authorityKeyIdentifier extensions, it works like a charm!

So, the former statements I found on this page only applies to CA cert, not EE cert.
How to pick up cert from trust store(or cert container as you say)
is decided by different implementation themselves, do I understand correctly?

Since GnuTls and golang could pick up the right cert in this kind of scenario,
they must implement their own logic to pick up the right cert, do you think OpenSSL
will implement this logic too? Or it's a more appropriate approach to just
use the extensions you suggested?

Regards,
Dingping

David von Oheimb <[hidden email]> 于2020年12月26日周六 下午5:17写道:
On 25.12.20 00:35, 定平袁 wrote:
[hidden email] I will update to a new version and try again.

Good. Ideally try also a current 3.0.0 alpha release because there have been some changes to cert chain building and verification recently.

To append cert is to make sure new cert and old cert both exist in trust store, thus when server switches cert, it can be trusted by client.
Understood, but my point was on a different aspect:
The chain building will take the first matching cert, so if you want to prefer the new cert, it must be in the list *before* the old one -
in other words, prepend the new cert to the list rather than appending to it.

@Jochen actually, the certs have different SN, which indeed is not consistent with the man doc

Different certs with the same issuer indeed must have different SNs (except in the special case I mention below).
See also RFC 5280 section 4.1.2.2 https://tools.ietf.org/html/rfc5280#section-4.1.2.2:

  It MUST be unique for each certificate issued by a given CA
     (i.e., the issuer name and serial number identify a unique certificate). 


Yet there is a different inconsistency in what you write:

The thing that confuses me is that CURL (compiled with gnutls) and Golang works.
below is my ca.crt file, I am not sure where it went wrong, maybe just my wrong behavior?
You refer to them as CA certs, but they are not: they do no have a basicConstraints field with the cA bit set.
And as far as I understand your scenario, they are not used to issue other certs but by some (TLS) server,
so they really are end-entity (EE) certs, not CA certs, and it looks like this is correct in your application scenario.

Directly trusted self-issued EE certs (which may be self-signed or not) are a special situation.
This has been clarified in RFC 6818 (which updates RFC 5280) https://tools.ietf.org/html/rfc6818#section-2:
| Consistent with Section 3.4.61 of X.509 (11/2008) [X.509], we note
| that use of self-issued certificates and self-signed certificates
| issued by entities other than CAs are outside the scope of this
| specification.  Thus, for example, a web server or client might
| generate a self-signed certificate to identify itself.  These
| certificates and how a relying party uses them to authenticate
| asserted identities are both outside the scope of RFC 5280.
So the path building and verification, as well as other checks defined RFC 5280, does not apply to them at all!
They are essentially just a convenient container for a public key, where it is optional to check expiration etc.


Unfortunately, when using such certs for TLS connections etc., still verification is done on them, which may fail.
After renaming your ca.crt file to ee.crt for clarity and extracting the first cert in ee1.crt and the second one in ee2.crt,
when verifying these directly trusted certs one gets the problem you reported:

openssl verify -x509_strict -trusted ee.crt ee1.crt
ee1.crt: OK

openssl verify -x509_strict -trusted ee.crt ee2.crt
C = US, ST = CA, L = Palo Alto, O = VMware, CN = nsxmanager.pks.vmware.local
error 18 at 0 depth lookup: self signed certificate
error ee2.crt: verification failed

So as I wrote before, unfortunately the path building picks up the first matching cert from ee.crt,
which is the one in ee1.crt (i.e., your old one), and does not try the second one (i.e., your new one).
This happens also with the latest OpenSSL pre-3.0.0 master.


A solution is to add both the subjectKeyIdentifier and authorityKeyIdentifier extensions to your certs,
for instance like this:

echo >ee.cnf "
prompt = no
distinguished_name = my_server
x509_extensions = my_exts
[my_server]
commonName = test
[my_exts]
basicConstraints = CA:false
subjectKeyIdentifier=hash
authorityKeyIdentifier = keyid"

openssl req -config ee.cnf -new -x509 -out ee1.crt -nodes -keyout ee1.pem
openssl req -config ee.cnf -new -x509 -out ee2.crt -nodes -keyout ee2.pem
cat ee1.crt ee2.crt >ee.crt

The subjectKeyIdentifier and authorityKeyIdentifier extensions are generally recommend
(and actually required to add for certs that are RFC 5280 compliant)
because they help for correct chain building, and indeed also in this case they do:

openssl verify -x509_strict -trusted ee.crt ee1.crt
ee1.crt: OK
openssl verify -x509_strict -trusted ee.crt ee2.crt
ee2.crt: OK

Regards,

    David


Reply | Threaded
Open this post in threaded view
|

RE: Directly trusted self-issued end-entity certs - Re: How to rotate cert when only first matching cert been verified

Michael Wojcik
> From: openssl-users <[hidden email]> On Behalf Of ???
> Sent: Friday, 1 January, 2021 00:08

> How to pick up cert from trust store(or cert container as you say)
> is decided by different implementation themselves, do I understand correctly?

Yes, in some cases under partial or complete control by the application. Some APIs, including OpenSSL, give the application a lot of control over the building of the chain; others don't.

And almost everyone does it incorrectly. See for example:

https://duo.com/labs/research/chain-of-fools

https://nakedsecurity.sophos.com/2020/06/02/the-mystery-of-the-expiring-sectigo-web-certificate/

https://crypto.stanford.edu/~dabo/pubs/abstracts/ssl-client-bugs.html

https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/CPT/CPT_Tool_Test-Report_Findings.pdf

(There was another article published not that long ago that surveyed a number of TLS implementations and how they built chains, pointing out how they failed to follow various requirements of PKIX, and what kinds of errors and failures they were prone to. It's similar to the CPT paper linked above, but included comparisons of different OpenSSL versions. I can't seem to find it at the moment.)

The path-validation algorithm in RFC 5280 and the path-building algorithm from RFC 4158 are agonizingly complex. Note, for example, that the description of the path-building algorithm in 4158 is 20 pages, without including the preliminary material or the longer section on optimizations.

TLS simplifies the general problem of X.509 chain construction by limiting what entities are supposed to send (X.509 lets you send any random collection of certificates, or for that matter any other data, in addition to the entity certificate; TLS says "send just a single chain from entity to root or to a certificate signed by the root"). But it's still awful, particularly when things like expiration and cross-signing come into play, and no version of OpenSSL (or any other popular library, as far as I remember) gets it entirely right for all cases.

In practice, if you use a supported OpenSSL release at the latest fix level (that means 1.1.1i at the moment), and you follow good advice about how to use it, and your use case isn't too complex, you provably achieve reasonable security under a typical application threat model. You'll want to make it relatively straightforward to update your trust-anchor collection. If you have to support an environment where things like cross-signing and multiple validation paths become important, that makes things harder. If you have stringent security requirements, that makes things harder. On the other hand, there are so many applications which fail to do even minimal certificate validation, so you can take comfort in knowing you're better than them, anyway.

--
Michael Wojcik

Reply | Threaded
Open this post in threaded view
|

Re: Directly trusted self-issued end-entity certs - Re: How to rotate cert when only first matching cert been verified

David von Oheimb-2
In reply to this post by 定平袁
On 01.01.21 08:07, 定平袁 wrote:
[hidden email]
Thank you so much for your deep investigation!
My pleasure!

With subjectKeyIdentifier and authorityKeyIdentifier extensions, it works like a charm!
Good to hear.
I've meanwhile submitted a pull request that fixed the behavior also  in case no SKID and AKID are included in the certs
and briefly mentioned your use case there: https://github.com/openssl/openssl/pull/13748

So, the former statements I found on this page only applies to CA cert, not EE cert.
How to pick up cert from trust store(or cert container as you say)
is decided by different implementation themselves, do I understand correctly?
It looks like my explanations were a bit mistakable.
Although self-signed (and more generally, self-issued) EE certs are out of scope of RFC 5280, OpenSSL still tries to build a cert chain for them and then to verify it.
Please also note that I did not write "cert container", but that these certs are essentially just a convenient container for a public key.
In other words, they have the format of an X.509 certificate, but the only thing that really matters in such a cert is the public key.
Yet since they look like a certificate, they can be used where a certificate is expected, e.g., in TLS handshake and in trust stores.

Since GnuTls and golang could pick up the right cert in this kind of scenario,
they must implement their own logic to pick up the right cert, do you think OpenSSL
will implement this logic too? Or it's a more appropriate approach to just
use the extensions you suggested?
With the fix mentioned above, chain building and verification will always succeed,
regardless how the cert looks like because in this case it is sufficient to find the target certificate in the trust store,
without having to check and further data that may be included in it.
Although not required by RFC 5280 for such a cert, OpenSSL does check for its expiration
(and may check policy restrictions etc.) because this is helpful in most application scenarios.

Regards,

    David


David von Oheimb <[hidden email]> 于2020年12月26日周六 下午5:17写道:
On 25.12.20 00:35, 定平袁 wrote:
[hidden email] I will update to a new version and try again.

Good. Ideally try also a current 3.0.0 alpha release because there have been some changes to cert chain building and verification recently.

To append cert is to make sure new cert and old cert both exist in trust store, thus when server switches cert, it can be trusted by client.
Understood, but my point was on a different aspect:
The chain building will take the first matching cert, so if you want to prefer the new cert, it must be in the list *before* the old one -
in other words, prepend the new cert to the list rather than appending to it.

@Jochen actually, the certs have different SN, which indeed is not consistent with the man doc

Different certs with the same issuer indeed must have different SNs (except in the special case I mention below).
See also RFC 5280 section 4.1.2.2 https://tools.ietf.org/html/rfc5280#section-4.1.2.2:

  It MUST be unique for each certificate issued by a given CA
     (i.e., the issuer name and serial number identify a unique certificate). 


Yet there is a different inconsistency in what you write:

The thing that confuses me is that CURL (compiled with gnutls) and Golang works.
below is my ca.crt file, I am not sure where it went wrong, maybe just my wrong behavior?
You refer to them as CA certs, but they are not: they do no have a basicConstraints field with the cA bit set.
And as far as I understand your scenario, they are not used to issue other certs but by some (TLS) server,
so they really are end-entity (EE) certs, not CA certs, and it looks like this is correct in your application scenario.

Directly trusted self-issued EE certs (which may be self-signed or not) are a special situation.
This has been clarified in RFC 6818 (which updates RFC 5280) https://tools.ietf.org/html/rfc6818#section-2:
| Consistent with Section 3.4.61 of X.509 (11/2008) [X.509], we note
| that use of self-issued certificates and self-signed certificates
| issued by entities other than CAs are outside the scope of this
| specification.  Thus, for example, a web server or client might
| generate a self-signed certificate to identify itself.  These
| certificates and how a relying party uses them to authenticate
| asserted identities are both outside the scope of RFC 5280.
So the path building and verification, as well as other checks defined RFC 5280, does not apply to them at all!
They are essentially just a convenient container for a public key, where it is optional to check expiration etc.


Unfortunately, when using such certs for TLS connections etc., still verification is done on them, which may fail.
After renaming your ca.crt file to ee.crt for clarity and extracting the first cert in ee1.crt and the second one in ee2.crt,
when verifying these directly trusted certs one gets the problem you reported:

openssl verify -x509_strict -trusted ee.crt ee1.crt
ee1.crt: OK

openssl verify -x509_strict -trusted ee.crt ee2.crt
C = US, ST = CA, L = Palo Alto, O = VMware, CN = nsxmanager.pks.vmware.local
error 18 at 0 depth lookup: self signed certificate
error ee2.crt: verification failed

So as I wrote before, unfortunately the path building picks up the first matching cert from ee.crt,
which is the one in ee1.crt (i.e., your old one), and does not try the second one (i.e., your new one).
This happens also with the latest OpenSSL pre-3.0.0 master.


A solution is to add both the subjectKeyIdentifier and authorityKeyIdentifier extensions to your certs,
for instance like this:

echo >ee.cnf "
prompt = no
distinguished_name = my_server
x509_extensions = my_exts
[my_server]
commonName = test
[my_exts]
basicConstraints = CA:false
subjectKeyIdentifier=hash
authorityKeyIdentifier = keyid"

openssl req -config ee.cnf -new -x509 -out ee1.crt -nodes -keyout ee1.pem
openssl req -config ee.cnf -new -x509 -out ee2.crt -nodes -keyout ee2.pem
cat ee1.crt ee2.crt >ee.crt

The subjectKeyIdentifier and authorityKeyIdentifier extensions are generally recommend
(and actually required to add for certs that are RFC 5280 compliant)
because they help for correct chain building, and indeed also in this case they do:

openssl verify -x509_strict -trusted ee.crt ee1.crt
ee1.crt: OK
openssl verify -x509_strict -trusted ee.crt ee2.crt
ee2.crt: OK

Regards,

    David


Reply | Threaded
Open this post in threaded view
|

Re: Directly trusted self-issued end-entity certs - Re: How to rotate cert when only first matching cert been verified

定平袁
Thanks for your reply! It makes sense more that you said, the RFC pages are indeed hard to read...

thanks for the clarification! Sorry for my misunderstanding, also thanks for your fix, I will follow up on that too.

BTW,  OpenSSL is such a friendly community.

Regards,
Dingping

David von Oheimb <[hidden email]> 于2021年1月2日周六 上午5:17写道:
On 01.01.21 08:07, 定平袁 wrote:
[hidden email]
Thank you so much for your deep investigation!
My pleasure!

With subjectKeyIdentifier and authorityKeyIdentifier extensions, it works like a charm!
Good to hear.
I've meanwhile submitted a pull request that fixed the behavior also  in case no SKID and AKID are included in the certs
and briefly mentioned your use case there: https://github.com/openssl/openssl/pull/13748

So, the former statements I found on this page only applies to CA cert, not EE cert.
How to pick up cert from trust store(or cert container as you say)
is decided by different implementation themselves, do I understand correctly?
It looks like my explanations were a bit mistakable.
Although self-signed (and more generally, self-issued) EE certs are out of scope of RFC 5280, OpenSSL still tries to build a cert chain for them and then to verify it.
Please also note that I did not write "cert container", but that these certs are essentially just a convenient container for a public key.
In other words, they have the format of an X.509 certificate, but the only thing that really matters in such a cert is the public key.
Yet since they look like a certificate, they can be used where a certificate is expected, e.g., in TLS handshake and in trust stores.

Since GnuTls and golang could pick up the right cert in this kind of scenario,
they must implement their own logic to pick up the right cert, do you think OpenSSL
will implement this logic too? Or it's a more appropriate approach to just
use the extensions you suggested?
With the fix mentioned above, chain building and verification will always succeed,
regardless how the cert looks like because in this case it is sufficient to find the target certificate in the trust store,
without having to check and further data that may be included in it.
Although not required by RFC 5280 for such a cert, OpenSSL does check for its expiration
(and may check policy restrictions etc.) because this is helpful in most application scenarios.

Regards,

    David


David von Oheimb <[hidden email]> 于2020年12月26日周六 下午5:17写道:
On 25.12.20 00:35, 定平袁 wrote:
[hidden email] I will update to a new version and try again.

Good. Ideally try also a current 3.0.0 alpha release because there have been some changes to cert chain building and verification recently.

To append cert is to make sure new cert and old cert both exist in trust store, thus when server switches cert, it can be trusted by client.
Understood, but my point was on a different aspect:
The chain building will take the first matching cert, so if you want to prefer the new cert, it must be in the list *before* the old one -
in other words, prepend the new cert to the list rather than appending to it.

@Jochen actually, the certs have different SN, which indeed is not consistent with the man doc

Different certs with the same issuer indeed must have different SNs (except in the special case I mention below).
See also RFC 5280 section 4.1.2.2 https://tools.ietf.org/html/rfc5280#section-4.1.2.2:

  It MUST be unique for each certificate issued by a given CA
     (i.e., the issuer name and serial number identify a unique certificate). 


Yet there is a different inconsistency in what you write:

The thing that confuses me is that CURL (compiled with gnutls) and Golang works.
below is my ca.crt file, I am not sure where it went wrong, maybe just my wrong behavior?
You refer to them as CA certs, but they are not: they do no have a basicConstraints field with the cA bit set.
And as far as I understand your scenario, they are not used to issue other certs but by some (TLS) server,
so they really are end-entity (EE) certs, not CA certs, and it looks like this is correct in your application scenario.

Directly trusted self-issued EE certs (which may be self-signed or not) are a special situation.
This has been clarified in RFC 6818 (which updates RFC 5280) https://tools.ietf.org/html/rfc6818#section-2:
| Consistent with Section 3.4.61 of X.509 (11/2008) [X.509], we note
| that use of self-issued certificates and self-signed certificates
| issued by entities other than CAs are outside the scope of this
| specification.  Thus, for example, a web server or client might
| generate a self-signed certificate to identify itself.  These
| certificates and how a relying party uses them to authenticate
| asserted identities are both outside the scope of RFC 5280.
So the path building and verification, as well as other checks defined RFC 5280, does not apply to them at all!
They are essentially just a convenient container for a public key, where it is optional to check expiration etc.


Unfortunately, when using such certs for TLS connections etc., still verification is done on them, which may fail.
After renaming your ca.crt file to ee.crt for clarity and extracting the first cert in ee1.crt and the second one in ee2.crt,
when verifying these directly trusted certs one gets the problem you reported:

openssl verify -x509_strict -trusted ee.crt ee1.crt
ee1.crt: OK

openssl verify -x509_strict -trusted ee.crt ee2.crt
C = US, ST = CA, L = Palo Alto, O = VMware, CN = nsxmanager.pks.vmware.local
error 18 at 0 depth lookup: self signed certificate
error ee2.crt: verification failed

So as I wrote before, unfortunately the path building picks up the first matching cert from ee.crt,
which is the one in ee1.crt (i.e., your old one), and does not try the second one (i.e., your new one).
This happens also with the latest OpenSSL pre-3.0.0 master.


A solution is to add both the subjectKeyIdentifier and authorityKeyIdentifier extensions to your certs,
for instance like this:

echo >ee.cnf "
prompt = no
distinguished_name = my_server
x509_extensions = my_exts
[my_server]
commonName = test
[my_exts]
basicConstraints = CA:false
subjectKeyIdentifier=hash
authorityKeyIdentifier = keyid"

openssl req -config ee.cnf -new -x509 -out ee1.crt -nodes -keyout ee1.pem
openssl req -config ee.cnf -new -x509 -out ee2.crt -nodes -keyout ee2.pem
cat ee1.crt ee2.crt >ee.crt

The subjectKeyIdentifier and authorityKeyIdentifier extensions are generally recommend
(and actually required to add for certs that are RFC 5280 compliant)
because they help for correct chain building, and indeed also in this case they do:

openssl verify -x509_strict -trusted ee.crt ee1.crt
ee1.crt: OK
openssl verify -x509_strict -trusted ee.crt ee2.crt
ee2.crt: OK

Regards,

    David