sendmail, openssl 1.1.1, tls1.3

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

sendmail, openssl 1.1.1, tls1.3

Carl Byington
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

I have a build of sendmail with openssl 1.1.1. It can deliver to
localhost via tls1.3, but nowhere else.

STARTTLS=client, error: connect failed=-1, reason=internal error,
SSL_error=1, errno=0, retry=-1

STARTTLS=client: error:14228044:SSL routines:construct_ca_names:internal
error:ssl/statem/statem_lib.c
:2289:

It works correctly if I disable tls1.3 via:

O ServerSSLOptions=+SSL_OP_NO_SSLv2 +SSL_OP_NO_SSLv3 +SSL_OP_NO_TLSv1_3
+SSL_OP_CIPHER_SERVER_PREFERENCE
O ClientSSLOptions=+SSL_OP_NO_SSLv2 +SSL_OP_NO_SSLv3 +SSL_OP_NO_TLSv1_3

Is this another symptom of
https://github.com/openssl/openssl/issues/7384, or is there something
else going on here?


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)

iEYEAREKAAYFAlvE0V8ACgkQL6j7milTFsGFgACfRH9BudLTi8hPCN12nv18TW4S
MTcAmwRNdzY/tMwskbmJx1bm81cNndDN
=HnJ/
-----END PGP SIGNATURE-----


--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: sendmail, openssl 1.1.1, tls1.3

Viktor Dukhovni
On Mon, Oct 15, 2018 at 10:42:26AM -0700, Carl Byington wrote:

> I have a build of sendmail with openssl 1.1.1. It can deliver to
> localhost via tls1.3, but nowhere else.
>
> STARTTLS=client, error: connect failed=-1, reason=internal error,
> SSL_error=1, errno=0, retry=-1
>
> STARTTLS=client: error:14228044:SSL routines:construct_ca_names:internal error:ssl/statem/statem_lib.c:2289:

This fails when the list of acceptable CAs sent to the peer contains
a NULL name or the ASN.1 encoding routines fail, or memory is not
available to encode the name into the output packet.

    if (ca_sk != NULL) {
        int i;

        for (i = 0; i < sk_X509_NAME_num(ca_sk); i++) {
            unsigned char *namebytes;
            X509_NAME *name = sk_X509_NAME_value(ca_sk, i);
            int namelen;

            if (name == NULL
                    || (namelen = i2d_X509_NAME(name, NULL)) < 0
                    || !WPACKET_sub_allocate_bytes_u16(pkt, namelen,
                                                       &namebytes)
                    || i2d_X509_NAME(name, &namebytes) != namelen) {
                SSLfatal(s, SSL_AD_INTERNAL_ERROR, SSL_F_CONSTRUCT_CA_NAMES,
                         ERR_R_INTERNAL_ERROR);
                return 0;
            }
        }
    }

> It works correctly if I disable tls1.3 via:
>
> O ServerSSLOptions=+SSL_OP_NO_SSLv2 +SSL_OP_NO_SSLv3 +SSL_OP_NO_TLSv1_3 +SSL_OP_CIPHER_SERVER_PREFERENCE
> O ClientSSLOptions=+SSL_OP_NO_SSLv2 +SSL_OP_NO_SSLv3 +SSL_OP_NO_TLSv1_3

Unlike TLS 1.2, in TLS 1.3 the client MAY send a list of acceptable
certificate authorities to the server as an extension in its client
HELLO: https://tools.ietf.org/html/rfc8446#section-4.2.4

Perhaps Sendmail is setting the CA names the client side, and then
OpenSSL is trying to serialize the names of all your CAs to the
server.  This is a bad idea.  Don't do that.  Try using CApath, and
no or an explicitly empty CAfile, and see if that helps.

> Is this another symptom of
> https://github.com/openssl/openssl/issues/7384, or is there something
> else going on here?

No something else.  A pointer to source code of the Sendmail in
question may be helpful.  Do you see any calls to SSL_CTX_set0_CA_list()?

I guess Sendmail being both a client and server in one, may be using
a single SSL context for both purposes, and may therefore end up
configuring the CA list for both.

If you use an empty CA file, Sendmail will probably not leak all
your trusted CA names with every connection, and be more reliable
as well.

I suspect some sort of Sendmail-specific misuse of the API, but
it is also possible that use of CA names on the client side is
not sufficiently well tested on the OpenSSL side.

--
        Viktor.
--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: sendmail, openssl 1.1.1, tls1.3

Carl Byington
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512


> Perhaps Sendmail is setting the CA names the client side, and then
> OpenSSL is trying to serialize the names of all your CAs to the
> server.  This is a bad idea.  Don't do that.  Try using CApath, and
> no or an explicitly empty CAfile, and see if that helps.

Do you mean CACertFile and CACertPath? Redhat/Centos stock
sendmail.mc/cf uses:

O CACertFile=/etc/pki/tls/certs/ca-bundle.crt
O CACertPath=/etc/pki/tls/certs

pointing the CACertFile to 750KB file with 149 certificates. That just
seems wrong, but perhaps there is some reason for it. If CACertFile is
not specified, sendmail won't advertise STARTTLS. So we need to give it
something there, and the docs imply that it should at least contain the
certificate of the CA that signed the sendmail certificate. I have a
private CA that signed my sendmail certificate, so using:

O CACertFile=/etc/pki/tls/certs/my-ca-certificate.pem
O CACertPath=/etc/pki/tls/certs
O ServerSSLOptions=+SSL_OP_NO_SSLv2 +SSL_OP_NO_SSLv3
+SSL_OP_CIPHER_SERVER_PREFERENCE
O ClientSSLOptions=+SSL_OP_NO_SSLv2 +SSL_OP_NO_SSLv3

on both the client and server sendmail machines, we get TLSv1.3 !

Perhaps some certificate in the stock ca-bundle.crt is malformed?


> No something else.  A pointer to source code of the Sendmail in
> question may be helpful.

http://www.five-ten-sg.com/mapper/blog/DANE

ftp://ftp.sendmail.org/pub/sendmail/snapshots/sendmail.8.16.0.29.tar.gz

http://www.five-ten-sg.com/util/sendmail-8.16.0-dane.patch




> Do you see any calls to SSL_CTX_set0_CA_list()?

No, but there is a call to  SSL_CTX_set_client_CA_list(*ctx,
SSL_load_client_CA_file(cacertfile)) which would read that ca-bundle.crt
file.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)

iEYEAREKAAYFAlvFJ38ACgkQL6j7milTFsH3wwCeNh0ZAOIRq4kG/Nh5gCeZaAvK
MPUAn0a7NaSk5edTMGcLa0SHpskOxTYW
=Yi1x
-----END PGP SIGNATURE-----


--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: sendmail, openssl 1.1.1, tls1.3

Claus Assmann
On Mon, Oct 15, 2018, Carl Byington wrote:

> O CACertFile=/etc/pki/tls/certs/ca-bundle.crt

> pointing the CACertFile to 750KB file with 149 certificates. That just
> seems wrong, but perhaps there is some reason for it. If CACertFile is

sendmail: op.*:
         However, do not list too many root CAs in that
         file, otherwise the TLS handshake may fail; e.g.,

Please tell whoever is responsible for that default to fix it.
The certs should be in CACertPath if at all.
--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: sendmail, openssl 1.1.1, tls1.3

Viktor Dukhovni
In reply to this post by Carl Byington


> On Oct 15, 2018, at 7:49 PM, Carl Byington <[hidden email]> wrote:
>
>> Perhaps Sendmail is setting the CA names the client side, and then
>> OpenSSL is trying to serialize the names of all your CAs to the
>> server.  This is a bad idea.  Don't do that.  Try using CApath, and
>> no or an explicitly empty CAfile, and see if that helps.
>
> Do you mean CACertFile and CACertPath? Redhat/Centos stock
> sendmail.mc/cf uses:
>
> O CACertFile=/etc/pki/tls/certs/ca-bundle.crt
> O CACertPath=/etc/pki/tls/certs

Yes, these.  It is common practice with TLS servers to use the CAfile
(in Sendmail aka CACertFile) as the source of CA names to send to clients
in certificate requests (when those are enabled).  Most MTAs don't request
client certificates, so this "overloading" of the parameter is mostly
harmless.  Still, in Postfix the documentation recommends keeping the
CAfile short (or even not specified at all), and having any CA cerficates
used for validating client certs be specified via CApath.  CApath is used
for indexed lookup and not enumeration.

With TLS 1.3, you suddenly have clients optionally soliciting certificates
by specific CA from servers, and if you use the same SSL_CTX for both the
client and server roles, the same CA list will be sent to remote servers
that you'd use to solicit certificates from clients.

My advice is to make sure that the CAfile is mercifully short or not
set at all (assuming that's possible in Sendmail without disabling TLS).

> pointing the CACertFile to 750KB file with 149 certificates.

With 149 certs, and typical CA names O(80) bytes, we're looking at
~12KB of cert names, which should fit into an extension that can be
up to 64KB in size.  So overflowing the extension size limit would
not be my first guess.  If you make the CA bundle available (send
it to me off-list?) I can take a closer look.

> I have a
> private CA that signed my sendmail certificate, so using:
>
> O CACertFile=/etc/pki/tls/certs/my-ca-certificate.pem
> O CACertPath=/etc/pki/tls/certs
> O ServerSSLOptions=+SSL_OP_NO_SSLv2 +SSL_OP_NO_SSLv3
> +SSL_OP_CIPHER_SERVER_PREFERENCE
> O ClientSSLOptions=+SSL_OP_NO_SSLv2 +SSL_OP_NO_SSLv3
>
> on both the client and server sendmail machines, we get TLSv1.3 !

So the there's something toxic about the CA list as a whole.
Either indeed some problem certiificate, or a problem encoding
the lot of them?

>> Do you see any calls to SSL_CTX_set0_CA_list()?
>
> No, but there is a call to  SSL_CTX_set_client_CA_list(*ctx,
> SSL_load_client_CA_file(cacertfile)) which would read that ca-bundle.crt
> file.

Right, but I think that the underlying list is the same for client->server
as for server->client.  So this has the unfortunate effect of making
the client fill its HELLO message with an enormous list of CA names.

A separate SSL_CTX for the client would make Sendmail less prone to
such mishaps.  Try Postfix some time, we don't have that issue... :-)

--
        Viktor.

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: sendmail, openssl 1.1.1, tls1.3

Carl Byington
In reply to this post by Claus Assmann
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Mon, 2018-10-15 at 16:57 -0700, Claus Assmann wrote:
> Please tell whoever is responsible for that default to fix it.

I will do that.

> The certs should be in CACertPath if at all.

Nothing to do with openssl, but for sendmail, suppose we have

O CACertFile=/etc/pki/tls/certs/one-ca-certificate.pem
O CACertPath=/etc/pki/tls/certs
O ServerCertFile=/etc/pki/tls/certs/sendmail.pem

where one-ca-certificate.pem is the certificate of the CA that signed
the sendmail.pem certificate, and /etc/pki/tls/certs/ca-bundle.crt
contains many CA certificates that we want to use for certificate
validation.

https://www.sendmail.org/~ca/email/starttls.html

I presume that means we need to split this ca-bundle.crt into 150
separate files, and compute hashes for each, with another 150 symbolic
links. Is that true, or am I missing some shortcut?



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)

iEYEAREKAAYFAlvFWT8ACgkQL6j7milTFsHnswCdElJTGjCGao0n4xWqWB2nb2Bn
HyUAnj17PT/b/x26P4WGGD4TTq6Mjvuc
=O8T0
-----END PGP SIGNATURE-----


--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: sendmail, openssl 1.1.1, tls1.3

Viktor Dukhovni
In reply to this post by Viktor Dukhovni


> On Oct 15, 2018, at 9:22 PM, Viktor Dukhovni <[hidden email]> wrote:
>
>> pointing the CACertFile to 750KB file with 149 certificates.
>
> With 149 certs, and typical CA names O(80) bytes, we're looking at
> ~12KB of cert names, which should fit into an extension that can be
> up to 64KB in size.  So overflowing the extension size limit would
> not be my first guess.  If you make the CA bundle available (send
> it to me off-list?) I can take a closer look.

[ Carl sent me the CA bundle off-list ]  With the provided CA bundle
I was able to easily reproduce the same symptoms with:

 $ openssl s_client -requestCAfile bundle.pem -connect localhost:12345

Running this under a debugger the failure happens at certificate #143
because the client hello packet overflows its maximum allocation:

$6 = {
  buf = 0x0000000100724200
  staticbuf = 0x0000000000000000 <no value available>
  curr = 16364
  written = 16364
  maxsize = 16384
  subs = 0x0000000100727700
}

So even though the extension is allowed to be up to 2^16 bytes, it
seems the client HELLO is expected to fit into a single TLS record
of at most 16K bytes.

Given enough crud in the CA bundle, and a client misconfigured to
to spill its guts to the server, by sending the names of all the
trusted CAs, the limit is not too hard to exceed.

The work-around is to send *zero* CA names to the server, Sendmail
SHOULD NOT configure the SSL_CTX for the client connection to with
preferred CA names.  If that is not configurable, then keep the
CA file as short as possible, empty if possible, else just one
root CA, and everything else in CApath (yes "hashed" with the
various symlinks in place to each cert, one per file).

As for the 16K limit, and whether we should be sending client
CA names without further indication from the (TLS 1.3) client
to do so, I'm hoping Matt Caswell and or other team members
will chime in.

--
        Viktor.

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: sendmail, openssl 1.1.1, tls1.3

OpenSSL - User mailing list
On 16/10/2018 06:19, Viktor Dukhovni wrote:

>> On Oct 15, 2018, at 9:22 PM, Viktor Dukhovni <[hidden email]> wrote:
>>
>>> pointing the CACertFile to 750KB file with 149 certificates.
>> With 149 certs, and typical CA names O(80) bytes, we're looking at
>> ~12KB of cert names, which should fit into an extension that can be
>> up to 64KB in size.  So overflowing the extension size limit would
>> not be my first guess.  If you make the CA bundle available (send
>> it to me off-list?) I can take a closer look.
> [ Carl sent me the CA bundle off-list ]  With the provided CA bundle
> I was able to easily reproduce the same symptoms with:
>
>   $ openssl s_client -requestCAfile bundle.pem -connect localhost:12345
>
> Running this under a debugger the failure happens at certificate #143
> because the client hello packet overflows its maximum allocation:
>
> $6 = {
>    buf = 0x0000000100724200
>    staticbuf = 0x0000000000000000 <no value available>
>    curr = 16364
>    written = 16364
>    maxsize = 16384
>    subs = 0x0000000100727700
> }
>
> So even though the extension is allowed to be up to 2^16 bytes, it
> seems the client HELLO is expected to fit into a single TLS record
> of at most 16K bytes.
>
> Given enough crud in the CA bundle, and a client misconfigured to
> to spill its guts to the server, by sending the names of all the
> trusted CAs, the limit is not too hard to exceed.
>
> The work-around is to send *zero* CA names to the server, Sendmail
> SHOULD NOT configure the SSL_CTX for the client connection to with
> preferred CA names.  If that is not configurable, then keep the
> CA file as short as possible, empty if possible, else just one
> root CA, and everything else in CApath (yes "hashed" with the
> various symlinks in place to each cert, one per file).
>
> As for the 16K limit, and whether we should be sending client
> CA names without further indication from the (TLS 1.3) client
> to do so, I'm hoping Matt Caswell and or other team members
> will chime in.
>
Just for clarity, how is an OpenSSL 1.1.1 client supposed to tell
the local validation code which CAs to trust, especially if loading
the list before entering a chroot jail?

How is this different from the OpenSSL 1.0.x API?

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: sendmail, openssl 1.1.1, tls1.3

Viktor Dukhovni
On Tue, Oct 16, 2018 at 08:13:11AM +0200, Jakob Bohm via openssl-users wrote:

> > As for the 16K limit, and whether we should be sending client
> > CA names without further indication from the (TLS 1.3) client
> > to do so, I'm hoping Matt Caswell and or other team members
> > will chime in.
>
> Just for clarity, how is an OpenSSL 1.1.1 client supposed to tell
> the local validation code which CAs to trust, especially if loading
> the list before entering a chroot jail?

Loading CA files is not a problem in itself, the issue is that some
(typically server) applications overload the CAfile as also being
the source of CA hints to clients in certificate requests.  This
creates bloated server certificate request messages.  With TLS 1.3,
this is further compounded in applications that are *both* a server
and client, and use the *same* context for both purposes.  Once
that happens, the CA hints are used in both directions.

> How is this different from the OpenSSL 1.0.x API?

The API is identical, what's different is that TLS 1.3 makes the
CA hints bidirectional, given such hints have never been useful
before, it is IMHO just needless generality that's counter-productive.
Perhaps OpenSSL should require an additional non-default configuration
to enable transmission of the client's CA DNs to the server.

Or perhaps separate the server->client and client->server CA name
lists in the SSL context, so that setting one does not (surprise!)
also set the other.  Overloading the same setting for both directions
may not have anticipated bidirectional use of contexts.

Someone should perhaps open an issue to track whether anything needs
to change here beyond advice to users, and if so what.

--
        Viktor.
--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: sendmail, openssl 1.1.1, tls1.3

Tomas Mraz-2
On 10/16/2018 09:27 AM, Viktor Dukhovni wrote:

> On Tue, Oct 16, 2018 at 08:13:11AM +0200, Jakob Bohm via openssl-users wrote:
>
>>> As for the 16K limit, and whether we should be sending client
>>> CA names without further indication from the (TLS 1.3) client
>>> to do so, I'm hoping Matt Caswell and or other team members
>>> will chime in.
>>
>> Just for clarity, how is an OpenSSL 1.1.1 client supposed to tell
>> the local validation code which CAs to trust, especially if loading
>> the list before entering a chroot jail?
>
> Loading CA files is not a problem in itself, the issue is that some
> (typically server) applications overload the CAfile as also being
> the source of CA hints to clients in certificate requests.  This
> creates bloated server certificate request messages.  With TLS 1.3,
> this is further compounded in applications that are *both* a server
> and client, and use the *same* context for both purposes.  Once
> that happens, the CA hints are used in both directions.

What are the CA hints sent from client to server good for? This looks
like missing API in 1.1.1 as we clearly do not want to start sending
huge client helos just because of client sending the CA hints to servers
in TLS-1.3. This needs to be something that requires an explicit new API
call to set.

>
>> How is this different from the OpenSSL 1.0.x API?
>
> The API is identical, what's different is that TLS 1.3 makes the
> CA hints bidirectional, given such hints have never been useful
> before, it is IMHO just needless generality that's counter-productive.
> Perhaps OpenSSL should require an additional non-default configuration
> to enable transmission of the client's CA DNs to the server.
>
> Or perhaps separate the server->client and client->server CA name
> lists in the SSL context, so that setting one does not (surprise!)
> also set the other.  Overloading the same setting for both directions
> may not have anticipated bidirectional use of contexts.
>
> Someone should perhaps open an issue to track whether anything needs
> to change here beyond advice to users, and if so what.

I've opened it:

https://github.com/openssl/openssl/issues/7411

Tomas Mraz
--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: sendmail, openssl 1.1.1, tls1.3

Matt Caswell-2
In reply to this post by Viktor Dukhovni


On 16/10/2018 05:19, Viktor Dukhovni wrote:
> [ Carl sent me the CA bundle off-list ]  With the provided CA bundle
> I was able to easily reproduce the same symptoms with:

Please can someone send me the same CA bundle so that I might also
reproduce this?

Thanks

Matt


>
>  $ openssl s_client -requestCAfile bundle.pem -connect localhost:12345
>
> Running this under a debugger the failure happens at certificate #143
> because the client hello packet overflows its maximum allocation:
>
> $6 = {
>   buf = 0x0000000100724200
>   staticbuf = 0x0000000000000000 <no value available>
>   curr = 16364
>   written = 16364
>   maxsize = 16384
>   subs = 0x0000000100727700
> }
>
> So even though the extension is allowed to be up to 2^16 bytes, it
> seems the client HELLO is expected to fit into a single TLS record
> of at most 16K bytes.
>
> Given enough crud in the CA bundle, and a client misconfigured to
> to spill its guts to the server, by sending the names of all the
> trusted CAs, the limit is not too hard to exceed.
>
> The work-around is to send *zero* CA names to the server, Sendmail
> SHOULD NOT configure the SSL_CTX for the client connection to with
> preferred CA names.  If that is not configurable, then keep the
> CA file as short as possible, empty if possible, else just one
> root CA, and everything else in CApath (yes "hashed" with the
> various symlinks in place to each cert, one per file).
>
> As for the 16K limit, and whether we should be sending client
> CA names without further indication from the (TLS 1.3) client
> to do so, I'm hoping Matt Caswell and or other team members
> will chime in.
>
--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users