[RFC] enc utility & under-documented behavior changes: improving backward compatibility

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[RFC] enc utility & under-documented behavior changes: improving backward compatibility

Robin H. Johnson
Commit f8547f62c212837dbf44fb7e2755e5774a59a57b (documented in
9e8b6f042749ded556380227c9f2db7ffad9a3aa), changed the default digest
for the 'enc' utility from MD5 to SHA256.

While I do strongly encourage getting away from MD5, this has the
unfortunate side effect of silently breaking existing data.

An old encrypted file would be generated with MD5 as the digest used on
the passphrase. Then if you tried to decrypt it on a new version of
OpenSSL, that defaults to SHA256, you'd get just garbage output.

This recently bit me when trying to decrypt Amanda backups:
https://github.com/zmanda/amanda/blob/8136c076757d2e54fa7d4df15f002187823d8787/common-src/amcrypt-ossl.sh#L81
https://github.com/zmanda/amanda/blob/8136c076757d2e54fa7d4df15f002187823d8787/common-src/amcrypt-ossl-asym.sh#L134
(yes, I'm aware that the asym variant also passes in -nosalt).

This happens because there is no metadata that conveys what parameters
were used during encryption. Unless you happen to know exactly which
version of OpenSSL was used, and what parameters, you risk getting
garbage data back.

This can also happen already when the ciphers are mismatched between
encryption & decryption.

The -salt option already sets a precedent for adding a header with the
salt data, and I'd like to extend that, to improve backwards
compatibility.

1. Encrypted data should include a header block, that OPTIONALLY
   specifies each of:
   - cipher & parameters (e.g. salt, padding)
   - key derivation function & parameters (MD)
1.1. Some users might want to leave the fields empty, to increase
     security by obscurity.
1.2. This also opens the path to stronger key derivation (PBKDF2)
2. During decryption, if no header block is present, and no message
   digest was specified, the default digest SHOULD be MD5.

--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer
E-Mail   : [hidden email]
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

signature.asc (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] enc utility & under-documented behavior changes: improving backward compatibility

Matt Caswell-2


On 02/10/17 20:20, Robin H. Johnson wrote:

> Commit f8547f62c212837dbf44fb7e2755e5774a59a57b (documented in
> 9e8b6f042749ded556380227c9f2db7ffad9a3aa), changed the default digest
> for the 'enc' utility from MD5 to SHA256.
>
> While I do strongly encourage getting away from MD5, this has the
> unfortunate side effect of silently breaking existing data.
>
> An old encrypted file would be generated with MD5 as the digest used on
> the passphrase. Then if you tried to decrypt it on a new version of
> OpenSSL, that defaults to SHA256, you'd get just garbage output.
>
> This recently bit me when trying to decrypt Amanda backups:
> https://github.com/zmanda/amanda/blob/8136c076757d2e54fa7d4df15f002187823d8787/common-src/amcrypt-ossl.sh#L81
> https://github.com/zmanda/amanda/blob/8136c076757d2e54fa7d4df15f002187823d8787/common-src/amcrypt-ossl-asym.sh#L134
> (yes, I'm aware that the asym variant also passes in -nosalt).
>
> This happens because there is no metadata that conveys what parameters
> were used during encryption. Unless you happen to know exactly which
> version of OpenSSL was used, and what parameters, you risk getting
> garbage data back.
>
> This can also happen already when the ciphers are mismatched between
> encryption & decryption.
>
> The -salt option already sets a precedent for adding a header with the
> salt data, and I'd like to extend that, to improve backwards
> compatibility.

I am in favour of a new output format. In fact a new output format has
been discussed before and I believe there was broad support for the idea:

https://github.com/openssl/openssl/pull/2083

That PR didn't turn into anything, but I'm not sure why.

A new output format will help to stop things getting worse if we change
the format again (which I think we should in order to use PBKDF2 and
more iterations). However it won't help handle the historic issue. If
you attempt to decrypt a file that isn't in the new format, you won't
know whether it was generated by OpenSSL 1.1.0 (and is therefore using
SHA256), or whether it was generated by some earlier version (and is
therefore using MD5).

>
> 1. Encrypted data should include a header block, that OPTIONALLY
>    specifies each of:
>    - cipher & parameters (e.g. salt, padding)
>    - key derivation function & parameters (MD)
> 1.1. Some users might want to leave the fields empty, to increase
>      security by obscurity.

I think we probably should have an option to not write out the header at
all - but not to "increase security by obscurity". Rather it would be
about having the ability to write out a "raw" encrypted file that could
be consumed by some other tool.


> 1.2. This also opens the path to stronger key derivation (PBKDF2)
> 2. During decryption, if no header block is present, and no message
>    digest was specified, the default digest SHOULD be MD5.

Should it? What about compatibility with OpenSSL 1.1.0? We cannot make
breaking changes in 1.1.1, so it has to be compatible with 1.1.0.

Matt


--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] enc utility & under-documented behavior changes: improving backward compatibility

Tomas Mraz-2
On Tue, 2017-10-03 at 08:23 +0100, Matt Caswell wrote:
>
> > 1.2. This also opens the path to stronger key derivation (PBKDF2)
> > 2. During decryption, if no header block is present, and no message
> >    digest was specified, the default digest SHOULD be MD5.
>
> Should it? What about compatibility with OpenSSL 1.1.0? We cannot
> make
> breaking changes in 1.1.1, so it has to be compatible with 1.1.0.

Yeah, the ship has sailed. SHA-256 should be used by default as in
1.1.0.

--
Tomáš Mráz
Red Hat

No matter how far down the wrong road you've gone, turn back.
                                              Turkish proverb
[You'll know whether the road is wrong if you carefully listen to your
conscience.]

 * Google and NSA associates, this message is none of your business.
 * Please leave it alone, and consider whether your actions are
 * authorized by the contract with Red Hat, or by the US constitution.
 * If you feel you're being encouraged to disregard the limits built
 * into them, remember Edward Snowden and Wikileaks.
--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] enc utility & under-documented behavior changes: improving backward compatibility

Robin H. Johnson
On Tue, Oct 03, 2017 at 09:45:43AM +0200, Tomas Mraz wrote:

> On Tue, 2017-10-03 at 08:23 +0100, Matt Caswell wrote:
> >
> > > 1.2. This also opens the path to stronger key derivation (PBKDF2)
> > > 2. During decryption, if no header block is present, and no message
> > >    digest was specified, the default digest SHOULD be MD5.
> >
> > Should it? What about compatibility with OpenSSL 1.1.0? We cannot
> > make
> > breaking changes in 1.1.1, so it has to be compatible with 1.1.0.
> Yeah, the ship has sailed. SHA-256 should be used by default as in
> 1.1.0.
It's a breaking change from 1.0.

At the very least, it should be added to the big notes:
https://www.openssl.org/news/openssl-1.1.0-notes.html
(this was in fact the first place I looked when my data was broken,
there was nothing about the enc tool here).

--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer
E-Mail   : [hidden email]
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

signature.asc (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] enc utility & under-documented behavior changes: improving backward compatibility

Michel
In reply to this post by Matt Caswell-2

[...]
I think we probably should have an option to not write out the header at all
- but not to "increase security by obscurity". Rather it would be about
having the ability to write out a "raw" encrypted file that could be
consumed by some other tool.
[...]


+1

I will probably come back on this soon.

Michel

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] enc utility & under-documented behavior changes: improving backward compatibility

Matt Caswell-2
In reply to this post by Robin H. Johnson


On 03/10/17 18:51, Robin H. Johnson wrote:

> On Tue, Oct 03, 2017 at 09:45:43AM +0200, Tomas Mraz wrote:
>> On Tue, 2017-10-03 at 08:23 +0100, Matt Caswell wrote:
>>>
>>>> 1.2. This also opens the path to stronger key derivation (PBKDF2)
>>>> 2. During decryption, if no header block is present, and no message
>>>>    digest was specified, the default digest SHOULD be MD5.
>>>
>>> Should it? What about compatibility with OpenSSL 1.1.0? We cannot
>>> make
>>> breaking changes in 1.1.1, so it has to be compatible with 1.1.0.
>> Yeah, the ship has sailed. SHA-256 should be used by default as in
>> 1.1.0.
> It's a breaking change from 1.0.
As Tomas said - that ship has sailed. In my mind that change was a
mistake. It could have been done in a non-breaking way by introducing a
new header format at that time. That way if the header was not present
then we would have known to use MD5 - otherwise use the hash as
specified in the header. But its too late now. Breaking it again back to
what it was before is the wrong answer.

> At the very least, it should be added to the big notes:
> https://www.openssl.org/news/openssl-1.1.0-notes.html
> (this was in fact the first place I looked when my data was broken,
> there was nothing about the enc tool here).

Well in fact it is there:

  *) Changed default digest for the dgst and enc commands from MD5 to
     sha256
     [Rich Salz]

Perhaps that is a little brief - it doesn't really explain the
implications of the change.

Matt


--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

signature.asc (491 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] enc utility & under-documented behavior changes: improving backward compatibility

Dr. Stephen Henson
On Wed, Oct 04, 2017, Matt Caswell wrote:

>
> As Tomas said - that ship has sailed. In my mind that change was a
> mistake. It could have been done in a non-breaking way by introducing a
> new header format at that time.
>

As regards a new header format. In the case of some of the structures we use
there is an appropriate password based encryption ASN.1 AlgorithmIdentifier.
The code to encode and decode that format is already in OpenSSL and it
is documented in various standards. It is currently used for private key
encryption (via PKCS#8) and PKCS#7 EncryptedData format (used by PKCS#12).
One of several advantages of using that form is that any new forms we want to
support just need to be a new password based encryption algorithm and "enc"
handles it automatically. So we'd get scrypt, PBKDF2 and all the older PKCS#12
algorithms automatically. We wouldn't be able to use the current non-standard
password based encrytion algorithm without defining our own OID but I can't
see why we'd want to use that when much better alternatives are available.

In fact if we were feeling particularly adventurous we could output a CMS or
PKCS#7 EncryptedData ContentInfo in the enc utility.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] enc utility & under-documented behavior changes: improving backward compatibility

Robin H. Johnson
In reply to this post by Matt Caswell-2
On Wed, Oct 04, 2017 at 10:39:03AM +0100, Matt Caswell wrote:

> > At the very least, it should be added to the big notes:
> > https://www.openssl.org/news/openssl-1.1.0-notes.html
> > (this was in fact the first place I looked when my data was broken,
> > there was nothing about the enc tool here).
> Well in fact it is there:
>
>   *) Changed default digest for the dgst and enc commands from MD5 to
>      sha256
>      [Rich Salz]
>
> Perhaps that is a little brief - it doesn't really explain the
> implications of the change.
The entry you just copied is from the full ChangeLog, not the notes
page.

--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer
E-Mail   : [hidden email]
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

signature.asc (1K) Download Attachment