Decrypt error when using openssl 1.1.1b during SSL handshake

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Decrypt error when using openssl 1.1.1b during SSL handshake

Mahendra SP
Hi All.

We have upgraded openssl version to 1.1.1b

With this, we are seeing decryption error during SSL handshake for the below explained scenario. Our device acts as an SSL server.

We have external hardware to offload RSA private key operations using the engine.
Decryption of pre-master secret is done using hardware and is successful. We compared the pre-master secret on both server and client and they match.
However, we see that SSL handshake fails with "decrypt error (51)" with an alert number 21. Verifying the encrypted finish message on the server side fails.

This issue does not happen with software performing RSA private key operations.

Can someone help with the reason for decryption failure? Below is the compiler and processor details. It is 64 bit.
arm-linux-gnueabihf-gcc  -march=armv7ve -mthumb -mfpu=neon -mfloat-abi=hard

Thanks
Mahendra
Reply | Threaded
Open this post in threaded view
|

Re: Decrypt error when using openssl 1.1.1b during SSL handshake

Matt Caswell-2


On 30/10/2020 09:18, Mahendra SP wrote:

> Hi All.
>
> We have upgraded openssl version to 1.1.1b
>
> With this, we are seeing decryption error during SSL handshake for the
> below explained scenario. Our device acts as an SSL server.
>
> We have external hardware to offload RSA private key operations using
> the engine.
> Decryption of pre-master secret is done using hardware and is
> successful. We compared the pre-master secret on both server and client
> and they match.
> However, we see that SSL handshake fails with "decrypt error (51)" with
> an alert number 21. Verifying the encrypted finish message on the server
> side fails.
>
> This issue does not happen with software performing RSA private key
> operations.
>
> Can someone help with the reason for decryption failure? Below is the
> compiler and processor details. It is 64 bit.
> arm-linux-gnueabihf-gcc  -march=armv7ve -mthumb -mfpu=neon -mfloat-abi=hard

Potentially this is related to the use of PSS padding in libssl which is
mandated in TLSv1.3. The TLSv1.3 spec also requires its use even in TLSv1.2.

The PSS padding is implemented within the EVP layer. Ultimately EVP
calls the function RSA_private_encrypt() with padding set to RSA_NO_PADDING.

Assuming your engine is implemented via a custom RSA_METHOD does it
support RSA_private_encrypt(() with RSA_NO_PADDING? If not this is
likely to be the problem.

More discussion of this is here:

https://github.com/openssl/openssl/issues/7968

Also related is the recent discussion on this list about the CAPI engine
and this issue:

https://github.com/openssl/openssl/issues/8872

Matt
Reply | Threaded
Open this post in threaded view
|

Re: Decrypt error when using openssl 1.1.1b during SSL handshake

Mahendra SP
Hi Matt,

Thank you for the inputs.
Yes, we had encountered the padding issue initially. But we added support for RSA_NO_PADDING in our hardware. That's why we are able to successfully decrypt the premaster secret in the hardware.
Hence the issue does not seem to be related to padding. We have confirmed this by comparing the premaster secret on both client and server and they are the same.

We suspect in this case, verification of "encrypted handshake message" failure is happening. 
We understand constant_time_xx APIs get used for CBC padding validation. Will this have  any dependency on the compiler optimization or asm flags? Will this issue be seen if hardware takes more time for the operation?

Here is the snippet of the wireshark where our device acting as server closes the connection with decryption failure.
If you need any further info, please let us know. 
image.png
Thanks
Mahendra

Please suggest.



On Fri, Oct 30, 2020 at 3:32 PM Matt Caswell <[hidden email]> wrote:


On 30/10/2020 09:18, Mahendra SP wrote:
> Hi All.
>
> We have upgraded openssl version to 1.1.1b
>
> With this, we are seeing decryption error during SSL handshake for the
> below explained scenario. Our device acts as an SSL server.
>
> We have external hardware to offload RSA private key operations using
> the engine.
> Decryption of pre-master secret is done using hardware and is
> successful. We compared the pre-master secret on both server and client
> and they match.
> However, we see that SSL handshake fails with "decrypt error (51)" with
> an alert number 21. Verifying the encrypted finish message on the server
> side fails.
>
> This issue does not happen with software performing RSA private key
> operations.
>
> Can someone help with the reason for decryption failure? Below is the
> compiler and processor details. It is 64 bit.
> arm-linux-gnueabihf-gcc  -march=armv7ve -mthumb -mfpu=neon -mfloat-abi=hard

Potentially this is related to the use of PSS padding in libssl which is
mandated in TLSv1.3. The TLSv1.3 spec also requires its use even in TLSv1.2.

The PSS padding is implemented within the EVP layer. Ultimately EVP
calls the function RSA_private_encrypt() with padding set to RSA_NO_PADDING.

Assuming your engine is implemented via a custom RSA_METHOD does it
support RSA_private_encrypt(() with RSA_NO_PADDING? If not this is
likely to be the problem.

More discussion of this is here:

https://github.com/openssl/openssl/issues/7968

Also related is the recent discussion on this list about the CAPI engine
and this issue:

https://github.com/openssl/openssl/issues/8872

Matt
Reply | Threaded
Open this post in threaded view
|

Re: Decrypt error when using openssl 1.1.1b during SSL handshake

Matt Caswell-2


On 30/10/2020 11:22, Mahendra SP wrote:
> Hi Matt,
>
> Thank you for the inputs.
> Yes, we had encountered the padding issue initially. But we added
> support for RSA_NO_PADDING in our hardware. That's why we are able to
> successfully decrypt the premaster secret in the hardware.
> Hence the issue does not seem to be related to padding. We have
> confirmed this by comparing the premaster secret on both client and
> server and they are the same.

Ok, good.

>
> We suspect in this case, verification of "encrypted handshake message"
> failure is happening.

It's possible. It would be helpful if you can get more information from
the error stack on the server, e.g. by using ERR_print_errors_fp() or
something similar. I'm particularly interested in identifying the source
file and line number where the decrypt_error is coming from. Printing
the error stack should give us that information. There are a number of
places that a "decrypt error" can occur and it would be helpful to
identify which one is the cause of the problem.


> We understand constant_time_xx APIs get used for CBC padding validation.
> Will this have  any dependency on the compiler optimization or asm
> flags?

CBC padding validation is fairly independent of anything to do with RSA,
so I think its unlikely to be the culprit here. Of course sometimes
compiler optimization/asm issues do occur so it can't be ruled out
entirely - but it's not where I would start looking.

> Will this issue be seen if hardware takes more time for the
> operation?
>

No. Constant time here just means that we implement the code without any
branching based on secret data (e.g. no "if" statements/while loops etc
based on secret dependent data). It has very little to do with how long
something actually takes to process.


> Here is the snippet of the wireshark where our device acting as server
> closes the connection with decryption failure.

Thanks. To narrow it down further I need to figure out which line of
code the decrypt_error is coming from as described above.

Matt



> If you need any further info, please let us know. 
> image.png
> Thanks
> Mahendra
>
> Please suggest.
>
>
>
> On Fri, Oct 30, 2020 at 3:32 PM Matt Caswell <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>
>
>     On 30/10/2020 09:18, Mahendra SP wrote:
>     > Hi All.
>     >
>     > We have upgraded openssl version to 1.1.1b
>     >
>     > With this, we are seeing decryption error during SSL handshake for the
>     > below explained scenario. Our device acts as an SSL server.
>     >
>     > We have external hardware to offload RSA private key operations using
>     > the engine.
>     > Decryption of pre-master secret is done using hardware and is
>     > successful. We compared the pre-master secret on both server and
>     client
>     > and they match.
>     > However, we see that SSL handshake fails with "decrypt error (51)"
>     with
>     > an alert number 21. Verifying the encrypted finish message on the
>     server
>     > side fails.
>     >
>     > This issue does not happen with software performing RSA private key
>     > operations.
>     >
>     > Can someone help with the reason for decryption failure? Below is the
>     > compiler and processor details. It is 64 bit.
>     > arm-linux-gnueabihf-gcc  -march=armv7ve -mthumb -mfpu=neon
>     -mfloat-abi=hard
>
>     Potentially this is related to the use of PSS padding in libssl which is
>     mandated in TLSv1.3. The TLSv1.3 spec also requires its use even in
>     TLSv1.2.
>
>     The PSS padding is implemented within the EVP layer. Ultimately EVP
>     calls the function RSA_private_encrypt() with padding set to
>     RSA_NO_PADDING.
>
>     Assuming your engine is implemented via a custom RSA_METHOD does it
>     support RSA_private_encrypt(() with RSA_NO_PADDING? If not this is
>     likely to be the problem.
>
>     More discussion of this is here:
>
>     https://github.com/openssl/openssl/issues/7968
>
>     Also related is the recent discussion on this list about the CAPI engine
>     and this issue:
>
>     https://github.com/openssl/openssl/issues/8872
>
>     Matt
>
Reply | Threaded
Open this post in threaded view
|

Re: Decrypt error when using openssl 1.1.1b during SSL handshake

Mahendra SP
Hi Matt,

Error is reported from this:
FILE:../openssl-1.1.1b/ssl/statem/statem_srvr.c, FUNCTION:415, LINE:3055, reason=147, alert=51

We see that hardware is returning 48 bytes. Even if the decrypted premaster data is correct, openssl is expecting more than 48 bytes in return.
This check fails as decrypt_len is 48.
decrypt_len < 11 + SSL_MAX_MASTER_KEY_LENGTH 

We compared the data returned when software is used. Decrypt_len is 256 bytes and the last 48 bytes are actual premaster secret. Also, openssl checks for if the first byte is 0 and second byte is 2. 
We are trying to rectify this issue in hardware and return the correct data.

Please suggest if you have any comments for the above info.

Thanks
Mahendra

On Fri, Oct 30, 2020 at 7:50 PM Matt Caswell <[hidden email]> wrote:


On 30/10/2020 11:22, Mahendra SP wrote:
> Hi Matt,
>
> Thank you for the inputs.
> Yes, we had encountered the padding issue initially. But we added
> support for RSA_NO_PADDING in our hardware. That's why we are able to
> successfully decrypt the premaster secret in the hardware.
> Hence the issue does not seem to be related to padding. We have
> confirmed this by comparing the premaster secret on both client and
> server and they are the same.

Ok, good.

>
> We suspect in this case, verification of "encrypted handshake message"
> failure is happening.

It's possible. It would be helpful if you can get more information from
the error stack on the server, e.g. by using ERR_print_errors_fp() or
something similar. I'm particularly interested in identifying the source
file and line number where the decrypt_error is coming from. Printing
the error stack should give us that information. There are a number of
places that a "decrypt error" can occur and it would be helpful to
identify which one is the cause of the problem.


> We understand constant_time_xx APIs get used for CBC padding validation.
> Will this have  any dependency on the compiler optimization or asm
> flags?

CBC padding validation is fairly independent of anything to do with RSA,
so I think its unlikely to be the culprit here. Of course sometimes
compiler optimization/asm issues do occur so it can't be ruled out
entirely - but it's not where I would start looking.

> Will this issue be seen if hardware takes more time for the
> operation?
>

No. Constant time here just means that we implement the code without any
branching based on secret data (e.g. no "if" statements/while loops etc
based on secret dependent data). It has very little to do with how long
something actually takes to process.


> Here is the snippet of the wireshark where our device acting as server
> closes the connection with decryption failure.

Thanks. To narrow it down further I need to figure out which line of
code the decrypt_error is coming from as described above.

Matt



> If you need any further info, please let us know. 
> image.png
> Thanks
> Mahendra
>
> Please suggest.
>
>
>
> On Fri, Oct 30, 2020 at 3:32 PM Matt Caswell <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>
>
>     On 30/10/2020 09:18, Mahendra SP wrote:
>     > Hi All.
>     >
>     > We have upgraded openssl version to 1.1.1b
>     >
>     > With this, we are seeing decryption error during SSL handshake for the
>     > below explained scenario. Our device acts as an SSL server.
>     >
>     > We have external hardware to offload RSA private key operations using
>     > the engine.
>     > Decryption of pre-master secret is done using hardware and is
>     > successful. We compared the pre-master secret on both server and
>     client
>     > and they match.
>     > However, we see that SSL handshake fails with "decrypt error (51)"
>     with
>     > an alert number 21. Verifying the encrypted finish message on the
>     server
>     > side fails.
>     >
>     > This issue does not happen with software performing RSA private key
>     > operations.
>     >
>     > Can someone help with the reason for decryption failure? Below is the
>     > compiler and processor details. It is 64 bit.
>     > arm-linux-gnueabihf-gcc  -march=armv7ve -mthumb -mfpu=neon
>     -mfloat-abi=hard
>
>     Potentially this is related to the use of PSS padding in libssl which is
>     mandated in TLSv1.3. The TLSv1.3 spec also requires its use even in
>     TLSv1.2.
>
>     The PSS padding is implemented within the EVP layer. Ultimately EVP
>     calls the function RSA_private_encrypt() with padding set to
>     RSA_NO_PADDING.
>
>     Assuming your engine is implemented via a custom RSA_METHOD does it
>     support RSA_private_encrypt(() with RSA_NO_PADDING? If not this is
>     likely to be the problem.
>
>     More discussion of this is here:
>
>     https://github.com/openssl/openssl/issues/7968
>
>     Also related is the recent discussion on this list about the CAPI engine
>     and this issue:
>
>     https://github.com/openssl/openssl/issues/8872
>
>     Matt
>
Reply | Threaded
Open this post in threaded view
|

Re: Decrypt error when using openssl 1.1.1b during SSL handshake

Matt Caswell-2


On 02/11/2020 09:00, Mahendra SP wrote:

> Hi Matt,
>
> Error is reported from this:
> FILE:../openssl-1.1.1b/ssl/statem/statem_srvr.c, FUNCTION:415,
> LINE:3055, reason=147, alert=51
>
> We see that hardware is returning 48 bytes. Even if the decrypted
> premaster data is correct, openssl is expecting more than 48 bytes in
> return.
> This check fails as decrypt_len is 48.
> decrypt_len < 11 + SSL_MAX_MASTER_KEY_LENGTH

Just above this line we call RSA_private_decrypt() with padding set to
RSA_NO_PADDING.

We expect the output *once padding is removed* to be 48 bytes. But
RSA_private_decrypt() should be returning the data *with the padding
included* (because we called it with RSA_NO_PADDING). The minimum valid
padding length is 11 bytes (hence the check above). So it looks to me
like the engine is incorrectly ignoring the RSA_NO_PADDING, and
stripping the padding anyway.

Matt

>
> We compared the data returned when software is used. Decrypt_len is 256
> bytes and the last 48 bytes are actual premaster secret. Also, openssl
> checks for if the first byte is 0 and second byte is 2. 
> We are trying to rectify this issue in hardware and return the correct data.
>
> Please suggest if you have any comments for the above info.
>
> Thanks
> Mahendra
>
> On Fri, Oct 30, 2020 at 7:50 PM Matt Caswell <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>
>
>     On 30/10/2020 11:22, Mahendra SP wrote:
>     > Hi Matt,
>     >
>     > Thank you for the inputs.
>     > Yes, we had encountered the padding issue initially. But we added
>     > support for RSA_NO_PADDING in our hardware. That's why we are able to
>     > successfully decrypt the premaster secret in the hardware.
>     > Hence the issue does not seem to be related to padding. We have
>     > confirmed this by comparing the premaster secret on both client and
>     > server and they are the same.
>
>     Ok, good.
>
>     >
>     > We suspect in this case, verification of "encrypted handshake message"
>     > failure is happening.
>
>     It's possible. It would be helpful if you can get more information from
>     the error stack on the server, e.g. by using ERR_print_errors_fp() or
>     something similar. I'm particularly interested in identifying the source
>     file and line number where the decrypt_error is coming from. Printing
>     the error stack should give us that information. There are a number of
>     places that a "decrypt error" can occur and it would be helpful to
>     identify which one is the cause of the problem.
>
>
>     > We understand constant_time_xx APIs get used for CBC padding
>     validation.
>     > Will this have  any dependency on the compiler optimization or asm
>     > flags?
>
>     CBC padding validation is fairly independent of anything to do with RSA,
>     so I think its unlikely to be the culprit here. Of course sometimes
>     compiler optimization/asm issues do occur so it can't be ruled out
>     entirely - but it's not where I would start looking.
>
>     > Will this issue be seen if hardware takes more time for the
>     > operation?
>     >
>
>     No. Constant time here just means that we implement the code without any
>     branching based on secret data (e.g. no "if" statements/while loops etc
>     based on secret dependent data). It has very little to do with how long
>     something actually takes to process.
>
>
>     > Here is the snippet of the wireshark where our device acting as server
>     > closes the connection with decryption failure.
>
>     Thanks. To narrow it down further I need to figure out which line of
>     code the decrypt_error is coming from as described above.
>
>     Matt
>
>
>
>     > If you need any further info, please let us know. 
>     > image.png
>     > Thanks
>     > Mahendra
>     >
>     > Please suggest.
>     >
>     >
>     >
>     > On Fri, Oct 30, 2020 at 3:32 PM Matt Caswell <[hidden email]
>     <mailto:[hidden email]>
>     > <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
>     >
>     >
>     >
>     >     On 30/10/2020 09:18, Mahendra SP wrote:
>     >     > Hi All.
>     >     >
>     >     > We have upgraded openssl version to 1.1.1b
>     >     >
>     >     > With this, we are seeing decryption error during SSL
>     handshake for the
>     >     > below explained scenario. Our device acts as an SSL server.
>     >     >
>     >     > We have external hardware to offload RSA private key
>     operations using
>     >     > the engine.
>     >     > Decryption of pre-master secret is done using hardware and is
>     >     > successful. We compared the pre-master secret on both server and
>     >     client
>     >     > and they match.
>     >     > However, we see that SSL handshake fails with "decrypt error
>     (51)"
>     >     with
>     >     > an alert number 21. Verifying the encrypted finish message
>     on the
>     >     server
>     >     > side fails.
>     >     >
>     >     > This issue does not happen with software performing RSA
>     private key
>     >     > operations.
>     >     >
>     >     > Can someone help with the reason for decryption failure?
>     Below is the
>     >     > compiler and processor details. It is 64 bit.
>     >     > arm-linux-gnueabihf-gcc  -march=armv7ve -mthumb -mfpu=neon
>     >     -mfloat-abi=hard
>     >
>     >     Potentially this is related to the use of PSS padding in
>     libssl which is
>     >     mandated in TLSv1.3. The TLSv1.3 spec also requires its use
>     even in
>     >     TLSv1.2.
>     >
>     >     The PSS padding is implemented within the EVP layer.
>     Ultimately EVP
>     >     calls the function RSA_private_encrypt() with padding set to
>     >     RSA_NO_PADDING.
>     >
>     >     Assuming your engine is implemented via a custom RSA_METHOD
>     does it
>     >     support RSA_private_encrypt(() with RSA_NO_PADDING? If not this is
>     >     likely to be the problem.
>     >
>     >     More discussion of this is here:
>     >
>     >     https://github.com/openssl/openssl/issues/7968
>     >
>     >     Also related is the recent discussion on this list about the
>     CAPI engine
>     >     and this issue:
>     >
>     >     https://github.com/openssl/openssl/issues/8872
>     >
>     >     Matt
>     >
>