FIPS version of RSA_generate_key_ex puts FIPS library in a bad state

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

FIPS version of RSA_generate_key_ex puts FIPS library in a bad state

gperrow

If I attempt to create an RSA key pair with a size of >4096 bits using the FIPS library (FIPS 2.0.5, OpenSSL 1.0.1h), I get an error (“data too large for modulus”), but doing so seems to put the FIPS library into a bad state. Subsequent calls return failure and the error stack indicates that the FIPS self-test has failed.

 

Source for a short C program is attached. If I attempt to use a small key size (say 500 bits), generating the key fails but subsequent actions are OK. A key size between 1024 and 4096 works as expected. If I try to use a key size of 4097, generating the key fails but then subsequent FIPS calls also fail.

 

I am seeing this on both Windows 7 and Linux.

 

Graeme Perrow

 


rsabug.cpp (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: FIPS version of RSA_generate_key_ex puts FIPS library in a bad state

gperrow

Sorry for the extra message but my description below is not quite correct. The problem is not that the key size is > 4096. It seems to happen whenever the key size is not a multiple of 8.

 

Graeme

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Perrow, Graeme
Sent: Friday, August 01, 2014 11:50 AM
To: [hidden email]
Subject: FIPS version of RSA_generate_key_ex puts FIPS library in a bad state

 

If I attempt to create an RSA key pair with a size of >4096 bits using the FIPS library (FIPS 2.0.5, OpenSSL 1.0.1h), I get an error (“data too large for modulus”), but doing so seems to put the FIPS library into a bad state. Subsequent calls return failure and the error stack indicates that the FIPS self-test has failed.

 

Source for a short C program is attached. If I attempt to use a small key size (say 500 bits), generating the key fails but subsequent actions are OK. A key size between 1024 and 4096 works as expected. If I try to use a key size of 4097, generating the key fails but then subsequent FIPS calls also fail.

 

I am seeing this on both Windows 7 and Linux.

 

Graeme Perrow