[openssl.org #1090] [BUG] Segfault in dgst signing with rsa private key

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[openssl.org #1090] [BUG] Segfault in dgst signing with rsa private key

Rich Salz via RT

Hello!

I use 0.9.8-stable-SNAP-20050601 snapshot. Problem occurs on digest
signing.

make report:
========
OpenSSL self-test report:

OpenSSL version:  0.9.8-beta4-dev
Last change:      Correct naming of the 'chil' and '4758cca' ENGINEs. Thi...
Options:          -g enable-shared enable-zlib no-gmp no-krb5 no-mdc2 no-rc5 no-zlib-dynamic
OS (uname):       Linux manul 2.4.26-1-386 #1 Tue Aug 24 13:31:19 JST 2004 i686 GNU/Linux
OS (config):      i686-whatever-linux2
Target (default): linux-elf
Target:           linux-elf
Compiler:         Configured with: ../src/configure -v --enable-languages=c,c++,java,f77,pascal,objc,ada,treelang --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-gxx-include-dir=/usr/include/c++/3.3 --enable-shared --enable-__cxa_atexit --with-system-zlib --enable-nls --without-included-gettext --enable-clocale=gnu --enable-debug --enable-java-gc=boehm --enable-java-awt=xlib --enable-objc-gc i486-linux
Thread model: posix
gcc version 3.3.5 (Debian 1:3.3.5-12)

Test skipped.
=======
Native tests are passed. So I do:

LD_LIBRARY_PATH=. ./apps/openssl req -newkey rsa:512 -nodes -batch -keyout keyrsa.pem -out reqrsa.pem -config apps/openssl.cnf
LD_LIBRARY_PATH=. ./apps/openssl x509 -req -set_serial 1 -signkey keyrsa.pem -in reqrsa.pem -out certrsa.pem
LD_LIBRARY_PATH=. ./apps/openssl dgst -sha1 -sign keyrsa.pem -out dsignrsa.bin CHANGES

It causes a segfault with backtrace
======
#0  0x400bdca8 in BN_BLINDING_set_thread_id (b=0x803, n=1076728596)
     at bn_blind.c:267
267             b->thread_id = n;
(gdb) bt
#0  0x400bdca8 in BN_BLINDING_set_thread_id (b=0x803, n=1076728596)
     at bn_blind.c:267
#1  0x400d7f85 in RSA_setup_blinding (rsa=0x80b4068, in_ctx=0x80b3e18)
     at rsa_lib.c:405
#2  0x400d6fc9 in rsa_get_blinding (rsa=0x80b4068, r=0xbffff198,
     local=0xbffff19c, ctx=0x80b3e18) at rsa_eay.c:251
#3  0x400d5e3e in RSA_eay_private_encrypt (flen=35,
     from=0x80b3c30 "0!0\t\006\005+\016\003\002\032\005",
     to=0x80b1a10 "S. Engelschall]\n\n  *) Fix the various library and apps files to free up pkeys obtained from\n     X509_PUBKEY_get() et al. Also allow x509.c to handle netscape extensions.\n     [Steve Henson]\n\n  *) Fix"...,
     rsa=0x80b4068, padding=1) at rsa_eay.c:361
#4  0x400d7c50 in RSA_private_encrypt (flen=2051,
     from=0x803 <Address 0x803 out of bounds>,
     to=0x803 <Address 0x803 out of bounds>, rsa=0x402d9314, padding=2051)
     at rsa_lib.c:288
#5  0x400d83f7 in RSA_sign (type=64, m=0x1 <Address 0x1 out of bounds>,
     m_len=64,
     sigret=0x80b1a10 "S. Engelschall]\n\n  *) Fix the various library and apps files to free up pkeys obtained from\n     X509_PUBKEY_get() et al. Also allow x509.c to handle netscape extensions.\n     [Steve Henson]\n\n  *) Fix"...,
     siglen=0xbffff330, rsa=0x23) at rsa_sign.c:132
#6  0x400fb7e1 in EVP_SignFinal (ctx=0x80b1a10,
     sigret=0x803 <Address 0x803 out of bounds>, siglen=0xbffff330,
     pkey=0xbffff2c0) at p_sign.c:111
#7  0x0805c722 in do_fp (out=0x80b3be8,
     buf=0x80b1a10 "S. Engelschall]\n\n  *) Fix the various library and apps files to free up pkeys obtained from\n     X509_PUBKEY_get() et al. Also allow x509.c to handle netscape extensions.\n     [Steve Henson]\n\n  *) Fix"...,
     bp=0xbffff330, sep=0, binout=1, key=0x80b3de8, sigin=0x0, siglen=2051,
     title=0x80920cb "", file=0x803 <Address 0x803 out of bounds>) at dgst.c:453
#8  0x0805bea5 in dgst_main (argc=0, argv=0xbffff9d0) at dgst.c:385
#9  0x08055d3a in do_cmd (prog=0x80b1290, argc=7, argv=0xbffff9b8)
     at openssl.c:382
#10 0x08055b6e in main (Argc=7, Argv=0xbffff9b8) at openssl.c:301
======

When I specify -rand, segfault doesn't occur

LD_LIBRARY_PATH=. ./apps/openssl dgst -sha1 -sign keyrsa.pem -out dsignrsa.bin -rand /dev/random CHANGES

finish successfully.

The same problem is on smime.

I do:
======
LD_LIBRARY_PATH=. ./apps/openssl smime -encrypt -binary -in CHANGES -aes256 -out encryptionrsa.pem -outform pem certrsa.pem
LD_LIBRARY_PATH=. ./apps/openssl smime -decrypt -binary -in encryptionrsa.pem -recip certrsa.pem -inkey keyrsa.pem -out smime_decrrsa.dump -inform pem
=====

Segfault occurs on decrypt with the same backtrace:
=========
#0  0x400bdca8 in BN_BLINDING_set_thread_id (b=0xb51, n=1076728596)
     at bn_blind.c:267
267             b->thread_id = n;
(gdb) bt
#0  0x400bdca8 in BN_BLINDING_set_thread_id (b=0xb51, n=1076728596)
     at bn_blind.c:267
#1  0x400d7f85 in RSA_setup_blinding (rsa=0x80b1e48, in_ctx=0x80b4050)
     at rsa_lib.c:405
#2  0x400d6fc9 in rsa_get_blinding (rsa=0x80b1e48, r=0xbfffe198,
     local=0xbfffe19c, ctx=0x80b4050) at rsa_eay.c:251
#3  0x400d649f in RSA_eay_private_decrypt (flen=64,
     from=0xb51 <Address 0xb51 out of bounds>,
     to=0x80b4000 "MobvxUlZUTDmtnqei5qEsbdjUzWrlWk/yhAu1MpYYjtAOmUh/0OwN+ske\nKGegsfJuRc1C1alZTc1", rsa=0x80b1e48, padding=1) at rsa_eay.c:482
#4  0x400d7c90 in RSA_private_decrypt (flen=2897,
     from=0xb51 <Address 0xb51 out of bounds>,
     to=0xb51 <Address 0xb51 out of bounds>, rsa=0x402d9314, padding=2897)
     at rsa_lib.c:294
#5  0x400fc61f in EVP_PKEY_decrypt (key=0xb51 <Address 0xb51 out of bounds>,
     ek=0xb51 <Address 0xb51 out of bounds>, ekl=2897, priv=0xb51) at p_dec.c:83
#6  0x4013edf9 in PKCS7_dataDecode (p7=0x80b1bf8, pkey=0x80b1bd8, in_bio=0x0,
     pcert=0x80b1eb8) at pk7_doit.c:442
#7  0x40141126 in PKCS7_decrypt (p7=0x80b1bf8, pkey=0x80b1bd8, cert=0x0,
     data=0x80b1d98, flags=128) at pk7_smime.c:450
#8  0x08089995 in smime_main (argc=13, argv=0xbffff968) at smime.c:687
#9  0x08055d3a in do_cmd (prog=0x80b1290, argc=13, argv=0xbffff968)
     at openssl.c:382
#10 0x08055b6e in main (Argc=13, Argv=0xbffff968) at openssl.c:301
=========

The problem doesn't exist on 20050523 snapshot.

Thank you.

PS. We have a extra test suite testing openssl executable.
Unfortunately, it's GNU make specific. Are you interested in it?

--
SY, Dmitry Belyavsky (ICQ UIN 11116575)

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [openssl.org #1090] [BUG] Segfault in dgst signing with rsa private key

Jostein Tveit
"Dmitry Belyavsky via RT" <[hidden email]> writes:

> I use 0.9.8-stable-SNAP-20050601 snapshot. Problem occurs on digest
> signing.

This seems to be the same bug as I reported just a couple of
hours ago on openssl-dev.
(Se post with subject "openssl-0.9.8-beta3, Sun Forte compiler,
64-bit Solaris 8, core dump and RNG error.")

Try to see if you get the same error as me if you apply the error
checking patch:

diff -ur openssl-0.9.8-beta3-orig/crypto/rsa/rsa_lib.c openssl-0.9.8-beta3-work/crypto/rsa/rsa_lib.c
--- openssl-0.9.8-beta3-orig/crypto/rsa/rsa_lib.c 2005-05-11 03:45:33.000000000 +0000
+++ openssl-0.9.8-beta3-work/crypto/rsa/rsa_lib.c 2005-06-01 12:31:33.490002000 +0000
@@ -402,6 +402,11 @@

        ret = BN_BLINDING_create_param(NULL, e, rsa->n, ctx,
                        rsa->meth->bn_mod_exp, rsa->_method_mod_n);
+       if (!ret)
+               {
+               RSAerr(RSA_F_RSA_SETUP_BLINDING, ERR_R_MALLOC_FAILURE);
+               goto err;
+               }
        BN_BLINDING_set_thread_id(ret, CRYPTO_thread_id());
 err:
        BN_CTX_end(ctx);


--
Jostein Tveit <[hidden email]>
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Side note to <openssl-dev> subscribers Was: Re: [openssl.org #1090] [BUG] Segfault in dgst signing with rsa private key

Andy Polyakov
>>I use 0.9.8-stable-SNAP-20050601 snapshot. Problem occurs on digest
>>signing.
>
>
> This seems to be the same bug as I reported just a couple of
> hours ago on openssl-dev.
>
> Try to see if you get the same error as me if you apply the error
> checking patch:

OpenSSL request tracker and <openssl-dev@...> list are not as
"interchangeable" as one might expect. Most notably, when you reply to
RT messages forwarded to <openssl-dev@...> [such as previous one],
originator who filed the problem report does *not* receive your post,
unless [s]he is also subscribed to <openssl-dev@...> list. In other
words, when you address a person in reply to RT message and reply to
<openssl-dev@...>, you don't actually know if [s]he will read your
comment. This is because subscription to <openssl-dev@...> is not
required, when you file a problem report through RT. If you want to
*reliably* reach any particular request originator *replace*
<openssl-dev@...> with <rt@...> when replying.

A side note to side note:-) The problem in question is adverse effect of
quoting DEVRANDOM... A.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [openssl.org #1090] [BUG] Segfault in dgst signing with rsa private key

Dmitry Belyavsky
In reply to this post by Jostein Tveit
Hello!

On Wed, 1 Jun 2005, Jostein Tveit wrote:

> "Dmitry Belyavsky via RT" <[hidden email]> writes:
>
>> I use 0.9.8-stable-SNAP-20050601 snapshot. Problem occurs on digest
>> signing.
>
> This seems to be the same bug as I reported just a couple of
> hours ago on openssl-dev.
> (Se post with subject "openssl-0.9.8-beta3, Sun Forte compiler,
> 64-bit Solaris 8, core dump and RNG error.")

It seems to be the same with the 1st part of bug report.
But smime -decrypt shouldn't use random data.

> Try to see if you get the same error as me if you apply the error
> checking patch:
>
> diff -ur openssl-0.9.8-beta3-orig/crypto/rsa/rsa_lib.c openssl-0.9.8-beta3-work/crypto/rsa/rsa_lib.c
> --- openssl-0.9.8-beta3-orig/crypto/rsa/rsa_lib.c 2005-05-11 03:45:33.000000000 +0000
> +++ openssl-0.9.8-beta3-work/crypto/rsa/rsa_lib.c 2005-06-01 12:31:33.490002000 +0000
> @@ -402,6 +402,11 @@
>
>        ret = BN_BLINDING_create_param(NULL, e, rsa->n, ctx,
>                        rsa->meth->bn_mod_exp, rsa->_method_mod_n);
> +       if (!ret)
> +               {
> +               RSAerr(RSA_F_RSA_SETUP_BLINDING, ERR_R_MALLOC_FAILURE);
> +               goto err;
> +               }
>        BN_BLINDING_set_thread_id(ret, CRYPTO_thread_id());
> err:
>        BN_CTX_end(ctx);

So if error is occurs in BN_BLINDING_create_param() because of expecting
random data, your patch shouldn't help to decrypt a message.

Thank you.
--
SY, Dmitry Belyavsky (ICQ UIN 11116575)

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]