Why would RSA_size() crash?

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Why would RSA_size() crash?

Stokestack
Hi all.

I'm trying to use the OpenSSL crypto lib.  I've generated a public/private RSA key pair.  Then I wrote some code to try to encrypt an eight-byte random string.  But it crashes in RSA_size().  Here's the code:

            BIO* bp = BIO_new_mem_buf(_publicKey, -1);                // Create a new memory buffer BIO.
            RSA* pubKey = PEM_read_bio_RSA_PUBKEY(bp, 0, 0, 0);        // And read the RSA key from it.
            BIO_free(bp);

            // Allocate a string to hold the encrypted result.
            unsigned char encryptedRandKey[RSA_size(pubKey)];

_pubKey is a null-terminated character string that contains the RSA public key in PEM format (including -----BEGIN and so forth).  RSA_size() crashes with bad access.  Removing BIO_free() didn't make any difference.  I also just tried assigning the value of RSA_size() to an int, and it crashed.

Any ideas here?

Thanks!

Gavin
Reply | Threaded
Open this post in threaded view
|

Re: Why would RSA_size() crash?

John Hascall


> I'm trying to use the OpenSSL crypto lib.  I've generated a public/private
> RSA key pair.  Then I wrote some code to try to encrypt an eight-byte random
> string.  But it crashes in RSA_size().  Here's the code:
>
>             BIO* bp = BIO_new_mem_buf(_publicKey, -1);                //
> Create a new memory buffer BIO.
>             RSA* pubKey = PEM_read_bio_RSA_PUBKEY(bp, 0, 0, 0);        //
> And read the RSA key from it.
>             BIO_free(bp);

        Are you checking the pointer pubKey after PEM_read_bio_RSA_PUBKEY?
        If it is NULL, RSA_size will die.

>             // Allocate a string to hold the encrypted result.
>             unsigned char encryptedRandKey[RSA_size(pubKey)];


John
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

R: Why would RSA_size() crash?

francesco.petruzzi
In reply to this post by Stokestack

Is pubKey a valid pointer after PEM_read_bio_RSA_PUBKEY?

If it  is NULL there is an error in PEM data.

 

Da: [hidden email] [mailto:[hidden email]] Per conto di G S
Inviato: lunedì 16 maggio 2011 12:13
A: [hidden email]
Oggetto: Why would RSA_size() crash?

 

Hi all.

I'm trying to use the OpenSSL crypto lib.  I've generated a public/private RSA key pair.  Then I wrote some code to try to encrypt an eight-byte random string.  But it crashes in RSA_size().  Here's the code:

            BIO* bp = BIO_new_mem_buf(_publicKey, -1);                // Create a new memory buffer BIO.
            RSA* pubKey = PEM_read_bio_RSA_PUBKEY(bp, 0, 0, 0);        // And read the RSA key from it.
            BIO_free(bp);

            // Allocate a string to hold the encrypted result.
            unsigned char encryptedRandKey[RSA_size(pubKey)];

_pubKey is a null-terminated character string that contains the RSA public key in PEM format (including -----BEGIN and so forth).  RSA_size() crashes with bad access.  Removing BIO_free() didn't make any difference.  I also just tried assigning the value of RSA_size() to an int, and it crashed.

Any ideas here?

Thanks!

Gavin

Reply | Threaded
Open this post in threaded view
|

Re: Why would RSA_size() crash?

Stokestack
In reply to this post by Stokestack
Duh, thanks to the people who pointed out that the pointer returned by PEM_read_bio_RSA_PUBKEY might be null, and indeed it is (sadly I have to use Xcode, which refuses to show any local variables and GDB claims they don't exist).

The question now is why it's null, since I know the string itself is populated.  It may be that newlines are getting stripped somehow.  I store all of the key text in a database and return it to the app as an XML element.  Dumping the element to the console shows newlines, but a printf of the character string I use as the BIO source does not show newlines.
Reply | Threaded
Open this post in threaded view
|

Re: Why would RSA_size() crash?

John Hascall

> Duh, thanks to the people who pointed out that the pointer returned by
> PEM_read_bio_RSA_PUBKEY might be null, and indeed it is (sadly I have to use
> Xcode, which refuses to show any local variables and GDB claims they don't
> exist).

    This is probably the optimizer, try compiling with -O0 -g3
    (and make sure you/the linker aren't stripping symbols)
    Pretty much always a good idea when developing.

John


> The question now is why it's null, since I know the string itself is
> populated.  It may be that newlines are getting stripped somehow.  I store
> all of the key text in a database and return it to the app as an XML
> element.  Dumping the element to the console shows newlines, but a printf of
> the character string I use as the BIO source does not show newlines.
>
> --485b397dd4e3cb172604a364d672
> Content-Type: text/html; charset=ISO-8859-1
> Content-Transfer-Encoding: quoted-printable
>
> Duh, thanks to the people who pointed out that the pointer returned by PEM_=
> read_bio_RSA_PUBKEY might be null, and indeed it is (sadly I have to use Xc=
> ode, which refuses to show any local variables and GDB claims they don'=
> t exist).<br>
> <br>The question now is why it&#39;s null, since I know the string itself i=
> s populated.=A0 It may be that newlines are getting stripped somehow.=A0 I =
> store all of the key text in a database and return it to the app as an XML =
> element.=A0 Dumping the element to the console shows newlines, but a printf=
>  of the character string I use as the BIO source does not show newlines.<br=
> >
>
> --485b397dd4e3cb172604a364d672--
> ______________________________________________________________________
> OpenSSL Project                                 http://www.openssl.org
> User Support Mailing List                    [hidden email]
> Automated List Manager                           [hidden email]
>

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Why would RSA_size() crash?

Stokestack
On Mon, May 16, 2011 at 6:53 AM, John Hascall <[hidden email]> wrote:

> (sadly I have to use
> Xcode, which refuses to show any local variables and GDB claims they don't
> exist).

   This is probably the optimizer, try compiling with -O0 -g3

Thanks, John, I am building Debug.  I verified that -O0 is being used, however -g3 does not appear in the transcript.  What does that option do?
Reply | Threaded
Open this post in threaded view
|

Re: Why would RSA_size() crash?

Stokestack
In reply to this post by John Hascall
Ah, I see the g3 option generates extra debugging info.  I'll give it a shot.  I have a bug open with Apple about this anyway.  It has proven to be very hard to pin down.  Restarting Xcode will usually eliminate the problem and let you step through code... ONCE.  If you want to do it again, you have to quit Xcode, reload the project, and go.  Infuriating.

And often it will afflict some files but not others.  I checked the currently troublesome one to make sure it was -O0.


Reply | Threaded
Open this post in threaded view
|

Re: Why would RSA_size() crash?

Stokestack
In reply to this post by John Hascall
OK, this is perplexing.  I have a PEM-format RSA key in a character string called _publicKey, with newlines between the header, key data, and trailer.  Like this:

-----BEGIN PUBLIC KEY----
MCwwHRTJKoZIhvcNAQEBBQADGwAwGAIRALPMoZzXMLIKhidteVfdR28CAwEAAQ==
-----END PUBLIC KEY-----

But PEM_read_bio_RSA_PUBKEY is still returning NULL, as in this:

            BIO* bp = BIO_new_mem_buf(_publicKey, -1);  // Create a new memory buffer BIO.
            RSA* pubKey = 0;
            if(bp)
            {
                pubKey = PEM_read_bio_RSA_PUBKEY(bp, 0, 0, 0);  // And read the RSA key from it.
                BIO_free(bp);
            }

Does anybody see anything I'm doing wrong?  I verified that the key string matches what's in the PEM file that was generated on the command line.

Thanks!
Reply | Threaded
Open this post in threaded view
|

Re: Why would RSA_size() crash?

Stokestack
In reply to this post by John Hascall
A follow-up: After seeing an example, I tried printing the result of ERR_reason_error_string(ERR_get_error()).  It's null.
Reply | Threaded
Open this post in threaded view
|

Re: Why would RSA_size() crash?

Jeffrey Walton-3
In reply to this post by John Hascall
On Mon, May 16, 2011 at 9:53 AM, John Hascall <[hidden email]> wrote:
>
>> Duh, thanks to the people who pointed out that the pointer returned by
>> PEM_read_bio_RSA_PUBKEY might be null, and indeed it is (sadly I have to use
>> Xcode, which refuses to show any local variables and GDB claims they don't
>> exist).
>
>    This is probably the optimizer, try compiling with -O0 -g3
>    (and make sure you/the linker aren't stripping symbols)
>    Pretty much always a good idea when developing.
Don't forget to define DEBUG and NDEBUG for debug and release. And for
debug builds, -Wall -Wextra goes a long way (it beats checking boxes).

Jeff
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Why would RSA_size() crash?

Dr. Stephen Henson
In reply to this post by Stokestack
On Mon, May 16, 2011, G S wrote:

> A follow-up: After seeing an example, I tried printing the result of
> ERR_reason_error_string(ERR_get_error()).  It's null.

ERR_print_errors_fp(stderr) might be more useful: see FAQ.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Why would RSA_size() crash?

Stokestack
Thanks to those who answered.  It was a simple goof using std::string's substr() method; I was treating the second parameter as the end position rather than the character count, thus lopping off some essential characters and causing the failure.  It works now.