[openssl.org #4614] pthread_once and malloc failures

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[openssl.org #4614] pthread_once and malloc failures

Rich Salz via RT
On Mon Jul 11 16:20:29 2016, [hidden email] wrote:

> Hi,
>
> When trying to check what happens if we simulate malloc()
> returning NULL I'm running into a problem that I'm not sure how to
> deal with.
>
> We have CRYPTO_THREAD_run_once(), which takes an init() function
> that returns void, so it can't return failures. At least the
> pthread_once() function also has it as void.
>
> But if those functions call malloc() and that returns NULL, we now
> don't catch that error, and later just try to use a NULL pointer.
>
> Anybody a good idea how to solve this?

Rethinking this...

Most of all, we use CRYPTO_THREAD_run_once() internally to initiate the first
locks, so pretty much in an initial state of the library (not entirely true,
since we do these inits opportunistically, but it's probable that they happen
very early on). If they are having memory allocation, the running app is
probably in deep shit anyway, so a hard assert in our diverse inits would
probably be appropriate either way.

It was suggested that we change the return type of the init routine to int and
have it return something appropriate (0 for error, 1 for success, say?), but
since at least pthread_once() takes a function returning void (and thereby
doesn't care about any returned value), it seems like our hands are forced
anyway.

--
Richard Levitte
[hidden email]

--
Ticket here: http://rt.openssl.org/Ticket/Display.html?id=4614
Please log in as guest with password guest if prompted

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: [openssl.org #4614] pthread_once and malloc failures

Rich Salz via RT


On 19/07/16 16:23, Richard Levitte via RT wrote:

> On Mon Jul 11 16:20:29 2016, [hidden email] wrote:
>> Hi,
>>
>> When trying to check what happens if we simulate malloc()
>> returning NULL I'm running into a problem that I'm not sure how to
>> deal with.
>>
>> We have CRYPTO_THREAD_run_once(), which takes an init() function
>> that returns void, so it can't return failures. At least the
>> pthread_once() function also has it as void.
>>
>> But if those functions call malloc() and that returns NULL, we now
>> don't catch that error, and later just try to use a NULL pointer.
>>
>> Anybody a good idea how to solve this?
>
> Rethinking this...
>
> Most of all, we use CRYPTO_THREAD_run_once() internally to initiate the first
> locks, so pretty much in an initial state of the library (not entirely true,
> since we do these inits opportunistically, but it's probable that they happen
> very early on). If they are having memory allocation, the running app is
> probably in deep shit anyway, so a hard assert in our diverse inits would
> probably be appropriate either way.

You are assuming that the application loads and inits OpenSSL early and
that it is critical to its function. It may not be.

Matt


--
Ticket here: http://rt.openssl.org/Ticket/Display.html?id=4614
Please log in as guest with password guest if prompted

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

[openssl.org #4614] pthread_once and malloc failures

Rich Salz via RT
In reply to this post by Rich Salz via RT
On Tue Jul 19 15:26:58 2016, matt wrote:

>
>
> On 19/07/16 16:23, Richard Levitte via RT wrote:
> > On Mon Jul 11 16:20:29 2016, [hidden email] wrote:
> >> Hi,
> >>
> >> When trying to check what happens if we simulate malloc()
> >> returning NULL I'm running into a problem that I'm not sure how to
> >> deal with.
> >>
> >> We have CRYPTO_THREAD_run_once(), which takes an init() function
> >> that returns void, so it can't return failures. At least the
> >> pthread_once() function also has it as void.
> >>
> >> But if those functions call malloc() and that returns NULL, we now
> >> don't catch that error, and later just try to use a NULL pointer.
> >>
> >> Anybody a good idea how to solve this?
> >
> > Rethinking this...
> >
> > Most of all, we use CRYPTO_THREAD_run_once() internally to initiate
> > the first
> > locks, so pretty much in an initial state of the library (not
> > entirely true,
> > since we do these inits opportunistically, but it's probable that
> > they happen
> > very early on). If they are having memory allocation, the running app
> > is
> > probably in deep shit anyway, so a hard assert in our diverse inits
> > would
> > probably be appropriate either way.
>
> You are assuming that the application loads and inits OpenSSL early
> and
> that it is critical to its function. It may not be.

True, I made that assumption.

Ok, so the other way I can think of is a bit ugly, but... using oob data, in
this case a variable that the init routine must set to signal that everything
went well. I'm thinking a variable in CRYPTO_THREAD_run_once() that gets passed
by reference to the init routine

To be noted is that we never check the value CRYPTO_THREAD_run_once()
returns... Should we make it __owur?

--
Richard Levitte
[hidden email]

--
Ticket here: http://rt.openssl.org/Ticket/Display.html?id=4614
Please log in as guest with password guest if prompted

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

[openssl.org #4614] pthread_once and malloc failures

Rich Salz via RT
In reply to this post by Rich Salz via RT
On Tue Jul 19 15:36:04 2016, levitte wrote:
> To be noted is that we never check the value CRYPTO_THREAD_run_once()
> returns... Should we make it __owur?

I spoke too fast. We do... just not always.

--
Richard Levitte
[hidden email]

--
Ticket here: http://rt.openssl.org/Ticket/Display.html?id=4614
Please log in as guest with password guest if prompted

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

[openssl.org #4614] pthread_once and malloc failures

Rich Salz via RT
In reply to this post by Rich Salz via RT
On Tue Jul 19 15:36:04 2016, levitte wrote:
> Ok, so the other way I can think of is a bit ugly, but... using oob
> data, in
> this case a variable that the init routine must set to signal that
> everything
> went well. I'm thinking a variable in CRYPTO_THREAD_run_once() that
> gets passed
> by reference to the init routine

Never mind that, I wasn't thinking straight...

--
Richard Levitte
[hidden email]

--
Ticket here: http://rt.openssl.org/Ticket/Display.html?id=4614
Please log in as guest with password guest if prompted

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: [openssl.org #4614] pthread_once and malloc failures

Rich Salz via RT
In reply to this post by Rich Salz via RT

> On Jul 19, 2016, at 5:26 PM, Matt Caswell via RT <[hidden email]> wrote:
>
>> Most of all, we use CRYPTO_THREAD_run_once() internally to initiate the first
>> locks, so pretty much in an initial state of the library (not entirely true,
>> since we do these inits opportunistically, but it's probable that they happen
>> very early on). If they are having memory allocation, the running app is
>> probably in deep shit anyway, so a hard assert in our diverse inits would
>> probably be appropriate either way.
>
> You are assuming that the application loads and inits OpenSSL early and
> that it is critical to its function. It may not be.
Postfix expects to be able to continue "degraded" operation without TLS,
when TLS initialization fails.  Asserts are not acceptable.  They would
cause Postfix to look for a different SSL library.

When initialization fails, all subsequent calls can return errors, but
assert is not an option.

--
        Viktor.


--
Ticket here: http://rt.openssl.org/Ticket/Display.html?id=4614
Please log in as guest with password guest if prompted


--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

smime.p7s (2K) Download Attachment