Shouldn't no-pinshared be the default?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Shouldn't no-pinshared be the default?

Yann Ylavic
Hi,

after quite some time trying to convert Apache httpd (and libapr) to
new the OPENSSL_init/cleanup() 1.1 API, and wondering why openssl libs
would not unload with mod_ssl as before (1.0 and earlier), I found the
ELF NODELETE flag (gcc's -znodelete) and the new (no-)pinshared config
option (though available in 1.1.1 only, not in 1.1.0 AFAICT).

Unfortunately pinshared openssl can't work with the way httpd
dlopen/close()s mod_ssl, which itself initializes and cleans up
openssl, at each restart. Once openssl is cleaned up, it won't
re-initialize due to internal static variables (and it worked so far
in httpd because all the legacy cleanup methods currently used, like
OBJ/EVP/ENGINE_cleanup, are all no-ops now).

IIUC from some previous threads on this list, the new way to cleanup
openssl is to not cleanup (explicitely), which causes issues with some
usages obviously (and is possibly why build time no-pinshared and run
time OPENSSL_INIT_NO_ATEXIT appeared in 1.1.1).

So my question is, why isn't no-pinshared the default?
ISTM that pinshared is enabled on linux only, and linux has
__cxa_atexit semantics for atexit() already, so dlclose() should
already call OPENSSL_cleanup() when needed.
Thus with OPENSSL_INIT_NO_ATEXIT now available the user could choose
at runtime whether (s)he wants to call OPENSSL_cleanup() explicitely
or let openssl clean up by itself.

Of course one can build his/her own openssl with no-pinshared, but I
suppose distros don't, so all software that need to unload openssl
need to build their own one... Another concern is that none of
no-pinshared and OPENSSL_INIT_NO_ATEXIT is available in 1.1.0.

Am I missing something?

Regards,
Yann.
Reply | Threaded
Open this post in threaded view
|

Re: Shouldn't no-pinshared be the default?

Matt Caswell-2


On 04/03/2019 23:37, Yann Ylavic wrote:
> So my question is, why isn't no-pinshared the default?
> ISTM that pinshared is enabled on linux only,

That isn't correct. pinshared is the default everywhere. The way it is achieved
is different for different platforms (so on Linux we use -znodelete).

> and linux has
> __cxa_atexit semantics for atexit() already, so dlclose() should
> already call OPENSSL_cleanup() when needed.
> Thus with OPENSSL_INIT_NO_ATEXIT now available the user could choose
> at runtime whether (s)he wants to call OPENSSL_cleanup() explicitely
> or let openssl clean up by itself.

Actually if all platforms behaved like Linux then there would be no need for
pinshared at all. Unfortunately they don't and on some platforms atexit handlers
can get called even after they have been unloaded - which obviously leads to
crashes.

Feasibly we could make no-pinshared the default on platforms where it isn't
really needed (such as Linux). However:

1) This introduces a change of OpenSSL behaviour based on platform - which isn't
ideal for application developers targeting multiple platforms. Not sure how big
a deal this is.

2) The no-pinshared option does not appear in 1.1.1 or 1.1.1a. It first appears
in 1.1.1b. Backporting the option was considered ok. But changing the default
mid-series is probably not a good idea.

Changing the default could be considered for 3.0.

Matt
Reply | Threaded
Open this post in threaded view
|

Re: Shouldn't no-pinshared be the default?

Yann Ylavic
On Tue, Mar 5, 2019 at 12:51 PM Matt Caswell <[hidden email]> wrote:
>
> On 04/03/2019 23:37, Yann Ylavic wrote:
> > So my question is, why isn't no-pinshared the default?
> > ISTM that pinshared is enabled on linux only,
>
> That isn't correct. pinshared is the default everywhere. The way it is achieved
> is different for different platforms (so on Linux we use -znodelete).

Yes sorry, I meant -znodelete which, IMHO, is the worst option to keep
openssl loaded, at least double dlopen() can be worked around with
double dlclose(), while -znodelete is cast in stone at (distro-)build
time.
Also the double dlopen() could be opt-in/out at init time (some new
OPENSSL_INIT_[NO_]UNLOAD option), which looks far better for
usability, and with the right compatibility default could possibly
make it to 1.1 :p

>
> > and linux has
> > __cxa_atexit semantics for atexit() already, so dlclose() should
> > already call OPENSSL_cleanup() when needed.
> > Thus with OPENSSL_INIT_NO_ATEXIT now available the user could choose
> > at runtime whether (s)he wants to call OPENSSL_cleanup() explicitely
> > or let openssl clean up by itself.
>
> Actually if all platforms behaved like Linux then there would be no need for
> pinshared at all. Unfortunately they don't and on some platforms atexit handlers
> can get called even after they have been unloaded - which obviously leads to
> crashes.

Sure I understand that, but I think capable platforms shouldn't pay
the penalty, losing unload-ability looks like a serious 1.1 regression
to me.
This also begs the question of atexit() choice for cleanup, and why
there is a cleanup in the first place since it's called only when the
program exits (freeing memory is quite useless there, while
clean(s)ing it or the filesystem or whatever is definitely something
to be done on unload if that needs to happen).
Each system/compiler has probably its own way to run a callback on
dynamic unloading (__attribute__ destructor, #pragma fini, ...), so I
think it could/should be on a case by case basis. But since
OPENSSL_init_{crypto,ssl}/cleanup() is a nice step and now simple
solution for the user to control init/deinit, the poor man's choice of
"do it yourself" would still be better than "you can't do it", IMHO.

>
> Feasibly we could make no-pinshared the default on platforms where it isn't
> really needed (such as Linux). However:
>
> 1) This introduces a change of OpenSSL behaviour based on platform - which isn't
> ideal for application developers targeting multiple platforms. Not sure how big
> a deal this is.

Not sure either, though the current state is not ideal for developers
targeting other (not that insane) usages, which used to work
previously.
As an example, a server couldn't on restart, depending on its
configuration, switch to/from
INIT_{LOAD_CONFIG,ENGINE_*,ASYNC,ZLIB,...}.

>
> 2) The no-pinshared option does not appear in 1.1.1 or 1.1.1a. It first appears
> in 1.1.1b. Backporting the option was considered ok. But changing the default
> mid-series is probably not a good idea.
>
> Changing the default could be considered for 3.0.

Yes please, as it stands the 1.1 series is unloadable on the most used
openssl libraries, distros'. I find this a bit unfortunate, and more
#ifdef-ery to come (though I'd like the OPENSSL_INIT_[NO_]UNLOAD one
:) ).

Regards,
Yann.
Reply | Threaded
Open this post in threaded view
|

Re: Shouldn't no-pinshared be the default?

Tomas Mraz-2
On Tue, 2019-03-05 at 14:16 +0100, Yann Ylavic wrote:

> On Tue, Mar 5, 2019 at 12:51 PM Matt Caswell <[hidden email]>
> wrote:
> >
> > 2) The no-pinshared option does not appear in 1.1.1 or 1.1.1a. It
> > first appears
> > in 1.1.1b. Backporting the option was considered ok. But changing
> > the default
> > mid-series is probably not a good idea.
> >
> > Changing the default could be considered for 3.0.
>
> Yes please, as it stands the 1.1 series is unloadable on the most
> used
> openssl libraries, distros'. I find this a bit unfortunate, and more
> #ifdef-ery to come (though I'd like the OPENSSL_INIT_[NO_]UNLOAD one
> :) ).

But is it in reality at all possible to explicitly unload OpenSSL?
You're talking here about mod_ssl but what if the OpenSSL is loaded not
just by mod_ssl but by other shared library loaded into the httpd
process - for example libkrb5 or libldap. Then you can see what
disaster can happen if mod_ssl on unload explicitly calls
OpenSSL_cleanup().

The explicit cleanup is thus simply a no-go in distro-wide use of
OpenSSL.

--
Tomáš Mráz
No matter how far down the wrong road you've gone, turn back.
                                              Turkish proverb
[You'll know whether the road is wrong if you carefully listen to your
conscience.]


Reply | Threaded
Open this post in threaded view
|

Re: Shouldn't no-pinshared be the default?

OpenSSL - User mailing list
On 05/03/2019 14:47, Tomas Mraz wrote:

> On Tue, 2019-03-05 at 14:16 +0100, Yann Ylavic wrote:
>> On Tue, Mar 5, 2019 at 12:51 PM Matt Caswell <[hidden email]>
>> wrote:
>>> 2) The no-pinshared option does not appear in 1.1.1 or 1.1.1a. It
>>> first appears
>>> in 1.1.1b. Backporting the option was considered ok. But changing
>>> the default
>>> mid-series is probably not a good idea.
>>>
>>> Changing the default could be considered for 3.0.
>> Yes please, as it stands the 1.1 series is unloadable on the most
>> used
>> openssl libraries, distros'. I find this a bit unfortunate, and more
>> #ifdef-ery to come (though I'd like the OPENSSL_INIT_[NO_]UNLOAD one
>> :) ).
> But is it in reality at all possible to explicitly unload OpenSSL?
> You're talking here about mod_ssl but what if the OpenSSL is loaded not
> just by mod_ssl but by other shared library loaded into the httpd
> process - for example libkrb5 or libldap. Then you can see what
> disaster can happen if mod_ssl on unload explicitly calls
> OpenSSL_cleanup().
>
> The explicit cleanup is thus simply a no-go in distro-wide use of
> OpenSSL.
>

On any system with a shared library concept (perhaps except some
historic abominations), shared library unload reference counting
is already handled by the OS/loader.  Thus any shared library that
cleans itself up when actually unloaded by the OS should just work.

Unfortunately OpenSSL 1.1.x chose to deliberately prevent itself
from being unloaded instead of making an internal abstraction for
the 3 unload scenarios common to all shared library systems:

A. Process is exiting, shared library is being unloaded along
   with the rest of the process.
      A few resources (such as network sockets and C FILE* objects)
   should be graciously cleaned up, memory will be freed unless the
   OS is "no-MMU" style (Linux-noMMU, Win32s).  Any threads still
   accessing the library will exit very soon too.

B. Shared library is being unloaded before process exit was
   requested, all library resources must be cleaned up to avoid
   becoming memory leaks. No clients are trying to use the
   library, but any long-running library APIs might not have
   returned (due to application mistakes or lack of abort APIs).
Threads still accessing the library need to synchronize
   their orderly return to callers outside the shared library.
    If OS serialization allows, the library needs to delay
   completion of the unload handler until the threads have
   left the library.

C. A client application requests cleanup manually.  Shared
   libraries accessed by multiple in-process "applications"
   (Your example above) will need to sometimes ignore such
   requests, static libraries can assume only one caller and
   should do the requested cleanup.

About 25 years ago I struggled with another library that did
the same kind of unload-blocking that OpenSSL 1.1.x does.  It
was sad to see a big project like OpenSSL repeat that mistake.

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

Reply | Threaded
Open this post in threaded view
|

Re: Shouldn't no-pinshared be the default?

Yann Ylavic
In reply to this post by Tomas Mraz-2
On Tue, Mar 5, 2019 at 2:47 PM Tomas Mraz <[hidden email]> wrote:
>
> But is it in reality at all possible to explicitly unload OpenSSL?

Well, I don't want to forcibly unload openssl precisely, I want it to
unload when its refcount reaches zero.

> You're talking here about mod_ssl but what if the OpenSSL is loaded not
> just by mod_ssl but by other shared library loaded into the httpd
> process - for example libkrb5 or libldap. Then you can see what
> disaster can happen if mod_ssl on unload explicitly calls
> OpenSSL_cleanup().

Yes, I don't want to blindly cleanup openssl in mod_ssl either, I want
modules to cooperate on who initializes/cleans-up.
There is no such thing as httpd needing libkrb5 or libldap, modules
may, and the APR library (which httpd [h]e[a]vily relies on) may also
link to openssl and need its init/cleanup.
So my goal is to make the APR library able and responsible to
init/cleanup openssl for httpd and its modules (which is not an easy
thing to do properly from 0.9.8 to latest..., so a single place to do
it well makes sense), based on its lifetime handling mechanism, namely
APR pools.
If openssl is needed by (and for the whole lifetime of) httpd, bind it
to the process pool, if it's needed by module(s) bind it to the
configuration pool and init/cleanup will happen explictely once and
only once at the right time. Modules that cooperate with APR will use
its apr_crypto_lib helpers and trust init/cleanup to be done once and
only once. Modules that don't cooperate with APR (or using libs that
don't) will still OPENSSL_init_*() but it's not an issue (can be
called multiple times), same for OPENSSL_cleanup() although I doubt
one is doing this (besides mod_ssl, which precisely I want to fix).
Anyway modules are effectively unloaded on stop/restart in httpd, and
so are the libs they are linked with, and the next start does not
necessarily requires the same openssl intialization/configuration, but
it should not be an issue provided modules can cooperate. httpd is the
(final-)application which knows here, it's not a lib like openssl or
APR (both of which shouldn't impose their lifetime to their users). So
likewise APR will also have an option to not init/cleanup openssl at
all since it's possibly not the only consumer, but for those who wants
to cooperate fully through APR (like httpd), it's also possible.

>
> The explicit cleanup is thus simply a no-go in distro-wide use of
> OpenSSL.

Why? Distros know better than the applications they run?
Since we are here, why OPENSSL_cleanup() exists and is public in the
first place, and why no-pinshared or OPENSSL_INIT_NO_ATEXIT?


Regards,
Yann.
Reply | Threaded
Open this post in threaded view
|

Re: Shouldn't no-pinshared be the default?

Richard Levitte - VMS Whacker-2
In reply to this post by Tomas Mraz-2


Tomas Mraz <[hidden email]> skrev: (5 mars 2019 14:47:18 CET)

>On Tue, 2019-03-05 at 14:16 +0100, Yann Ylavic wrote:
>> On Tue, Mar 5, 2019 at 12:51 PM Matt Caswell <[hidden email]>
>> wrote:
>> >
>> > 2) The no-pinshared option does not appear in 1.1.1 or 1.1.1a. It
>> > first appears
>> > in 1.1.1b. Backporting the option was considered ok. But changing
>> > the default
>> > mid-series is probably not a good idea.
>> >
>> > Changing the default could be considered for 3.0.
>>
>> Yes please, as it stands the 1.1 series is unloadable on the most
>> used
>> openssl libraries, distros'. I find this a bit unfortunate, and more
>> #ifdef-ery to come (though I'd like the OPENSSL_INIT_[NO_]UNLOAD one
>> :) ).
>
>But is it in reality at all possible to explicitly unload OpenSSL?
>You're talking here about mod_ssl but what if the OpenSSL is loaded not
>just by mod_ssl but by other shared library loaded into the httpd
>process - for example libkrb5 or libldap. Then you can see what
>disaster can happen if mod_ssl on unload explicitly calls
>OpenSSL_cleanup().
>
>The explicit cleanup is thus simply a no-go in distro-wide use of
>OpenSSL.

It sounds like an allocatable library context that could be used to store all the "global" stuff would be a good thing.
Incidently, we've introduced that concept for 3.0.0. Exactly what will end up in it is not decided, apart from the new provider related stuff.

Cheers
Richard

--
Skickat från min Android-enhet med K-9 Mail. Ursäkta min fåordighet.
Reply | Threaded
Open this post in threaded view
|

Re: Shouldn't no-pinshared be the default?

Tomas Mraz-2
In reply to this post by Yann Ylavic
On Tue, 2019-03-05 at 16:00 +0100, Yann Ylavic wrote:
> On Tue, Mar 5, 2019 at 2:47 PM Tomas Mraz <[hidden email]> wrote:
> >
> Why? Distros know better than the applications they run?

They actually do, because applications cannot really know whats deep in
the chain of loaded shared libraries - for example getpwnam() can load
libnss_ldap which can load libldap which can load libssl. And the
application has no idea about what is your nsswitch.conf config.

> Since we are here, why OPENSSL_cleanup() exists and is public in the
> first place, and why no-pinshared or OPENSSL_INIT_NO_ATEXIT?

Yes, having the public OPENSSL_cleanup() to be anything else than no-op
is probably a mistake.

--
Tomáš Mráz
No matter how far down the wrong road you've gone, turn back.
                                              Turkish proverb
[You'll know whether the road is wrong if you carefully listen to your
conscience.]


Reply | Threaded
Open this post in threaded view
|

Re: Shouldn't no-pinshared be the default?

Yann Ylavic
On Tue, Mar 5, 2019 at 6:05 PM Tomas Mraz <[hidden email]> wrote:

>
> On Tue, 2019-03-05 at 16:00 +0100, Yann Ylavic wrote:
> > On Tue, Mar 5, 2019 at 2:47 PM Tomas Mraz <[hidden email]> wrote:
> > >
> > Why? Distros know better than the applications they run?
>
> They actually do, because applications cannot really know whats deep in
> the chain of loaded shared libraries - for example getpwnam() can load
> libnss_ldap which can load libldap which can load libssl. And the
> application has no idea about what is your nsswitch.conf config.

Who would do that seriously, configure a non-local User for httpd, and
read it (as root) preferably from a remote LDAP??
No, httpd shouldn't be run like this, and httpd developers know it
because they designed the root/main process with no other dependency
than the APR lib (both for portability and security reasons), and I'm
sure distros know it too.

Furthermore, if that scenario were a real use case, it'd mean that
libldap could initialize openssl with no regard to httpd needs,
possibly no-op'ing further OPENSSL_init_*() calls with its own
arbitrary init option (e.g. OPENSSL_INIT_[NO_]LOAD_CONFIG), while the
application really is httpd here (I'm sure openldap uses minimal init,
but since we are talking hypothetically..).

So really, please let standalone applications precisely alone and
choose what's best for them for their lifetime.

>
> > Since we are here, why OPENSSL_cleanup() exists and is public in the
> > first place, and why no-pinshared or OPENSSL_INIT_NO_ATEXIT?
>
> Yes, having the public OPENSSL_cleanup() to be anything else than no-op
> is probably a mistake.

We'll probably agree to disagree here..

Regards,
Yann.
Reply | Threaded
Open this post in threaded view
|

Re: Shouldn't no-pinshared be the default?

Yann Ylavic
In reply to this post by Yann Ylavic
On Tue, Mar 5, 2019 at 2:16 PM Yann Ylavic <[hidden email]> wrote:
>
> Also the double dlopen() could be opt-in/out at init time (some new
> OPENSSL_INIT_[NO_]UNLOAD option), which looks far better for
> usability, and with the right compatibility default could possibly
> make it to 1.1 :p

FWIW, I just opened PR #8420 for this..

Regards,
Yann.
Reply | Threaded
Open this post in threaded view
|

Re: Shouldn't no-pinshared be the default?

JordanBrown
In reply to this post by Yann Ylavic
On 3/5/2019 1:16 PM, Yann Ylavic wrote:
Furthermore, if that scenario were a real use case, it'd mean that
libldap could initialize openssl with no regard to httpd needs,


Everybody has to play nice, but ... yes.  Random libraries might need OpenSSL for their own reasons, and the application cannot predict which libraries might need it or what those reasons might be.  OpenSSL must be designed to be used by multiple non-coordinated components running in the same process, including by dynamically loaded and unloaded shared objects.

-- 
Jordan Brown, Oracle ZFS Storage Appliance, Oracle Solaris
Reply | Threaded
Open this post in threaded view
|

Re: Shouldn't no-pinshared be the default?

OpenSSL - User mailing list
On 06/03/2019 18:38, Jordan Brown wrote:

> On 3/5/2019 1:16 PM, Yann Ylavic wrote:
>> Furthermore, if that scenario were a real use case, it'd mean that
>> libldap could initialize openssl with no regard to httpd needs,
>
>
> Everybody has to play nice, but ... yes.  Random libraries might need
> OpenSSL for their own reasons, and the application cannot predict
> which libraries might need it or what those reasons might be.  OpenSSL
> must be designed to be used by multiple non-coordinated components
> running in the same process, including by dynamically loaded and
> unloaded shared objects.
>
Which is why using the reference count already kept by the OS
loader is such a nice solution.


Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded