Safety of using OpenSSL from multiple components in the one process

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Safety of using OpenSSL from multiple components in the one process

Steven Reddie
Hi All,
 
This is something that I think I've raised before but don't remember getting resolution on.
 
OpenSSL maintains various global variables and structures, and there are cleanup functions that must be used to properly release the resources when finished.  One example is the OID database managed by the "add_all_algorithms" function and it's associated release function, EVP_cleanup.  All is good when the use of OpenSSL is fairly simple, such as a single component using it for the lifetime of the process.
 
Where things get difficult/dangerous is when multiple seperate components in the one process, with no real knowledge of each other, make use of OpenSSL, and it's even worse if they dynamically load and unload OpenSSL using dlopen/LoadLibrary.  With large enterprise applications this is a common situation since different teams develop components that the large product makes us of, and with the increasing use of "plug-in" architectures the dynamic loading/unloading is not uncommon.
 
There seems to be no way offered by the OpenSSL API for these components to behave well.  If they each do a dlopen -> dlsym -> ... -> EVP_cleanup -> dlclose sequence then it seems that they will trample on each other.  If they take the extreme opposite and don't call EVP_cleanup then the process will leak until it falls over.
 
This is a serious issue that I believe impacts the stability and therefore limits the usefulness of OpenSSL in large enterprise applications.  Does anyone else have any thoughts on this?
 
Regards,
 
Steven
 
Reply | Threaded
Open this post in threaded view
|

Re: Safety of using OpenSSL from multiple components in the one process

Peter Waltenberg

You are correct it is a problem in enterprise applications.  
The good news is that it can be resolved, the bad news is that we did that outside the OpenSSL codebase itself.

We created new startup/shutdown functions in another library, added reference counting and load OpenSSL via that.
It might be possible to do something simillar within the existing API's, but brute force did work.

Peter

Peter Waltenberg



"Steven Reddie" <[hidden email]>
Sent by: [hidden email]

03/08/2005 06:42 PM

Please respond to
openssl-dev

To
<[hidden email]>
cc
Subject
Safety of using OpenSSL from multiple components in the one process





Hi All,
 
This is something that I think I've raised before but don't remember getting resolution on.
 
OpenSSL maintains various global variables and structures, and there are cleanup functions that must be used to properly release the resources when finished.  One example is the OID database managed by the "add_all_algorithms" function and it's associated release function, EVP_cleanup.  All is good when the use of OpenSSL is fairly simple, such as a single component using it for the lifetime of the process.
 
Where things get difficult/dangerous is when multiple seperate components in the one process, with no real knowledge of each other, make use of OpenSSL, and it's even worse if they dynamically load and unload OpenSSL using dlopen/LoadLibrary.  With large enterprise applications this is a common situation since different teams develop components that the large product makes us of, and with the increasing use of "plug-in" architectures the dynamic loading/unloading is not uncommon.
 
There seems to be no way offered by the OpenSSL API for these components to behave well.  If they each do a dlopen -> dlsym -> ... -> EVP_cleanup -> dlclose sequence then it seems that they will trample on each other.  If they take the extreme opposite and don't call EVP_cleanup then the process will leak until it falls over.
 
This is a serious issue that I believe impacts the stability and therefore limits the usefulness of OpenSSL in large enterprise applications.  Does anyone else have any thoughts on this?
 
Regards,
 
Steven
 
Reply | Threaded
Open this post in threaded view
|

RE: Safety of using OpenSSL from multiple components in the one process

Steven Reddie
Hi Peter,
 
There is a requirement in my case for the OpenSSL API to be accessible to several components.  In your case was your higher level component the only direct user of OpenSSL?  If not, how did you make the API available to other components?  Were the new functions accessed from one DLL with the OpenSSL functions accessed directly from the other DLL, or did you have redirecting stubs in your new DLL?
 
Steven


From: [hidden email] [mailto:[hidden email]] On Behalf Of Peter Waltenberg
Sent: Wednesday, 3 August 2005 8:48 PM
To: [hidden email]
Cc: [hidden email]
Subject: Re: Safety of using OpenSSL from multiple components in the one process


You are correct it is a problem in enterprise applications.  
The good news is that it can be resolved, the bad news is that we did that outside the OpenSSL codebase itself.

We created new startup/shutdown functions in another library, added reference counting and load OpenSSL via that.
It might be possible to do something simillar within the existing API's, but brute force did work.

Peter

Peter Waltenberg



"Steven Reddie" <[hidden email]>
Sent by: [hidden email]

03/08/2005 06:42 PM

Please respond to
openssl-dev

To
<[hidden email]>
cc
Subject
Safety of using OpenSSL from multiple components in the one process





Hi All,
 
This is something that I think I've raised before but don't remember getting resolution on.
 
OpenSSL maintains various global variables and structures, and there are cleanup functions that must be used to properly release the resources when finished.  One example is the OID database managed by the "add_all_algorithms" function and it's associated release function, EVP_cleanup.  All is good when the use of OpenSSL is fairly simple, such as a single component using it for the lifetime of the process.
 
Where things get difficult/dangerous is when multiple seperate components in the one process, with no real knowledge of each other, make use of OpenSSL, and it's even worse if they dynamically load and unload OpenSSL using dlopen/LoadLibrary.  With large enterprise applications this is a common situation since different teams develop components that the large product makes us of, and with the increasing use of "plug-in" architectures the dynamic loading/unloading is not uncommon.
 
There seems to be no way offered by the OpenSSL API for these components to behave well.  If they each do a dlopen -> dlsym -> ... -> EVP_cleanup -> dlclose sequence then it seems that they will trample on each other.  If they take the extreme opposite and don't call EVP_cleanup then the process will leak until it falls over.
 
This is a serious issue that I believe impacts the stability and therefore limits the usefulness of OpenSSL in large enterprise applications.  Does anyone else have any thoughts on this?
 
Regards,
 
Steven
 
Reply | Threaded
Open this post in threaded view
|

RE: Safety of using OpenSSL from multiple components in the one process

Peter Waltenberg

We created new entry points. There's a very shallow layer over the EVP layer which is mostly just a macro away from native OpenSSL.
That was done to satisfy other requirements, abstracting the OpenSSL API's a little so we could isolate IBM apps. from changes in OpenSSL, so we could slide in the FIPS self tests, so we could keep IBM owned code separate from OpenSSL licenced code etc.

I'm not going to claim that it's impossible to solve this in other ways, we just didn't have to try anything else since a solution that mostly worked already was imposed by other considerations in our case.

Our problems have been more with conflicts with other instances of OpenSSL in customer products. IBM libraries are used by customers who integrate them with their code as well.

Peter
Peter Waltenberg



"Steven Reddie" <[hidden email]>
Sent by: [hidden email]

03/08/2005 10:26 PM

Please respond to
openssl-dev

To
<[hidden email]>
cc
Subject
RE: Safety of using OpenSSL from multiple components in the one process





Hi Peter,
 
There is a requirement in my case for the OpenSSL API to be accessible to several components.  In your case was your higher level component the only direct user of OpenSSL?  If not, how did you make the API available to other components?  Were the new functions accessed from one DLL with the OpenSSL functions accessed directly from the other DLL, or did you have redirecting stubs in your new DLL?
 
Steven


From: [hidden email] [mailto:[hidden email]] On Behalf Of Peter Waltenberg
Sent:
Wednesday, 3 August 2005 8:48 PM
To:
[hidden email]
Cc:
[hidden email]
Subject:
Re: Safety of using OpenSSL from multiple components in the one process



You are correct it is a problem in enterprise applications.  

The good news is that it can be resolved, the bad news is that we did that outside the OpenSSL codebase itself.


We created new startup/shutdown functions in another library, added reference counting and load OpenSSL via that.
It might be possible to do something simillar within the existing API's, but brute force did work.


Peter


Peter Waltenberg



"Steven Reddie" <[hidden email]>
Sent by: [hidden email]

03/08/2005 06:42 PM

Please respond to
openssl-dev


To
<[hidden email]>
cc
Subject
Safety of using OpenSSL from multiple components in the one process







Hi All,

 

This is something that I think I've raised before but don't remember getting resolution on.

 

OpenSSL maintains various global variables and structures, and there are cleanup functions that must be used to properly release the resources when finished.  One example is the OID database managed by the "add_all_algorithms" function and it's associated release function, EVP_cleanup.  All is good when the use of OpenSSL is fairly simple, such as a single component using it for the lifetime of the process.

 

Where things get difficult/dangerous is when multiple seperate components in the one process, with no real knowledge of each other, make use of OpenSSL, and it's even worse if they dynamically load and unload OpenSSL using dlopen/LoadLibrary.  With large enterprise applications this is a common situation since different teams develop components that the large product makes us of, and with the increasing use of "plug-in" architectures the dynamic loading/unloading is not uncommon.

 

There seems to be no way offered by the OpenSSL API for these components to behave well.  If they each do a dlopen -> dlsym -> ... -> EVP_cleanup -> dlclose sequence then it seems that they will trample on each other.  If they take the extreme opposite and don't call EVP_cleanup then the process will leak until it falls over.

 

This is a serious issue that I believe impacts the stability and therefore limits the usefulness of OpenSSL in large enterprise applications.  Does anyone else have any thoughts on this?

 

Regards,

 

Steven

 

Reply | Threaded
Open this post in threaded view
|

RE: Safety of using OpenSSL from multiple components in the one process

Steven Reddie
Ah, the namespacing issue we've discussed previously.
 
The module that I manage at CA is a higher-level library that adds a similar abstraction layer as yours, but there's now an internal requirement to make the underlying OpenSSL interface also available to internal developers.  This was originally allowed for Open Source components that use OpenSSL (OpenLDAP, Apache for example) but has been extended to internal code that is already using OpenSSL.  Our higher-level library uses extensive reference counting to provide a robust interface to callers.  We of course do an EVP_cleanup in our xxx_lib_release() function.  However, with developers now also able to get at OpenSSL anyone of them could call EVP_cleanup at anytime.  Mandating that EVP_cleanup not be called other than by my module is ludicrous due to the resource leaks, particularly if our higher-level library isn't also used in the same process.
 
I stongly feel that for OpenSSL to advance to the next level that usability in enterprise applications has to be treated as a first-class requirement.  I don't see how this can be done without breaking the current interface, but hey, that's why the version number is <1.0.  What I'd like to see is:
1. A reference counted startup/shutdown function with all functions that tear-down global structures controlled by these and otherwise made unavailable to callers (ie. EVP_cleanup, CRYPTO_cleanup_all_ex_data, ERR_free_strings, etc.).  The init functions such as xxx_add_all_algorithms() may be left as-is but the associated cleanup functions are dangerous while public.
2. Namespacing supported.  I've implemented this in my own builds largely using the C preprocessor to "#define  RSA_new  CA1_RSA_new" for example with some other changes required in the perl scripts and some assembler modules.  The prefix can be specified at config time so that, in your case Peter, you could use a prefix of IBM_.  A default OpenSSL build continues to have no prefix.  This means that I no longer have to fear some other OpenSSL build finding its way into our process at runtime and subverting our FIPS routines.
 
An alternative to 2 is to move all functions into a structure of functions pointers, a la PKCS#11, and have a single function exported from OpenSSL, eg. OpenSSL_get_functions(), that returns this.  However, I'm guessing that the extra pointer indirection would be considered too heavy handed.  As is evident in the Python world though, trading a few cycles for increased developer productivity, and in this case determinism and stability, is a wise move.
 
I'm happy to contribute the changes, but don't want to do all of the work unless the OpenSSL core developers agree on the solution.
 
Regards,
 
Steven


From: Peter Waltenberg [mailto:[hidden email]]
Sent: Thursday, 4 August 2005 8:09 AM
To: [hidden email]
Cc: [hidden email]
Subject: RE: Safety of using OpenSSL from multiple components in the one process


We created new entry points. There's a very shallow layer over the EVP layer which is mostly just a macro away from native OpenSSL.
That was done to satisfy other requirements, abstracting the OpenSSL API's a little so we could isolate IBM apps. from changes in OpenSSL, so we could slide in the FIPS self tests, so we could keep IBM owned code separate from OpenSSL licenced code etc.

I'm not going to claim that it's impossible to solve this in other ways, we just didn't have to try anything else since a solution that mostly worked already was imposed by other considerations in our case.

Our problems have been more with conflicts with other instances of OpenSSL in customer products. IBM libraries are used by customers who integrate them with their code as well.

Peter
Peter Waltenberg



"Steven Reddie" <[hidden email]>
Sent by: [hidden email]

03/08/2005 10:26 PM

Please respond to
openssl-dev

To
<[hidden email]>
cc
Subject
RE: Safety of using OpenSSL from multiple components in the one process





Hi Peter,
 
There is a requirement in my case for the OpenSSL API to be accessible to several components.  In your case was your higher level component the only direct user of OpenSSL?  If not, how did you make the API available to other components?  Were the new functions accessed from one DLL with the OpenSSL functions accessed directly from the other DLL, or did you have redirecting stubs in your new DLL?
 
Steven


From: [hidden email] [mailto:[hidden email]] On Behalf Of Peter Waltenberg
Sent:
Wednesday, 3 August 2005 8:48 PM
To:
[hidden email]
Cc:
[hidden email]
Subject:
Re: Safety of using OpenSSL from multiple components in the one process



You are correct it is a problem in enterprise applications.  

The good news is that it can be resolved, the bad news is that we did that outside the OpenSSL codebase itself.


We created new startup/shutdown functions in another library, added reference counting and load OpenSSL via that.
It might be possible to do something simillar within the existing API's, but brute force did work.


Peter


Peter Waltenberg



"Steven Reddie" <[hidden email]>
Sent by: [hidden email]

03/08/2005 06:42 PM

Please respond to
openssl-dev


To
<[hidden email]>
cc
Subject
Safety of using OpenSSL from multiple components in the one process







Hi All,

 

This is something that I think I've raised before but don't remember getting resolution on.

 

OpenSSL maintains various global variables and structures, and there are cleanup functions that must be used to properly release the resources when finished.  One example is the OID database managed by the "add_all_algorithms" function and it's associated release function, EVP_cleanup.  All is good when the use of OpenSSL is fairly simple, such as a single component using it for the lifetime of the process.

 

Where things get difficult/dangerous is when multiple seperate components in the one process, with no real knowledge of each other, make use of OpenSSL, and it's even worse if they dynamically load and unload OpenSSL using dlopen/LoadLibrary.  With large enterprise applications this is a common situation since different teams develop components that the large product makes us of, and with the increasing use of "plug-in" architectures the dynamic loading/unloading is not uncommon.

 

There seems to be no way offered by the OpenSSL API for these components to behave well.  If they each do a dlopen -> dlsym -> ... -> EVP_cleanup -> dlclose sequence then it seems that they will trample on each other.  If they take the extreme opposite and don't call EVP_cleanup then the process will leak until it falls over.

 

This is a serious issue that I believe impacts the stability and therefore limits the usefulness of OpenSSL in large enterprise applications.  Does anyone else have any thoughts on this?

 

Regards,

 

Steven

 

Reply | Threaded
Open this post in threaded view
|

RE: Safety of using OpenSSL from multiple components in the one process

Peter Waltenberg

Our problems and the solutions we've come up with appear be be near identical conceptually.

I think if we were going to address those problems, we'd should also try to fix the current visibility of internal OpenSSL data types. I suspect that's more of a problem for the OpenSSL team who have to take extra care to avoid changes which break the ABI between releases.

We'd also be happy to contribute code and effort, but as you've pointed out, the core team will have to want these changes made.  

Peter

Peter Waltenberg



"Steven Reddie" <[hidden email]>

04/08/2005 09:01 AM

To
Peter Waltenberg/Australia/IBM@IBMAU, <[hidden email]>
cc
Subject
RE: Safety of using OpenSSL from multiple components in the one process





Ah, the namespacing issue we've discussed previously.
 
The module that I manage at CA is a higher-level library that adds a similar abstraction layer as yours, but there's now an internal requirement to make the underlying OpenSSL interface also available to internal developers.  This was originally allowed for Open Source components that use OpenSSL (OpenLDAP, Apache for example) but has been extended to internal code that is already using OpenSSL.  Our higher-level library uses extensive reference counting to provide a robust interface to callers.  We of course do an EVP_cleanup in our xxx_lib_release() function.  However, with developers now also able to get at OpenSSL anyone of them could call EVP_cleanup at anytime.  Mandating that EVP_cleanup not be called other than by my module is ludicrous due to the resource leaks, particularly if our higher-level library isn't also used in the same process.
 
I stongly feel that for OpenSSL to advance to the next level that usability in enterprise applications has to be treated as a first-class requirement.  I don't see how this can be done without breaking the current interface, but hey, that's why the version number is <1.0.  What I'd like to see is:
1. A reference counted startup/shutdown function with all functions that tear-down global structures controlled by these and otherwise made unavailable to callers (ie. EVP_cleanup, CRYPTO_cleanup_all_ex_data, ERR_free_strings, etc.).  The init functions such as xxx_add_all_algorithms() may be left as-is but the associated cleanup functions are dangerous while public.
2. Namespacing supported.  I've implemented this in my own builds largely using the C preprocessor to "#define  RSA_new  CA1_RSA_new" for example with some other changes required in the perl scripts and some assembler modules.  The prefix can be specified at config time so that, in your case Peter, you could use a prefix of IBM_.  A default OpenSSL build continues to have no prefix.  This means that I no longer have to fear some other OpenSSL build finding its way into our process at runtime and subverting our FIPS routines.
 
An alternative to 2 is to move all functions into a structure of functions pointers, a la PKCS#11, and have a single function exported from OpenSSL, eg. OpenSSL_get_functions(), that returns this.  However, I'm guessing that the extra pointer indirection would be considered too heavy handed.  As is evident in the Python world though, trading a few cycles for increased developer productivity, and in this case determinism and stability, is a wise move.
 
I'm happy to contribute the changes, but don't want to do all of the work unless the OpenSSL core developers agree on the solution.
 
Regards,
 
Steven


From: Peter Waltenberg [mailto:[hidden email]]
Sent:
Thursday, 4 August 2005 8:09 AM
To:
[hidden email]
Cc:
[hidden email]
Subject:
RE: Safety of using OpenSSL from multiple components in the one process



We created new entry points. There's a very shallow layer over the EVP layer which is mostly just a macro away from native OpenSSL.

That was done to satisfy other requirements, abstracting the OpenSSL API's a little so we could isolate IBM apps. from changes in OpenSSL, so we could slide in the FIPS self tests, so we could keep IBM owned code separate from OpenSSL licenced code etc.


I'm not going to claim that it's impossible to solve this in other ways, we just didn't have to try anything else since a solution that mostly worked already was imposed by other considerations in our case.


Our problems have been more with conflicts with other instances of OpenSSL in customer products. IBM libraries are used by customers who integrate them with their code as well.


Peter

Peter Waltenberg



"Steven Reddie" <[hidden email]>
Sent by: [hidden email]

03/08/2005 10:26 PM

Please respond to
openssl-dev


To
<[hidden email]>
cc
Subject
RE: Safety of using OpenSSL from multiple components in the one process







Hi Peter,

 

There is a requirement in my case for the OpenSSL API to be accessible to several components.  In your case was your higher level component the only direct user of OpenSSL?  If not, how did you make the API available to other components?  Were the new functions accessed from one DLL with the OpenSSL functions accessed directly from the other DLL, or did you have redirecting stubs in your new DLL?

 

Steven



From: [hidden email] [mailto:[hidden email]] On Behalf Of Peter Waltenberg
Sent:
Wednesday, 3 August 2005 8:48 PM
To:
[hidden email]
Cc:
[hidden email]
Subject:
Re: Safety of using OpenSSL from multiple components in the one process



You are correct it is a problem in enterprise applications.  

The good news is that it can be resolved, the bad news is that we did that outside the OpenSSL codebase itself.


We created new startup/shutdown functions in another library, added reference counting and load OpenSSL via that.
It might be possible to do something simillar within the existing API's, but brute force did work.

Peter


Peter Waltenberg


"Steven Reddie" <[hidden email]>
Sent by: [hidden email]

03/08/2005 06:42 PM

Please respond to
openssl-dev


To
<[hidden email]>
cc
Subject
Safety of using OpenSSL from multiple components in the one process









Hi All,


This is something that I think I've raised before but don't remember getting resolution on.


OpenSSL maintains various global variables and structures, and there are cleanup functions that must be used to properly release the resources when finished.  One example is the OID database managed by the "add_all_algorithms" function and it's associated release function, EVP_cleanup.  All is good when the use of OpenSSL is fairly simple, such as a single component using it for the lifetime of the process.


Where things get difficult/dangerous is when multiple seperate components in the one process, with no real knowledge of each other, make use of OpenSSL, and it's even worse if they dynamically load and unload OpenSSL using dlopen/LoadLibrary.  With large enterprise applications this is a common situation since different teams develop components that the large product makes us of, and with the increasing use of "plug-in" architectures the dynamic loading/unloading is not uncommon.


There seems to be no way offered by the OpenSSL API for these components to behave well.  If they each do a dlopen -> dlsym -> ... -> EVP_cleanup -> dlclose sequence then it seems that they will trample on each other.  If they take the extreme opposite and don't call EVP_cleanup then the process will leak until it falls over.


This is a serious issue that I believe impacts the stability and therefore limits the usefulness of OpenSSL in large enterprise applications.  Does anyone else have any thoughts on this?


Regards,


Steven

 

Reply | Threaded
Open this post in threaded view
|

RE: Safety of using OpenSSL from multiple components in the one process

Steven Reddie
In order to correct these things properly I think a break in the ABI is inevitable, and what better time to do it than for a 0.9.9 release.  I can appreciate the reasons for not breaking the ABI, but I see it in this case as moving the ABI to where it needs to be in order to not have to break it after a 1.0 release.
 
In the best case we could be talking about removing only a few functions: EVP_cleanup, etc.
 
There is precedent for breaking the ABI.  I discovered the other day that we had a call to RSA_new_method with an RSA_METH* parameter.  It turns out that the parameter was changed to an ENGINE* in 0.9.7 and we hadn't picked it up.
 
--Steven

From: Peter Waltenberg [mailto:[hidden email]]
Sent: Thursday, 4 August 2005 9:33 AM
To: [hidden email]
Cc: Steven Reddie
Subject: RE: Safety of using OpenSSL from multiple components in the one process


Our problems and the solutions we've come up with appear be be near identical conceptually.

I think if we were going to address those problems, we'd should also try to fix the current visibility of internal OpenSSL data types. I suspect that's more of a problem for the OpenSSL team who have to take extra care to avoid changes which break the ABI between releases.

We'd also be happy to contribute code and effort, but as you've pointed out, the core team will have to want these changes made.  

Peter

Peter Waltenberg



"Steven Reddie" <[hidden email]>

04/08/2005 09:01 AM

To
Peter Waltenberg/Australia/IBM@IBMAU, <[hidden email]>
cc
Subject
RE: Safety of using OpenSSL from multiple components in the one process





Ah, the namespacing issue we've discussed previously.
 
The module that I manage at CA is a higher-level library that adds a similar abstraction layer as yours, but there's now an internal requirement to make the underlying OpenSSL interface also available to internal developers.  This was originally allowed for Open Source components that use OpenSSL (OpenLDAP, Apache for example) but has been extended to internal code that is already using OpenSSL.  Our higher-level library uses extensive reference counting to provide a robust interface to callers.  We of course do an EVP_cleanup in our xxx_lib_release() function.  However, with developers now also able to get at OpenSSL anyone of them could call EVP_cleanup at anytime.  Mandating that EVP_cleanup not be called other than by my module is ludicrous due to the resource leaks, particularly if our higher-level library isn't also used in the same process.
 
I stongly feel that for OpenSSL to advance to the next level that usability in enterprise applications has to be treated as a first-class requirement.  I don't see how this can be done without breaking the current interface, but hey, that's why the version number is <1.0.  What I'd like to see is:
1. A reference counted startup/shutdown function with all functions that tear-down global structures controlled by these and otherwise made unavailable to callers (ie. EVP_cleanup, CRYPTO_cleanup_all_ex_data, ERR_free_strings, etc.).  The init functions such as xxx_add_all_algorithms() may be left as-is but the associated cleanup functions are dangerous while public.
2. Namespacing supported.  I've implemented this in my own builds largely using the C preprocessor to "#define  RSA_new  CA1_RSA_new" for example with some other changes required in the perl scripts and some assembler modules.  The prefix can be specified at config time so that, in your case Peter, you could use a prefix of IBM_.  A default OpenSSL build continues to have no prefix.  This means that I no longer have to fear some other OpenSSL build finding its way into our process at runtime and subverting our FIPS routines.
 
An alternative to 2 is to move all functions into a structure of functions pointers, a la PKCS#11, and have a single function exported from OpenSSL, eg. OpenSSL_get_functions(), that returns this.  However, I'm guessing that the extra pointer indirection would be considered too heavy handed.  As is evident in the Python world though, trading a few cycles for increased developer productivity, and in this case determinism and stability, is a wise move.
 
I'm happy to contribute the changes, but don't want to do all of the work unless the OpenSSL core developers agree on the solution.
 
Regards,
 
Steven


From: Peter Waltenberg [mailto:[hidden email]]
Sent:
Thursday, 4 August 2005 8:09 AM
To:
[hidden email]
Cc:
[hidden email]
Subject:
RE: Safety of using OpenSSL from multiple components in the one process



We created new entry points. There's a very shallow layer over the EVP layer which is mostly just a macro away from native OpenSSL.

That was done to satisfy other requirements, abstracting the OpenSSL API's a little so we could isolate IBM apps. from changes in OpenSSL, so we could slide in the FIPS self tests, so we could keep IBM owned code separate from OpenSSL licenced code etc.


I'm not going to claim that it's impossible to solve this in other ways, we just didn't have to try anything else since a solution that mostly worked already was imposed by other considerations in our case.


Our problems have been more with conflicts with other instances of OpenSSL in customer products. IBM libraries are used by customers who integrate them with their code as well.


Peter

Peter Waltenberg



"Steven Reddie" <[hidden email]>
Sent by: [hidden email]

03/08/2005 10:26 PM

Please respond to
openssl-dev


To
<[hidden email]>
cc
Subject
RE: Safety of using OpenSSL from multiple components in the one process







Hi Peter,

 

There is a requirement in my case for the OpenSSL API to be accessible to several components.  In your case was your higher level component the only direct user of OpenSSL?  If not, how did you make the API available to other components?  Were the new functions accessed from one DLL with the OpenSSL functions accessed directly from the other DLL, or did you have redirecting stubs in your new DLL?

 

Steven



From: [hidden email] [mailto:[hidden email]] On Behalf Of Peter Waltenberg
Sent:
Wednesday, 3 August 2005 8:48 PM
To:
[hidden email]
Cc:
[hidden email]
Subject:
Re: Safety of using OpenSSL from multiple components in the one process



You are correct it is a problem in enterprise applications.  

The good news is that it can be resolved, the bad news is that we did that outside the OpenSSL codebase itself.


We created new startup/shutdown functions in another library, added reference counting and load OpenSSL via that.
It might be possible to do something simillar within the existing API's, but brute force did work.

Peter


Peter Waltenberg


"Steven Reddie" <[hidden email]>
Sent by: [hidden email]

03/08/2005 06:42 PM

Please respond to
openssl-dev


To
<[hidden email]>
cc
Subject
Safety of using OpenSSL from multiple components in the one process









Hi All,


This is something that I think I've raised before but don't remember getting resolution on.


OpenSSL maintains various global variables and structures, and there are cleanup functions that must be used to properly release the resources when finished.  One example is the OID database managed by the "add_all_algorithms" function and it's associated release function, EVP_cleanup.  All is good when the use of OpenSSL is fairly simple, such as a single component using it for the lifetime of the process.


Where things get difficult/dangerous is when multiple seperate components in the one process, with no real knowledge of each other, make use of OpenSSL, and it's even worse if they dynamically load and unload OpenSSL using dlopen/LoadLibrary.  With large enterprise applications this is a common situation since different teams develop components that the large product makes us of, and with the increasing use of "plug-in" architectures the dynamic loading/unloading is not uncommon.


There seems to be no way offered by the OpenSSL API for these components to behave well.  If they each do a dlopen -> dlsym -> ... -> EVP_cleanup -> dlclose sequence then it seems that they will trample on each other.  If they take the extreme opposite and don't call EVP_cleanup then the process will leak until it falls over.


This is a serious issue that I believe impacts the stability and therefore limits the usefulness of OpenSSL in large enterprise applications.  Does anyone else have any thoughts on this?


Regards,


Steven