Thread safe callbacks never actually called

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Thread safe callbacks never actually called

Joshua Miller
Hello OpenSSL,

I am attempting to encrypt a data stream with multiple threads calling EVP_CipherUpdate.  I have set the thread_id and call_back functions as defined by the API.  These functions are never being called, however.  I've used CRYPTO_get_locking_callback() to verify that my function has been set correctly.

Everything works correctly when the threads are serialized, but not when created concurrently.  The main goal is to run EVP_CipherUpdate() on a large amount of data on many threads.  

It seems to me that this should be working, can you tell me what I have done wrong?

Thanks,
Joshua Miller______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Thread safe callbacks never actually called

Ludwig O'Hallorans
Hi:

I'm using CSMTP code, witch is based on OpenSSL, to send email from our software to SMTP servers. And everything works fine if the concurrency is low but as soon ad more than 3 thread start sending at the same time, after a while the module crash and for what I see it has something to do with a crash in crypto_free.
I've seen some post around the internet about this and for what I've read the issue should be solve.
I'm replaying here because I think it could have something to do with what is happening to us here.

Regards,


Ludwig O'Hallorans
Software Developer

Phone 1.786.454.8472
Fax 1.786.363.1784
email: [hidden email]

www.magaya.com

This email message and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the sender via returned email. Please note that any views or opinions expressed in this email are solely those of the author and do not necessarily represent those of the company. Although Magaya Corporation operates anti-virus programs, it does not accept responsibility for any damage whatsoever that is caused by viruses being passed. Please consider the environment before printing this email.


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Joshua Miller
Sent: Tuesday, August 13, 2013 12:45 PM
To: [hidden email]
Subject: Thread safe callbacks never actually called

Hello OpenSSL,

I am attempting to encrypt a data stream with multiple threads calling EVP_CipherUpdate.  I have set the thread_id and call_back functions as defined by the API.  These functions are never being called, however.  I've used CRYPTO_get_locking_callback() to verify that my function has been set correctly.

Everything works correctly when the threads are serialized, but not when created concurrently.  The main goal is to run EVP_CipherUpdate() on a large amount of data on many threads.  

It seems to me that this should be working, can you tell me what I have done wrong?

Thanks,
Joshua Miller______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Thread safe callbacks never actually called

Dr. Stephen Henson
In reply to this post by Joshua Miller
On Tue, Aug 13, 2013, Joshua Miller wrote:

> Hello OpenSSL,
>
> I am attempting to encrypt a data stream with multiple threads calling
> EVP_CipherUpdate.  I have set the thread_id and call_back functions as
> defined by the API.  These functions are never being called, however.  I've
> used CRYPTO_get_locking_callback() to verify that my function has been set
> correctly.
>
> Everything works correctly when the threads are serialized, but not when
> created concurrently.  The main goal is to run EVP_CipherUpdate() on a large
> amount of data on many threads.  
>
> It seems to me that this should be working, can you tell me what I have done
> wrong?
>

It depends on what you are trying to do. If you've several threads using the
same EVP_CIPHER_CTX that wont work in general as partial blocks get saved in
internal buffers and one thread could overwrite a partial block from another.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Thread safe callbacks never actually called

Ludwig O'Hallorans
Excuse my opinion...

But I think the structures should be thread safe as the functions
In my case as you can read in my previous reply I'm loading a dll multiple times and something similar is happening.
If I deploy more that 3 concurrent threads (each one call the dll) the system crash.

Regards,

Ludwig O'Hallorans
Software Developer

Phone 1.786.454.8472
Fax 1.786.363.1784
email: [hidden email]

www.magaya.com

This email message and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the sender via returned email. Please note that any views or opinions expressed in this email are solely those of the author and do not necessarily represent those of the company. Although Magaya Corporation operates anti-virus programs, it does not accept responsibility for any damage whatsoever that is caused by viruses being passed. Please consider the environment before printing this email.


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Dr. Stephen Henson
Sent: Thursday, August 15, 2013 12:15 PM
To: [hidden email]
Subject: Re: Thread safe callbacks never actually called

On Tue, Aug 13, 2013, Joshua Miller wrote:

> Hello OpenSSL,
>
> I am attempting to encrypt a data stream with multiple threads calling
> EVP_CipherUpdate.  I have set the thread_id and call_back functions as
> defined by the API.  These functions are never being called, however.  
> I've used CRYPTO_get_locking_callback() to verify that my function has
> been set correctly.
>
> Everything works correctly when the threads are serialized, but not
> when created concurrently.  The main goal is to run EVP_CipherUpdate()
> on a large amount of data on many threads.
>
> It seems to me that this should be working, can you tell me what I
> have done wrong?
>

It depends on what you are trying to do. If you've several threads using the same EVP_CIPHER_CTX that wont work in general as partial blocks get saved in internal buffers and one thread could overwrite a partial block from another.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org ______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Thread safe callbacks never actually called

Salz, Rich
> But I think the structures should be thread safe as the functions

Then where and how do you propose to store the state of any ongoing computation?

--  
Principal Security Engineer
Akamai Technology
Cambridge, MA

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Thread safe callbacks never actually called

Ludwig O'Hallorans
Thanks for the reply, some time ago I wrote asking for help and nobody answer me.

There should be a way.
There are syncronization method to keep the same structure used by many threads at the same time, and ussually this is transaparent to developers.

In my case looks like some crypto structure is been destroyed twice.
 
Regards,

Ludwig O'Hallorans
Software Developer

Phone 1.786.454.8472
Fax 1.786.363.1784
email: [hidden email]

www.magaya.com

This email message and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the sender via returned email. Please note that any views or opinions expressed in this email are solely those of the author and do not necessarily represent those of the company. Although Magaya Corporation operates anti-virus programs, it does not accept responsibility for any damage whatsoever that is caused by viruses being passed. Please consider the environment before printing this email.


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Salz, Rich
Sent: Thursday, August 15, 2013 2:24 PM
To: [hidden email]
Subject: RE: Thread safe callbacks never actually called

> But I think the structures should be thread safe as the functions

Then where and how do you propose to store the state of any ongoing computation?

--
Principal Security Engineer
Akamai Technology
Cambridge, MA

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Thread safe callbacks never actually called

Salz, Rich
> There should be a way.

There isn't.

> There are syncronization method to keep the same structure used by many threads at the same time, and ussually this is transaparent to developers.

Are you new to multi-threaded C programming?


--  
Principal Security Engineer
Akamai Technology
Cambridge, MA
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Thread safe callbacks never actually called

Ludwig O'Hallorans
I don't mean the type, I mean the data
And not, I di mutythread in C++ for a while now, and work fine by the way
This is the first time I use OpenSSL though.

Any way in my case It shpuldn't be the same data because each time the dll is call, it should have a different set of data, and how can you explain the crash in crypto_free?
For what I read this was something that shoild be patched for a while already.

Regards,

 

Ludwig O'Hallorans
Software Developer

Phone 1.786.454.8472
Fax 1.786.363.1784
email: [hidden email]

www.magaya.com

This email message and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the sender via returned email. Please note that any views or opinions expressed in this email are solely those of the author and do not necessarily represent those of the company. Although Magaya Corporation operates anti-virus programs, it does not accept responsibility for any damage whatsoever that is caused by viruses being passed. Please consider the environment before printing this email.


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Salz, Rich
Sent: Thursday, August 15, 2013 2:39 PM
To: [hidden email]
Subject: RE: Thread safe callbacks never actually called

> There should be a way.

There isn't.

> There are syncronization method to keep the same structure used by many threads at the same time, and ussually this is transaparent to developers.

Are you new to multi-threaded C programming?


--
Principal Security Engineer
Akamai Technology
Cambridge, MA
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Thread safe callbacks never actually called

Salz, Rich
> I don't mean the type, I mean the data

By putting EVERYTHING with __declspec(thread)?  That's not right either, as it completely prevents sharing.  And the Windows DLL malloc model isn't the same Unix/Linux.

Enough pedanticism.  Most objects aren't safe to be used by multiple threads at the same time; they're not designed for that kind of simultaneous use model.  Almost anything that has "context" in it is generally intended to hold the state for something and be uniquely used.  (SSL and SSL_CTX are notable counter-examples.)

        /r$

--  
Principal Security Engineer
Akamai Technology
Cambridge, MA

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Thread safe callbacks never actually called

J. J. Farrell-2
In reply to this post by Ludwig O'Hallorans
A crash in crypto_free most likely means that some code outside the OpenSSL library has corrupted the heap, perhaps by freeing an area more than once or simply scribbling over its control data. One of the usual memory allocation debugging tools should be able to help you pin down the guilty party.

Regards,
               jjf


> From: Ludwig O'Hallorans [mailto:[hidden email]]
> Sent: Thursday, August 15, 2013 7:45 PM
>
> I don't mean the type, I mean the data
> And not, I di mutythread in C++ for a while now, and work fine by the
> way
> This is the first time I use OpenSSL though.
>
> Any way in my case It shpuldn't be the same data because each time the
> dll is call, it should have a different set of data, and how can you
> explain the crash in crypto_free?
> For what I read this was something that shoild be patched for a while
> already.
>
> Regards,
>
>
>
> Ludwig O'Hallorans
> Software Developer
>
> Phone 1.786.454.8472
> Fax 1.786.363.1784
> email: [hidden email]
>
> www.magaya.com
>
> This email message and any files transmitted with it are confidential
> and intended solely for the use of the individual or entity to whom
> they are addressed. If you have received this email in error, please
> notify the sender via returned email. Please note that any views or
> opinions expressed in this email are solely those of the author and do
> not necessarily represent those of the company. Although Magaya
> Corporation operates anti-virus programs, it does not accept
> responsibility for any damage whatsoever that is caused by viruses
> being passed. Please consider the environment before printing this
> email.
>
>
> -----Original Message-----
> From: [hidden email] [mailto:owner-openssl-
> [hidden email]] On Behalf Of Salz, Rich
> Sent: Thursday, August 15, 2013 2:39 PM
>
> > There should be a way.
>
> There isn't.
>
> > There are syncronization method to keep the same structure used by
> many threads at the same time, and ussually this is transaparent to
> developers.
>
> Are you new to multi-threaded C programming?
>
>
> --
> Principal Security Engineer
> Akamai Technology
> Cambridge, MA
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Thread safe callbacks never actually called

Ludwig O'Hallorans
Thank you.

I've been able to pointing it to a call to WSACleanup();
After a call to CleanupOpenSSL();

void CSmtp::CleanupOpenSSL()
{
        if(m_ssl != NULL)
        {
                SSL_shutdown (m_ssl);  /* send SSL/TLS close_notify */
                SSL_free (m_ssl);

                m_ssl = NULL;
        }

        if(m_ctx != NULL)
        {
                SSL_CTX_free (m_ctx);
                m_ctx = NULL;

                ERR_free_strings();
                EVP_cleanup();
                CRYPTO_cleanup_all_ex_data();
        }
}

What is curious here is that is able to run fine for a while an then it crash.
Some times take 2 minutes, sometime hours.
Look like it depends on the speed the SMTP server can proces request.
We have a client with a really slow server that made crash in less than 10 minutes always.
But I manage to send to gmail (with same TLS) with no crash for a very long time.
I'm using CSMTP class from http://www.codeproject.com/Articles/98355/SMTP-Client-with-SSL-TLS
As base for what I've done and found no problem until that slow server came into the scene.

Any ideas would be appreciated.

Thanks,
 

Ludwig O'Hallorans
Software Developer

Phone 1.786.454.8472
Fax 1.786.363.1784
email: [hidden email]

www.magaya.com

This email message and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the sender via returned email. Please note that any views or opinions expressed in this email are solely those of the author and do not necessarily represent those of the company. Although Magaya Corporation operates anti-virus programs, it does not accept responsibility for any damage whatsoever that is caused by viruses being passed. Please consider the environment before printing this email.


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Jeremy Farrell
Sent: Thursday, August 15, 2013 3:24 PM
To: [hidden email]
Subject: RE: Thread safe callbacks never actually called

A crash in crypto_free most likely means that some code outside the OpenSSL library has corrupted the heap, perhaps by freeing an area more than once or simply scribbling over its control data. One of the usual memory allocation debugging tools should be able to help you pin down the guilty party.

Regards,
               jjf


> From: Ludwig O'Hallorans [mailto:[hidden email]]
> Sent: Thursday, August 15, 2013 7:45 PM
>
> I don't mean the type, I mean the data And not, I di mutythread in C++
> for a while now, and work fine by the way This is the first time I use
> OpenSSL though.
>
> Any way in my case It shpuldn't be the same data because each time the
> dll is call, it should have a different set of data, and how can you
> explain the crash in crypto_free?
> For what I read this was something that shoild be patched for a while
> already.
>
> Regards,
>
>
>
> Ludwig O'Hallorans
> Software Developer
>
> Phone 1.786.454.8472
> Fax 1.786.363.1784
> email: [hidden email]
>
> www.magaya.com
>
> This email message and any files transmitted with it are confidential
> and intended solely for the use of the individual or entity to whom
> they are addressed. If you have received this email in error, please
> notify the sender via returned email. Please note that any views or
> opinions expressed in this email are solely those of the author and do
> not necessarily represent those of the company. Although Magaya
> Corporation operates anti-virus programs, it does not accept
> responsibility for any damage whatsoever that is caused by viruses
> being passed. Please consider the environment before printing this
> email.
>
>
> -----Original Message-----
> From: [hidden email] [mailto:owner-openssl-
> [hidden email]] On Behalf Of Salz, Rich
> Sent: Thursday, August 15, 2013 2:39 PM
>
> > There should be a way.
>
> There isn't.
>
> > There are syncronization method to keep the same structure used by
> many threads at the same time, and ussually this is transaparent to
> developers.
>
> Are you new to multi-threaded C programming?
>
>
> --
> Principal Security Engineer
> Akamai Technology
> Cambridge, MA
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]