Investigating a leak

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Investigating a leak

Axel Andersson
Hello,

I'm investigating a leak in a server application, which I've managed  
to reduce to a test case which involves OpenSSL and threads. I've put  
the code up at http://www.zankasoftware.com/ssltest/server.c if  
anyone would be willing to take a look and see what I'm doing wrong.

Every time I connect to this test with openssl s_client, it leaks  
some 516 KB virtual memory, which is never reclaimed. If pthread_join
() is inserted after pthread_create(), no leaks occur. It does not  
matter if I move all the SSL calls to the reply thread, or leave the  
SSL_accept() call in the main thread where it is now.

I'm testing this on Mac OS X with its default OpenSSL 0.9.7b. I also  
downloaded and built the recent 0.9.8 release and was able to  
reproduce the problem with that.

Thanks in advance,

-- Axel


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Investigating a leak

Amar Desai
http://www.openssl.org/support/faq.html

Search for the words *memory leak*

-Amar.

Axel Andersson wrote:

> Hello,
>
> I'm investigating a leak in a server application, which I've managed  
> to reduce to a test case which involves OpenSSL and threads. I've put  
> the code up at http://www.zankasoftware.com/ssltest/server.c if  
> anyone would be willing to take a look and see what I'm doing wrong.
>
> Every time I connect to this test with openssl s_client, it leaks  
> some 516 KB virtual memory, which is never reclaimed. If pthread_join
> () is inserted after pthread_create(), no leaks occur. It does not  
> matter if I move all the SSL calls to the reply thread, or leave the  
> SSL_accept() call in the main thread where it is now.
>
> I'm testing this on Mac OS X with its default OpenSSL 0.9.7b. I also  
> downloaded and built the recent 0.9.8 release and was able to  
> reproduce the problem with that.
>
> Thanks in advance,
>
> -- Axel
>
>
> ______________________________________________________________________
> OpenSSL Project                                 http://www.openssl.org
> User Support Mailing List                    [hidden email]
> Automated List Manager                           [hidden email]
>

smime.p7s (8K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Investigating a leak

Axel Andersson
On Jul 18, 2005, at 14:11, Amar Desai wrote:

> http://www.openssl.org/support/faq.html
>
> Search for the words *memory leak*

Yes, thank you, indeed I have read that, and I am using  
ERR_remove_state() at the end of the thread. Looks to me like the  
other functions should be used before terminating the application,  
not for per-thread cleanup, or did I misunderstand something?

-- Axel

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Investigating a leak

Amar Desai
Yes, You need to call other functions as well.

-Amar

Axel Andersson wrote:

> Yes, thank you, indeed I have read that, and I am using  
> ERR_remove_state() at the end of the thread. Looks to me like the  
> other functions should be used before terminating the application,  
> not for per-thread cleanup, or did I misunderstand something?
>
> -- Axel
>
> ______________________________________________________________________
> OpenSSL Project                                 http://www.openssl.org
> User Support Mailing List                    [hidden email]
> Automated List Manager                           [hidden email]
>

smime.p7s (8K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: Investigating a leak

Simon Edwards
In reply to this post by Axel Andersson
Hi Axel,

I don't know if it helps you much, but since the FAQ is a bit cryptic it may
help. I've used...

ERR_remove_state()

...at the end of each thread.

Plus...

CRYPTO_cleanup_all_ex_data()
CONF_modules_free()
ERR_free_strings()
EVP_cleanup()

... at the end of the application.

I have no leaks that I can detect when using 0.9.7g.

At the time I wrote my code I couldn't find any FAQ entry that worked along
the lines of "if you use function zzzz or function group xxxx, then you must
call yyyy to free underlying data". My app' isn't loading external engines
or doing anything fancy, so these are probably not a comprehensive list of
resource releasing functions.

Regards,

   Simon

- --
Simon Edwards

-----Original Message-----
From: Amar Desai [mailto:[hidden email]]
Sent: 18 July 2005 16:59
To: [hidden email]
Subject: Re: Investigating a leak


Yes, You need to call other functions as well.

-Amar

Axel Andersson wrote:

> Yes, thank you, indeed I have read that, and I am using  
> ERR_remove_state() at the end of the thread. Looks to me like the  
> other functions should be used before terminating the application,  
> not for per-thread cleanup, or did I misunderstand something?
>
> -- Axel
>
> ______________________________________________________________________
> OpenSSL Project                                 http://www.openssl.org
> User Support Mailing List                    [hidden email]
> Automated List Manager                           [hidden email]
>
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Investigating a leak

Axel Andersson
On Jul 18, 2005, at 18:57, Simon Edwards wrote:

> I don't know if it helps you much, but since the FAQ is a bit  
> cryptic it may
> help. I've used...
>
> ERR_remove_state()
>
> ...at the end of each thread.
>

Doing this.

> Plus...
>
> CRYPTO_cleanup_all_ex_data()
> CONF_modules_free()
> ERR_free_strings()
> EVP_cleanup()
>
> ... at the end of the application.
>

Doing this now, but since my app is a server, it never really reaches  
the end, and the leaks build up during its lifetime, which I'd rather  
hoped could be more than a day or so. As it is now, the process'  
virtual memory size quickly reaches 3.5 GB, and then malloc() starts  
failing.

Well, thanks for your help anyway, I appreciate it.

-- Axel


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Investigating a leak

Rich Salz
> Doing this now, but since my app is a server, it never really reaches
> the end, and the leaks build up during its lifetime, which I'd rather
> hoped could be more than a day or so. As it is now, the process'
> virtual memory size quickly reaches 3.5 GB, and then malloc() starts
> failing.

Something wrong is happening.  Lots of folks are running apache/openssl
servers for days, weeks, etc., at a time.

If you're seeing memory leaks like you describe, then it is far more
likely that the like is either in your code, or there's a bug in the way
you're calling OpenSSL.  Do you get the same growth when openssl isn't
used?  Make sure you "xxx_free" every object you "xxx_new".

        /r$

--
Rich Salz                  Chief Security Architect
DataPower Technology       http://www.datapower.com
XS40 XML Security Gateway  http://www.datapower.com/products/xs40.html

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Investigating a leak

Cesc Santa
On 7/19/05, Rich Salz <[hidden email]> wrote:

> > Doing this now, but since my app is a server, it never really reaches
> > the end, and the leaks build up during its lifetime, which I'd rather
> > hoped could be more than a day or so. As it is now, the process'
> > virtual memory size quickly reaches 3.5 GB, and then malloc() starts
> > failing.
>
> Something wrong is happening.  Lots of folks are running apache/openssl
> servers for days, weeks, etc., at a time.
>
> If you're seeing memory leaks like you describe, then it is far more
> likely that the like is either in your code, or there's a bug in the way
> you're calling OpenSSL.  Do you get the same growth when openssl isn't
> used?  Make sure you "xxx_free" every object you "xxx_new".
>
>         /r$
>

Hi,

I was looking at the code ... could it be that the memory leak is in
the main server loop?
Sometimes, the ssl_accept may return 0, but with an error code like
SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE, which just means that
could not complete the handshake, but not really an error (just call
ssl_accept again ... ).
But in you main server thread, you just do a continue, thus looping
again ... and creating a brand new ssl structure and socket ... you
should
loop() {
  create the tcp socket
  loop () {
      ssl_accept from the tcp socket till non-zero
   }
}
     
Take a look at the examples that come with openssl.

Regards,

Cesc
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Investigating a leak

Simon Edwards
In reply to this post by Axel Andersson
That'd do it. But if you're doing the loop in that sequence, and if you have
set non-blocking on the sockets, then instead of polling for the connection
you can use select or poll (depending on your platform) to wait for incoming
connection activity, then enter ssl_accept in blocking mode which would
probably cut down on those fake errors. Wouldn't this be a better sequence?

If you are worried about inserting another call in the sequence... Being in
blocking mode at this stage shouldn't adversely affect performance as the
connection thread would only be in a polling loop during this period anyway
in this code example. Being in a time-limited wait during select only aids
performance as it allows the client serving threads access to the time-slice
for the process which would otherwise be tied up in the polling loop. The
new select() call could adversely impact performance if you have to service
a continual stream of new client connections that  prevent the select() from
blocking at all, but that is unlikely in the real world.

Regards,

   Simon

- --
Simon Edwards



-----Original Message-----
From: Cesc [mailto:[hidden email]]
Sent: 19 July 2005 09:58
To: [hidden email]
Subject: Re: Investigating a leak


On 7/19/05, Rich Salz <[hidden email]> wrote:

> > Doing this now, but since my app is a server, it never really reaches
> > the end, and the leaks build up during its lifetime, which I'd rather
> > hoped could be more than a day or so. As it is now, the process'
> > virtual memory size quickly reaches 3.5 GB, and then malloc() starts
> > failing.
>
> Something wrong is happening.  Lots of folks are running apache/openssl
> servers for days, weeks, etc., at a time.
>
> If you're seeing memory leaks like you describe, then it is far more
> likely that the like is either in your code, or there's a bug in the way
> you're calling OpenSSL.  Do you get the same growth when openssl isn't
> used?  Make sure you "xxx_free" every object you "xxx_new".
>
>         /r$
>

Hi,

I was looking at the code ... could it be that the memory leak is in
the main server loop?
Sometimes, the ssl_accept may return 0, but with an error code like
SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE, which just means that
could not complete the handshake, but not really an error (just call
ssl_accept again ... ).
But in you main server thread, you just do a continue, thus looping
again ... and creating a brand new ssl structure and socket ... you
should
loop() {
  create the tcp socket
  loop () {
      ssl_accept from the tcp socket till non-zero
   }
}
     
Take a look at the examples that come with openssl.

Regards,

Cesc
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]


This message has been scanned for viruses by MailController -
www.MailController.altohiway.com
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Investigating a leak

Cesc Santa
Why don't you move the ssl handshake to the thread code?
Let the main loop accept tcp sockets, then create the thread with the
tcp socket as the argument you pass to the thread run method ...
perform then the ssl_accept loop (blocking or non-blocking, up to you)
in the thread ...

Just a suggestion.

Cesc

On 7/19/05, Simon Edwards <[hidden email]> wrote:

> That'd do it. But if you're doing the loop in that sequence, and if you have
> set non-blocking on the sockets, then instead of polling for the connection
> you can use select or poll (depending on your platform) to wait for incoming
> connection activity, then enter ssl_accept in blocking mode which would
> probably cut down on those fake errors. Wouldn't this be a better sequence?
>
> If you are worried about inserting another call in the sequence... Being in
> blocking mode at this stage shouldn't adversely affect performance as the
> connection thread would only be in a polling loop during this period anyway
> in this code example. Being in a time-limited wait during select only aids
> performance as it allows the client serving threads access to the time-slice
> for the process which would otherwise be tied up in the polling loop. The
> new select() call could adversely impact performance if you have to service
> a continual stream of new client connections that  prevent the select() from
> blocking at all, but that is unlikely in the real world.
>
> Regards,
>
>    Simon
>
> - --
> Simon Edwards
>
>
>
> -----Original Message-----
> From: Cesc [mailto:[hidden email]]
> Sent: 19 July 2005 09:58
> To: [hidden email]
> Subject: Re: Investigating a leak
>
>
> On 7/19/05, Rich Salz <[hidden email]> wrote:
> > > Doing this now, but since my app is a server, it never really reaches
> > > the end, and the leaks build up during its lifetime, which I'd rather
> > > hoped could be more than a day or so. As it is now, the process'
> > > virtual memory size quickly reaches 3.5 GB, and then malloc() starts
> > > failing.
> >
> > Something wrong is happening.  Lots of folks are running apache/openssl
> > servers for days, weeks, etc., at a time.
> >
> > If you're seeing memory leaks like you describe, then it is far more
> > likely that the like is either in your code, or there's a bug in the way
> > you're calling OpenSSL.  Do you get the same growth when openssl isn't
> > used?  Make sure you "xxx_free" every object you "xxx_new".
> >
> >         /r$
> >
>
> Hi,
>
> I was looking at the code ... could it be that the memory leak is in
> the main server loop?
> Sometimes, the ssl_accept may return 0, but with an error code like
> SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE, which just means that
> could not complete the handshake, but not really an error (just call
> ssl_accept again ... ).
> But in you main server thread, you just do a continue, thus looping
> again ... and creating a brand new ssl structure and socket ... you
> should
> loop() {
>   create the tcp socket
>   loop () {
>       ssl_accept from the tcp socket till non-zero
>    }
> }
>
> Take a look at the examples that come with openssl.
>
> Regards,
>
> Cesc
> ______________________________________________________________________
> OpenSSL Project                                 http://www.openssl.org
> User Support Mailing List                    [hidden email]
> Automated List Manager                           [hidden email]
>
>
> This message has been scanned for viruses by MailController -
> www.MailController.altohiway.com
> ______________________________________________________________________
> OpenSSL Project                                 http://www.openssl.org
> User Support Mailing List                    [hidden email]
> Automated List Manager                           [hidden email]
>
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Investigating a leak

JoelKatz
In reply to this post by Simon Edwards

> That'd do it. But if you're doing the loop in that sequence, and
> if you have
> set non-blocking on the sockets, then instead of polling for the
> connection
> you can use select or poll (depending on your platform) to wait
> for incoming
> connection activity, then enter ssl_accept in blocking mode which would
> probably cut down on those fake errors. Wouldn't this be a better
> sequence?

        Eww, no. Suppose the connection is aborted before you manage to call
SSL_accept. The call could block indefinitely. It is almost always a mistake
to use 'select' or 'poll' with blocking socket operations.

        DS


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]