where is the memory being held

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

where is the memory being held

zhu qun-ying
Hi,

I have an SSL apllication, that it suppose to run for a long time. After some time of running, I found the usage of the memory is growing.  I stop all SSL connections and checked all SSL * has been freed  but it could not release the memory back to the system.

After some investigation, I found there is no memory leak, but seems lot of memory are unable to release back to system.  mtrace found out there are quite a lot of fragmented memory being held by the SSL library.  I would like to know what could I do to reduce the memory held by SSL library after all connections have been dropped?

I am handling the SSL session through share memory myself and that part of the memory is allocated from the start.

mallinfo() reports after some test and no connection for a while:

system bytes     =     28271952
in use bytes     =      1809184
non-inuse bytes  =     26462768
non-inuse chunks =           81
mmap regions     =            4
mmap bytes       =      1773568
Total (incl. mmap):
system bytes     =     30045520
in use bytes     =      3582752
releasable bytes =       462496

--
qun-ying


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: where is the memory being held

JoelKatz
On 9/23/2010 11:42 AM, zhu qun-ying wrote:

> Hi,
>
> I have an SSL apllication, that it suppose to run for a long time. After some time of running, I found the usage of the memory is growing.  I stop all SSL connections and checked all SSL * has been freed  but it could not release the memory back to the system.
>
> After some investigation, I found there is no memory leak, but seems lot of memory are unable to release back to system.  mtrace found out there are quite a lot of fragmented memory being held by the SSL library.  I would like to know what could I do to reduce the memory held by SSL library after all connections have been dropped?
>
> I am handling the SSL session through share memory myself and that part of the memory is allocated from the start.
>
> mallinfo() reports after some test and no connection for a while:
>
> system bytes     =     28271952
> in use bytes     =      1809184
> non-inuse bytes  =     26462768
> non-inuse chunks =           81
> mmap regions     =            4
> mmap bytes       =      1773568
> Total (incl. mmap):
> system bytes     =     30045520
> in use bytes     =      3582752
> releasable bytes =       462496
>
> --
> qun-ying

This all seems normal. Virtual memory is not normally considered a
scarce resource and unless the consumption is really absurd, it's not
worth worrying about.

Unless your virtual memory use grows linearly with constant load, it's
generally not worth worrying about. If it grows in an exponentially
decreasing way with constant load or grows linearly with increasing peak
load, I wouldn't worry about it at all.

DS

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: where is the memory being held

zhu qun-ying
Hi,

I think I should clarify something here.  The app is running in a small device that does not have virtual memory (no swap space) and the memory is limited (256/512 M).  In peek connections, it may use up to 90% of the system memory, and when connection goes down, memory usage is not coming down.  This leave very little memory for other part of the system, as this app is only a small part of a bigger system. The memory usage is a big concern as it is always running with the box.

So far periodically restart the app is not a good solution.

--
qun-ying

> This all seems normal. Virtual memory is not normally
> considered a
> scarce resource and unless the consumption is really
> absurd, it's not
> worth worrying about.
>
> Unless your virtual memory use grows linearly with constant
> load, it's
> generally not worth worrying about. If it grows in an
> exponentially
> decreasing way with constant load or grows linearly with
> increasing peak
> load, I wouldn't worry about it at all.
>
> DS
>
> ______________________________________________________________________
> OpenSSL Project           
>                
>      http://www.openssl.org
> User Support Mailing List         
>           [hidden email]
> Automated List Manager         
>              
>    [hidden email]
>


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: where is the memory being held

Michael S. Zick-4
On Fri September 24 2010, zhu qun-ying wrote:
> Hi,
>
> I think I should clarify something here.  The app is running in a small device that does not have virtual memory (no swap space) and the memory is limited (256/512 M).  In peek connections, it may use up to 90% of the system memory, and when connection goes down, memory usage is not coming down.  This leave very little memory for other part of the system, as this app is only a small part of a bigger system. The memory usage is a big concern as it is always running with the box.
>

Perhaps you should consider a library intended for use on memory constrained devices?
Dropbear is one of the decent choices.  There are others.

Mike

> So far periodically restart the app is not a good solution.
>
> --
> qun-ying
>
> > This all seems normal. Virtual memory is not normally
> > considered a
> > scarce resource and unless the consumption is really
> > absurd, it's not
> > worth worrying about.
> >
> > Unless your virtual memory use grows linearly with constant
> > load, it's
> > generally not worth worrying about. If it grows in an
> > exponentially
> > decreasing way with constant load or grows linearly with
> > increasing peak
> > load, I wouldn't worry about it at all.
> >
> > DS
> >
> > ______________________________________________________________________
> > OpenSSL Project           
> >                
> >      http://www.openssl.org
> > User Support Mailing List         
> >           [hidden email]
> > Automated List Manager         
> >              
> >    [hidden email]
> >
>
>
> ______________________________________________________________________
> OpenSSL Project                                 http://www.openssl.org
> User Support Mailing List                    [hidden email]
> Automated List Manager                           [hidden email]
>
>


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: where is the memory being held

JoelKatz
In reply to this post by zhu qun-ying
On 9/24/2010 11:05 AM, zhu qun-ying wrote:

> I think I should clarify something here.  The app is running
 > in a small device that does not have virtual memory
> (no swap space) and the memory is limited (256/512 M).
 > In peek connections, it may use up to 90% of the system memory,
 > and when connection goes down, memory usage is not coming down.
 > This leave very little memory for other part of the system,
 > as this app is only a small part of a bigger system. The memory
 > usage is a big concern as it is always running with the box.

> So far periodically restart the app is not a good solution.

Sounds like OpenSSL wasn't what you wanted. OpenSSL is intended for use
on general-purpose computers with virtual memory. It is not designed to
return virtual memory to the system, which in your case means it won't
return physical memory to the system. Ouch.

DS

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: where is the memory being held

zhu qun-ying
Does it mean that it is hard to change the behavior?
--
qun-ying


--- On Fri, 9/24/10, David Schwartz <[hidden email]> wrote:
>
> Sounds like OpenSSL wasn't what you wanted. OpenSSL is
> intended for use on general-purpose computers with virtual
> memory. It is not designed to return virtual memory to the
> system, which in your case means it won't return physical
> memory to the system. Ouch.
>
> DS
>



______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: where is the memory being held

JoelKatz
On 9/26/2010 11:14 PM, zhu qun-ying wrote:

> Does it mean that it is hard to change the behavior?

Yes, because it's not implemented in any one particular place. It's a
fundamental design assumption throughout OpenSSL that it's aimed at
general-purpose computers with virtual memory subsystems.

DS

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: where is the memory being held

Scott Neugroschl-2
In reply to this post by zhu qun-ying
Re: where is the memory being held
As David said, yes.
 
On the other hand, you could re-implement malloc() and free() for your platform.


From: [hidden email] on behalf of zhu qun-ying
Sent: Sun 9/26/2010 11:14 PM
To: [hidden email]
Subject: Re: where is the memory being held

Does it mean that it is hard to change the behavior?
--
qun-ying


--- On Fri, 9/24/10, David Schwartz <[hidden email]> wrote:
>
> Sounds like OpenSSL wasn't what you wanted. OpenSSL is
> intended for use on general-purpose computers with virtual
> memory. It is not designed to return virtual memory to the
> system, which in your case means it won't return physical
> memory to the system. Ouch.
>
> DS
>



______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: where is the memory being held

JoelKatz
On 9/27/2010 4:13 PM, Scott Neugroschl wrote:
> As David said, yes.
> On the other hand, you could re-implement malloc() and free() for your
> platform.

There's really no way to make that help very much. It might help a
little, but the fundamental problem is this:

If you want to implement each 'malloc' so that a later 'free' can return
the memory to the operating system, you can. But that requires rounding
up even small allocations to at least a page, which increases your
memory footprint.

If you don't implement each 'malloc' that way, you still wind up with
the problem that one small allocation that has not been freed in the
middle of a bunch of larger allocations that have been freed prevents
you from returning any of the memory used by the larger allocations to
the operating system.

Generally, what you need are algorithms designed for low memory
footprint and a way to 'group' allocations that will tend to be freed as
a unit (such as those related to a single SSL session) such that when
they are all freed, the memory can be returned to the OS. OpenSSL simply
is not designed this way.

You could probably hack OpenSSL to pass a pointer to a session object to
calls to malloc/free (perhaps using TSD) and use that TSD pointer as an
allocation context. That might increase the chances that the whole
allocation context is freed. It may even be sufficient (or at least
helpful) just to hook all OpenSSL calls to malloc/free and process them
from their own arena.

DS

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: where is the memory being held

zhu qun-ying
I found a solution without hacking into the library itself. Since my system is
running glibc, I forced all mem request to use mmap with
mallopt(M_MMAP_THRESHOLD, 0), which release the memory back to the system when
free is called, testing shows so far so good.

 --
qun-ying



----- Original Message ----

> From: David Schwartz <[hidden email]>
> To: [hidden email]
> Cc: Scott Neugroschl <[hidden email]>
> Sent: Tue, September 28, 2010 3:08:48 PM
> Subject: Re: where is the memory being held
>
> On 9/27/2010 4:13 PM, Scott Neugroschl wrote:
> > As David said,  yes.
> > On the other hand, you could re-implement malloc() and free() for  your
> > platform.
>
> There's really no way to make that help very much.  It might help a little, but
>the fundamental problem is this:
>
> If you want  to implement each 'malloc' so that a later 'free' can return the
>memory to the  operating system, you can. But that requires rounding up even
>small allocations  to at least a page, which increases your memory footprint.
>
> If you don't  implement each 'malloc' that way, you still wind up with the
>problem that one  small allocation that has not been freed in the middle of a
>bunch of larger  allocations that have been freed prevents you from returning
>any of the memory  used by the larger allocations to the operating system.
>
> Generally, what  you need are algorithms designed for low memory footprint and
>a way to 'group'  allocations that will tend to be freed as a unit (such as
>those related to a  single SSL session) such that when they are all freed, the
>memory can be  returned to the OS. OpenSSL simply is not designed this way.
>
> You could  probably hack OpenSSL to pass a pointer to a session object to calls
>to  malloc/free (perhaps using TSD) and use that TSD pointer as an allocation  
>context. That might increase the chances that the whole allocation context is  
>freed. It may even be sufficient (or at least helpful) just to hook all OpenSSL  
>calls to malloc/free and process them from their own  arena.
>
> DS
>
> ______________________________________________________________________
> OpenSSL  Project                                 http://www.openssl.org
> User Support Mailing List                     [hidden email]
> Automated  List Manager                           [hidden email]
>


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]