Small memory leak on multithreaded server

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

Small memory leak on multithreaded server

Barbe, Charles
Hello,

I have noticed a small and consistent memory leak in my multithreaded openssl server and am wondering if somebody can help me figure out what I need to do to free it when my application closes. I am on OpenSSL version 1.0.1j. Here's how I reproduce the leak:

1) Start up my server
2) Load my homepage with Chrome
3) Load my homepage with IE
4) Load my homepage with Firefox

I can do any combination of steps 2,3 and 4 above (ie. leave some of them out) and I always get the same amount of memory left over after I shut down my application. I believe this means that this is some sort of global information that OpenSSL is hanging on to and not something in my SSL connection structure.

Specifically I get 20 blocks totaling 253 bytes. I have stack traces of where each block is allocated but I cannot figure out how this memory should be cleaned up. Each of the 20 blocks filter down to 1 of 5 root stack traces. The stack traces are:

Repeated 6 times:
0x085c519c: asn1_item_ex_combine_new at tasn_new.c:164
0x085c7fd8: ASN1_item_ex_d2i at tasn_dec.c:315 (discriminator 1)
0x085c81dc: asn1_template_noexp_d2i at tasn_dec.c:706
0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
0x08683698: x509v3_cache_extensions at v3_purp.c:722
 (inlined by) X509_check_issued at v3_purp.c:706
0x0866fddc: check_issued at x509_vfy.c:417
0x086716d0: X509_verify_cert at x509_vfy.c:212
0x086967c0: ssl3_output_cert_chain at s3_both.c:387
0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
0x0868b864: ssl3_accept at s3_srvr.c:418
0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
0x08698bb8: ssl23_accept at s23_srvr.c:213

Repeated 6 times:
0x085d12a4: ASN1_STRING_set at asn1_lib.c:389
0x085c715c: asn1_ex_c2i at tasn_dec.c:1056
0x085c7334: asn1_d2i_ex_primitive at tasn_dec.c:836
0x085c7fc8: ASN1_item_ex_d2i at tasn_dec.c:198
0x085c82a0: asn1_template_noexp_d2i at tasn_dec.c:734
0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
0x085c7a20: ASN1_item_ex_d2i at tasn_dec.c:331
0x085c81dc: asn1_template_noexp_d2i at tasn_dec.c:706
0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
0x08683698: x509v3_cache_extensions at v3_purp.c:722
 (inlined by) X509_check_issued at v3_purp.c:706
0x0866fddc: check_issued at x509_vfy.c:417
0x086716d0: X509_verify_cert at x509_vfy.c:212
0x086967c0: ssl3_output_cert_chain at s3_both.c:387
0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
0x0868b864: ssl3_accept at s3_srvr.c:418
0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
0x08698bb8: ssl23_accept at s23_srvr.c:213

Repeated 6 times:
0x085d13bc: ASN1_STRING_type_new at asn1_lib.c:429
0x085c71f4: asn1_ex_c2i at tasn_dec.c:1032
0x085c7334: asn1_d2i_ex_primitive at tasn_dec.c:836
0x085c7fc8: ASN1_item_ex_d2i at tasn_dec.c:198
0x085c82a0: asn1_template_noexp_d2i at tasn_dec.c:734
0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
0x085c7a20: ASN1_item_ex_d2i at tasn_dec.c:331
0x085c81dc: asn1_template_noexp_d2i at tasn_dec.c:706
0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
0x08683698: x509v3_cache_extensions at v3_purp.c:722
 (inlined by) X509_check_issued at v3_purp.c:706
0x0866fddc: check_issued at x509_vfy.c:417
0x086716d0: X509_verify_cert at x509_vfy.c:212
0x086967c0: ssl3_output_cert_chain at s3_both.c:387
0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
0x0868b864: ssl3_accept at s3_srvr.c:418
0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
0x08698bb8: ssl23_accept at s23_srvr.c:213

Once:
0x0866380c: sk_new at stack.c:125
0x0866380c: sk_new at stack.c:125
0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
0x08683698: x509v3_cache_extensions at v3_purp.c:722
 (inlined by) X509_check_issued at v3_purp.c:706
0x0866fddc: check_issued at x509_vfy.c:417
0x086716d0: X509_verify_cert at x509_vfy.c:212
0x086967c0: ssl3_output_cert_chain at s3_both.c:387
0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
0x0868b864: ssl3_accept at s3_srvr.c:418
0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
0x08698bb8: ssl23_accept at s23_srvr.c:213

Once:
0x08663990: sk_insert at stack.c:151
0x085c8204: asn1_template_noexp_d2i at tasn_dec.c:715
0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
0x08683698: x509v3_cache_extensions at v3_purp.c:722
 (inlined by) X509_check_issued at v3_purp.c:706
0x0866fddc: check_issued at x509_vfy.c:417
0x086716d0: X509_verify_cert at x509_vfy.c:212
0x086967c0: ssl3_output_cert_chain at s3_both.c:387
0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
0x0868b864: ssl3_accept at s3_srvr.c:418
0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
0x08698bb8: ssl23_accept at s23_srvr.c:213

Any help would be appreciated.

Charles A. Barbe
Senior Software Engineer
Allworx, a Windstream company
245 East Main St | Rochester NY | 14604
[hidden email] | 585.421.5565
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Small memory leak on multithreaded server

Jeffrey Walton-3
> Any help would be appreciated.

This could be one of two problems. First, it could be an issue with
your code and the way you handle cleanup. To help diagnose this,
please show us your cleanup code.

Second, it could be the memory leak from the compression methods. This
is a well known problem dating back a few years that has not been
fixed. See, for example,
http://rt.openssl.org/Ticket/Display.html?id=2561&user=guest&pass=guest.

On Thu, Nov 20, 2014 at 5:19 PM, Barbe, Charles
<[hidden email]> wrote:

> Hello,
>
> I have noticed a small and consistent memory leak in my multithreaded openssl server and am wondering if somebody can help me figure out what I need to do to free it when my application closes. I am on OpenSSL version 1.0.1j. Here's how I reproduce the leak:
>
> 1) Start up my server
> 2) Load my homepage with Chrome
> 3) Load my homepage with IE
> 4) Load my homepage with Firefox
>
> I can do any combination of steps 2,3 and 4 above (ie. leave some of them out) and I always get the same amount of memory left over after I shut down my application. I believe this means that this is some sort of global information that OpenSSL is hanging on to and not something in my SSL connection structure.
>
> Specifically I get 20 blocks totaling 253 bytes. I have stack traces of where each block is allocated but I cannot figure out how this memory should be cleaned up. Each of the 20 blocks filter down to 1 of 5 root stack traces. The stack traces are:
>
> Repeated 6 times:
> 0x085c519c: asn1_item_ex_combine_new at tasn_new.c:164
> 0x085c7fd8: ASN1_item_ex_d2i at tasn_dec.c:315 (discriminator 1)
> 0x085c81dc: asn1_template_noexp_d2i at tasn_dec.c:706
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
> 0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
> 0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
> 0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
> 0x08683698: x509v3_cache_extensions at v3_purp.c:722
>  (inlined by) X509_check_issued at v3_purp.c:706
> 0x0866fddc: check_issued at x509_vfy.c:417
> 0x086716d0: X509_verify_cert at x509_vfy.c:212
> 0x086967c0: ssl3_output_cert_chain at s3_both.c:387
> 0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
> 0x0868b864: ssl3_accept at s3_srvr.c:418
> 0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
> 0x08698bb8: ssl23_accept at s23_srvr.c:213
>
> Repeated 6 times:
> 0x085d12a4: ASN1_STRING_set at asn1_lib.c:389
> 0x085c715c: asn1_ex_c2i at tasn_dec.c:1056
> 0x085c7334: asn1_d2i_ex_primitive at tasn_dec.c:836
> 0x085c7fc8: ASN1_item_ex_d2i at tasn_dec.c:198
> 0x085c82a0: asn1_template_noexp_d2i at tasn_dec.c:734
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7a20: ASN1_item_ex_d2i at tasn_dec.c:331
> 0x085c81dc: asn1_template_noexp_d2i at tasn_dec.c:706
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
> 0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
> 0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
> 0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
> 0x08683698: x509v3_cache_extensions at v3_purp.c:722
>  (inlined by) X509_check_issued at v3_purp.c:706
> 0x0866fddc: check_issued at x509_vfy.c:417
> 0x086716d0: X509_verify_cert at x509_vfy.c:212
> 0x086967c0: ssl3_output_cert_chain at s3_both.c:387
> 0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
> 0x0868b864: ssl3_accept at s3_srvr.c:418
> 0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
> 0x08698bb8: ssl23_accept at s23_srvr.c:213
>
> Repeated 6 times:
> 0x085d13bc: ASN1_STRING_type_new at asn1_lib.c:429
> 0x085c71f4: asn1_ex_c2i at tasn_dec.c:1032
> 0x085c7334: asn1_d2i_ex_primitive at tasn_dec.c:836
> 0x085c7fc8: ASN1_item_ex_d2i at tasn_dec.c:198
> 0x085c82a0: asn1_template_noexp_d2i at tasn_dec.c:734
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7a20: ASN1_item_ex_d2i at tasn_dec.c:331
> 0x085c81dc: asn1_template_noexp_d2i at tasn_dec.c:706
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
> 0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
> 0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
> 0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
> 0x08683698: x509v3_cache_extensions at v3_purp.c:722
>  (inlined by) X509_check_issued at v3_purp.c:706
> 0x0866fddc: check_issued at x509_vfy.c:417
> 0x086716d0: X509_verify_cert at x509_vfy.c:212
> 0x086967c0: ssl3_output_cert_chain at s3_both.c:387
> 0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
> 0x0868b864: ssl3_accept at s3_srvr.c:418
> 0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
> 0x08698bb8: ssl23_accept at s23_srvr.c:213
>
> Once:
> 0x0866380c: sk_new at stack.c:125
> 0x0866380c: sk_new at stack.c:125
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
> 0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
> 0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
> 0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
> 0x08683698: x509v3_cache_extensions at v3_purp.c:722
>  (inlined by) X509_check_issued at v3_purp.c:706
> 0x0866fddc: check_issued at x509_vfy.c:417
> 0x086716d0: X509_verify_cert at x509_vfy.c:212
> 0x086967c0: ssl3_output_cert_chain at s3_both.c:387
> 0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
> 0x0868b864: ssl3_accept at s3_srvr.c:418
> 0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
> 0x08698bb8: ssl23_accept at s23_srvr.c:213
>
> Once:
> 0x08663990: sk_insert at stack.c:151
> 0x085c8204: asn1_template_noexp_d2i at tasn_dec.c:715
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
> 0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
> 0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
> 0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
> 0x08683698: x509v3_cache_extensions at v3_purp.c:722
>  (inlined by) X509_check_issued at v3_purp.c:706
> 0x0866fddc: check_issued at x509_vfy.c:417
> 0x086716d0: X509_verify_cert at x509_vfy.c:212
> 0x086967c0: ssl3_output_cert_chain at s3_both.c:387
> 0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
> 0x0868b864: ssl3_accept at s3_srvr.c:418
> 0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
> 0x08698bb8: ssl23_accept at s23_srvr.c:213
>
> Any help would be appreciated.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Small memory leak on multithreaded server

Viktor Dukhovni
In reply to this post by Barbe, Charles
On Thu, Nov 20, 2014 at 10:19:32PM +0000, Barbe, Charles wrote:

> I can do any combination of steps 2,3 and 4 above (ie. leave some
> of them out) and I always get the same amount of memory left over
> after I shut down my application. I believe this means that this
> is some sort of global information that OpenSSL is hanging on to
> and not something in my SSL connection structure.
>
> Specifically I get 20 blocks totaling 253 bytes. I have stack
> traces of where each block is allocated but I cannot figure out
> how this memory should be cleaned up. Each of the 20 blocks filter
> down to 1 of 5 root stack traces. The stack traces are:

A fixed amount of memory that is not deallocated and is independent
of the number of operations performed, is NOT a memory leak.
Librariers to allocate memory for the lifetime of the process during
one time initialization or first use of a function.  This is normal.

Tracking this down is a waste of time IMHO.

--
        Viktor.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Small memory leak on multithreaded server

Dr. Stephen Henson
In reply to this post by Barbe, Charles
On Thu, Nov 20, 2014, Barbe, Charles wrote:

> Hello,
>
> I have noticed a small and consistent memory leak in my multithreaded openssl server and am wondering if somebody can help me figure out what I need to do to free it when my application closes. I am on OpenSSL version 1.0.1j. Here's how I reproduce the leak:
>
> 1) Start up my server
> 2) Load my homepage with Chrome
> 3) Load my homepage with IE
> 4) Load my homepage with Firefox
>
> I can do any combination of steps 2,3 and 4 above (ie. leave some of them out) and I always get the same amount of memory left over after I shut down my application. I believe this means that this is some sort of global information that OpenSSL is hanging on to and not something in my SSL connection structure.
>
> Specifically I get 20 blocks totaling 253 bytes. I have stack traces of where each block is allocated but I cannot figure out how this memory should be cleaned up. Each of the 20 blocks filter down to 1 of 5 root stack traces. The stack traces are:
>
> Repeated 6 times:
> 0x085c519c: asn1_item_ex_combine_new at tasn_new.c:164
> 0x085c7fd8: ASN1_item_ex_d2i at tasn_dec.c:315 (discriminator 1)
> 0x085c81dc: asn1_template_noexp_d2i at tasn_dec.c:706
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
> 0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
> 0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
> 0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
> 0x08683698: x509v3_cache_extensions at v3_purp.c:722
>  (inlined by) X509_check_issued at v3_purp.c:706
> 0x0866fddc: check_issued at x509_vfy.c:417
> 0x086716d0: X509_verify_cert at x509_vfy.c:212
> 0x086967c0: ssl3_output_cert_chain at s3_both.c:387
> 0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
> 0x0868b864: ssl3_accept at s3_srvr.c:418
> 0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
> 0x08698bb8: ssl23_accept at s23_srvr.c:213
>
> Repeated 6 times:
> 0x085d12a4: ASN1_STRING_set at asn1_lib.c:389
> 0x085c715c: asn1_ex_c2i at tasn_dec.c:1056
> 0x085c7334: asn1_d2i_ex_primitive at tasn_dec.c:836
> 0x085c7fc8: ASN1_item_ex_d2i at tasn_dec.c:198
> 0x085c82a0: asn1_template_noexp_d2i at tasn_dec.c:734
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7a20: ASN1_item_ex_d2i at tasn_dec.c:331
> 0x085c81dc: asn1_template_noexp_d2i at tasn_dec.c:706
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
> 0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
> 0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
> 0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
> 0x08683698: x509v3_cache_extensions at v3_purp.c:722
>  (inlined by) X509_check_issued at v3_purp.c:706
> 0x0866fddc: check_issued at x509_vfy.c:417
> 0x086716d0: X509_verify_cert at x509_vfy.c:212
> 0x086967c0: ssl3_output_cert_chain at s3_both.c:387
> 0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
> 0x0868b864: ssl3_accept at s3_srvr.c:418
> 0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
> 0x08698bb8: ssl23_accept at s23_srvr.c:213
>
> Repeated 6 times:
> 0x085d13bc: ASN1_STRING_type_new at asn1_lib.c:429
> 0x085c71f4: asn1_ex_c2i at tasn_dec.c:1032
> 0x085c7334: asn1_d2i_ex_primitive at tasn_dec.c:836
> 0x085c7fc8: ASN1_item_ex_d2i at tasn_dec.c:198
> 0x085c82a0: asn1_template_noexp_d2i at tasn_dec.c:734
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7a20: ASN1_item_ex_d2i at tasn_dec.c:331
> 0x085c81dc: asn1_template_noexp_d2i at tasn_dec.c:706
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
> 0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
> 0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
> 0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
> 0x08683698: x509v3_cache_extensions at v3_purp.c:722
>  (inlined by) X509_check_issued at v3_purp.c:706
> 0x0866fddc: check_issued at x509_vfy.c:417
> 0x086716d0: X509_verify_cert at x509_vfy.c:212
> 0x086967c0: ssl3_output_cert_chain at s3_both.c:387
> 0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
> 0x0868b864: ssl3_accept at s3_srvr.c:418
> 0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
> 0x08698bb8: ssl23_accept at s23_srvr.c:213
>
> Once:
> 0x0866380c: sk_new at stack.c:125
> 0x0866380c: sk_new at stack.c:125
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
> 0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
> 0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
> 0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
> 0x08683698: x509v3_cache_extensions at v3_purp.c:722
>  (inlined by) X509_check_issued at v3_purp.c:706
> 0x0866fddc: check_issued at x509_vfy.c:417
> 0x086716d0: X509_verify_cert at x509_vfy.c:212
> 0x086967c0: ssl3_output_cert_chain at s3_both.c:387
> 0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
> 0x0868b864: ssl3_accept at s3_srvr.c:418
> 0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
> 0x08698bb8: ssl23_accept at s23_srvr.c:213
>
> Once:
> 0x08663990: sk_insert at stack.c:151
> 0x085c8204: asn1_template_noexp_d2i at tasn_dec.c:715
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
> 0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
> 0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
> 0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
> 0x08683698: x509v3_cache_extensions at v3_purp.c:722
>  (inlined by) X509_check_issued at v3_purp.c:706
> 0x0866fddc: check_issued at x509_vfy.c:417
> 0x086716d0: X509_verify_cert at x509_vfy.c:212
> 0x086967c0: ssl3_output_cert_chain at s3_both.c:387
> 0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
> 0x0868b864: ssl3_accept at s3_srvr.c:418
> 0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
> 0x08698bb8: ssl23_accept at s23_srvr.c:213
>
> Any help would be appreciated.
>

Not possible to be sure from that trace but at a guess I'd say a STACK of
certificates hasn't been freed up.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Small memory leak on multithreaded server

Barbe, Charles
In reply to this post by Viktor Dukhovni
I see your point but I cannot have memory allocated when my application shuts down. This constraint is related to the fact that this is an embedded VoIP system.

Thanks for the response!

Charles A. Barbe
Senior Software Engineer
Allworx, a Windstream company
245 East Main St | Rochester NY | 14604
[hidden email] | 585.421.5565

________________________________________
From: [hidden email] [[hidden email]] on behalf of Viktor Dukhovni [[hidden email]]
Sent: Thursday, November 20, 2014 7:26 PM
To: [hidden email]
Subject: Re: Small memory leak on multithreaded server

On Thu, Nov 20, 2014 at 10:19:32PM +0000, Barbe, Charles wrote:

> I can do any combination of steps 2,3 and 4 above (ie. leave some
> of them out) and I always get the same amount of memory left over
> after I shut down my application. I believe this means that this
> is some sort of global information that OpenSSL is hanging on to
> and not something in my SSL connection structure.
>
> Specifically I get 20 blocks totaling 253 bytes. I have stack
> traces of where each block is allocated but I cannot figure out
> how this memory should be cleaned up. Each of the 20 blocks filter
> down to 1 of 5 root stack traces. The stack traces are:

A fixed amount of memory that is not deallocated and is independent
of the number of operations performed, is NOT a memory leak.
Librariers to allocate memory for the lifetime of the process during
one time initialization or first use of a function.  This is normal.

Tracking this down is a waste of time IMHO.

--
        Viktor.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Small memory leak on multithreaded server

Barbe, Charles
In reply to this post by Jeffrey Walton-3
Thanks for the response... here is the code that runs when my connection closes:

void OpensslConnection::cleanup()
{
    if(ssl != NULL)
    {
        if(isConnected)
        {
        while(SSL_shutdown(ssl) == 0)
           ;
        }
        SSL_free(ssl);
        ERR_remove_state(0);
        ssl = NULL;
    }

    isConnected = false;
}

And here is the code that runs to shut down my SSL library:

static void
openSslShutdown ()
{
        CONF_modules_free();
        ERR_remove_state(0);
        CONF_modules_unload(1);
        ERR_free_strings();
        EVP_cleanup();
        CRYPTO_cleanup_all_ex_data();

        if (opensslLocks != NULL)
    {
        for(int i = 0; i < CRYPTO_num_locks(); i++)
        {
            PAL_mutexDestroy (opensslLocks[i]);
        }

        IST_FREE (opensslLocks);
    }
}

Also, I have numerous worker threads handling connections and they all do the following before they exit:

  ERR_remove_thread_state(0);

Charles A. Barbe
Senior Software Engineer
Allworx, a Windstream company
245 East Main St | Rochester NY | 14604
[hidden email] | 585.421.5565

________________________________________
From: [hidden email] [[hidden email]] on behalf of Jeffrey Walton [[hidden email]]
Sent: Thursday, November 20, 2014 6:03 PM
To: OpenSSL Users List
Subject: Re: Small memory leak on multithreaded server

> Any help would be appreciated.

This could be one of two problems. First, it could be an issue with
your code and the way you handle cleanup. To help diagnose this,
please show us your cleanup code.

Second, it could be the memory leak from the compression methods. This
is a well known problem dating back a few years that has not been
fixed. See, for example,
http://rt.openssl.org/Ticket/Display.html?id=2561&user=guest&pass=guest.

On Thu, Nov 20, 2014 at 5:19 PM, Barbe, Charles
<[hidden email]> wrote:

> Hello,
>
> I have noticed a small and consistent memory leak in my multithreaded openssl server and am wondering if somebody can help me figure out what I need to do to free it when my application closes. I am on OpenSSL version 1.0.1j. Here's how I reproduce the leak:
>
> 1) Start up my server
> 2) Load my homepage with Chrome
> 3) Load my homepage with IE
> 4) Load my homepage with Firefox
>
> I can do any combination of steps 2,3 and 4 above (ie. leave some of them out) and I always get the same amount of memory left over after I shut down my application. I believe this means that this is some sort of global information that OpenSSL is hanging on to and not something in my SSL connection structure.
>
> Specifically I get 20 blocks totaling 253 bytes. I have stack traces of where each block is allocated but I cannot figure out how this memory should be cleaned up. Each of the 20 blocks filter down to 1 of 5 root stack traces. The stack traces are:
>
> Repeated 6 times:
> 0x085c519c: asn1_item_ex_combine_new at tasn_new.c:164
> 0x085c7fd8: ASN1_item_ex_d2i at tasn_dec.c:315 (discriminator 1)
> 0x085c81dc: asn1_template_noexp_d2i at tasn_dec.c:706
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
> 0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
> 0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
> 0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
> 0x08683698: x509v3_cache_extensions at v3_purp.c:722
>  (inlined by) X509_check_issued at v3_purp.c:706
> 0x0866fddc: check_issued at x509_vfy.c:417
> 0x086716d0: X509_verify_cert at x509_vfy.c:212
> 0x086967c0: ssl3_output_cert_chain at s3_both.c:387
> 0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
> 0x0868b864: ssl3_accept at s3_srvr.c:418
> 0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
> 0x08698bb8: ssl23_accept at s23_srvr.c:213
>
> Repeated 6 times:
> 0x085d12a4: ASN1_STRING_set at asn1_lib.c:389
> 0x085c715c: asn1_ex_c2i at tasn_dec.c:1056
> 0x085c7334: asn1_d2i_ex_primitive at tasn_dec.c:836
> 0x085c7fc8: ASN1_item_ex_d2i at tasn_dec.c:198
> 0x085c82a0: asn1_template_noexp_d2i at tasn_dec.c:734
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7a20: ASN1_item_ex_d2i at tasn_dec.c:331
> 0x085c81dc: asn1_template_noexp_d2i at tasn_dec.c:706
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
> 0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
> 0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
> 0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
> 0x08683698: x509v3_cache_extensions at v3_purp.c:722
>  (inlined by) X509_check_issued at v3_purp.c:706
> 0x0866fddc: check_issued at x509_vfy.c:417
> 0x086716d0: X509_verify_cert at x509_vfy.c:212
> 0x086967c0: ssl3_output_cert_chain at s3_both.c:387
> 0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
> 0x0868b864: ssl3_accept at s3_srvr.c:418
> 0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
> 0x08698bb8: ssl23_accept at s23_srvr.c:213
>
> Repeated 6 times:
> 0x085d13bc: ASN1_STRING_type_new at asn1_lib.c:429
> 0x085c71f4: asn1_ex_c2i at tasn_dec.c:1032
> 0x085c7334: asn1_d2i_ex_primitive at tasn_dec.c:836
> 0x085c7fc8: ASN1_item_ex_d2i at tasn_dec.c:198
> 0x085c82a0: asn1_template_noexp_d2i at tasn_dec.c:734
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7a20: ASN1_item_ex_d2i at tasn_dec.c:331
> 0x085c81dc: asn1_template_noexp_d2i at tasn_dec.c:706
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
> 0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
> 0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
> 0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
> 0x08683698: x509v3_cache_extensions at v3_purp.c:722
>  (inlined by) X509_check_issued at v3_purp.c:706
> 0x0866fddc: check_issued at x509_vfy.c:417
> 0x086716d0: X509_verify_cert at x509_vfy.c:212
> 0x086967c0: ssl3_output_cert_chain at s3_both.c:387
> 0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
> 0x0868b864: ssl3_accept at s3_srvr.c:418
> 0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
> 0x08698bb8: ssl23_accept at s23_srvr.c:213
>
> Once:
> 0x0866380c: sk_new at stack.c:125
> 0x0866380c: sk_new at stack.c:125
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
> 0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
> 0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
> 0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
> 0x08683698: x509v3_cache_extensions at v3_purp.c:722
>  (inlined by) X509_check_issued at v3_purp.c:706
> 0x0866fddc: check_issued at x509_vfy.c:417
> 0x086716d0: X509_verify_cert at x509_vfy.c:212
> 0x086967c0: ssl3_output_cert_chain at s3_both.c:387
> 0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
> 0x0868b864: ssl3_accept at s3_srvr.c:418
> 0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
> 0x08698bb8: ssl23_accept at s23_srvr.c:213
>
> Once:
> 0x08663990: sk_insert at stack.c:151
> 0x085c8204: asn1_template_noexp_d2i at tasn_dec.c:715
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
> 0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
> 0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
> 0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
> 0x08683698: x509v3_cache_extensions at v3_purp.c:722
>  (inlined by) X509_check_issued at v3_purp.c:706
> 0x0866fddc: check_issued at x509_vfy.c:417
> 0x086716d0: X509_verify_cert at x509_vfy.c:212
> 0x086967c0: ssl3_output_cert_chain at s3_both.c:387
> 0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
> 0x0868b864: ssl3_accept at s3_srvr.c:418
> 0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
> 0x08698bb8: ssl23_accept at s23_srvr.c:213
>
> Any help would be appreciated.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Small memory leak on multithreaded server

Viktor Dukhovni
On Fri, Nov 21, 2014 at 02:26:45PM +0000, Barbe, Charles wrote:

> static void
> openSslShutdown ()
> {

You could try adding:

    ENGINE_cleanup();

if that helps.

> CONF_modules_free();
> ERR_remove_state(0);
> CONF_modules_unload(1);
> ERR_free_strings();
> EVP_cleanup();
> CRYPTO_cleanup_all_ex_data();
>
> if (opensslLocks != NULL)
>     {
>         for(int i = 0; i < CRYPTO_num_locks(); i++)
>         {
>             PAL_mutexDestroy (opensslLocks[i]);
>         }
>
>         IST_FREE (opensslLocks);
>     }
> }

You may need to patch the library to add deallocation code for any
one-time allocations not handled by the above.

--
        Viktor.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Small memory leak on multithreaded server

Jakob Bohm-7
In reply to this post by Barbe, Charles
On 21/11/2014 15:26, Barbe, Charles wrote:

> Thanks for the response... here is the code that runs when my connection closes:
>
> void OpensslConnection::cleanup()
> {
>      if(ssl != NULL)
>      {
>          if(isConnected)
>          {
>           while(SSL_shutdown(ssl) == 0)
>              ;
>          }
>          SSL_free(ssl);
>          ERR_remove_state(0);
>          ssl = NULL;
>      }
>
>      isConnected = false;
> }
>
> And here is the code that runs to shut down my SSL library:
>
> static void
> openSslShutdown ()
> {
> CONF_modules_free();
> ERR_remove_state(0);
> CONF_modules_unload(1);
> ERR_free_strings();
> EVP_cleanup();
> CRYPTO_cleanup_all_ex_data();
>
> if (opensslLocks != NULL)
>      {
>          for(int i = 0; i < CRYPTO_num_locks(); i++)
>          {
>              PAL_mutexDestroy (opensslLocks[i]);
>          }
>
>          IST_FREE (opensslLocks);
>      }
> }
>
> Also, I have numerous worker threads handling connections and they all do the following before they exit:
>
>    ERR_remove_thread_state(0);
 From the response by Dr. Henson, maybe you need code to unload your
server certificate and its certificate chain (a STACK of
certificates).

> ________________________________________
> From: [hidden email] [[hidden email]] on behalf of Jeffrey Walton [[hidden email]]
> Sent: Thursday, November 20, 2014 6:03 PM
> To: OpenSSL Users List
> Subject: Re: Small memory leak on multithreaded server
>
>> Any help would be appreciated.
> This could be one of two problems. First, it could be an issue with
> your code and the way you handle cleanup. To help diagnose this,
> please show us your cleanup code.
>
> Second, it could be the memory leak from the compression methods. This
> is a well known problem dating back a few years that has not been
> fixed. See, for example,
> http://rt.openssl.org/Ticket/Display.html?id=2561&user=guest&pass=guest.
>
> On Thu, Nov 20, 2014 at 5:19 PM, Barbe, Charles
> <[hidden email]> wrote:
>> Hello,
>>
>> I have noticed a small and consistent memory leak in my multithreaded openssl server and am wondering if somebody can help me figure out what I need to do to free it when my application closes. I am on OpenSSL version 1.0.1j. Here's how I reproduce the leak:
>>
>> 1) Start up my server
>> 2) Load my homepage with Chrome
>> 3) Load my homepage with IE
>> 4) Load my homepage with Firefox
>>
>> I can do any combination of steps 2,3 and 4 above (ie. leave some of them out) and I always get the same amount of memory left over after I shut down my application. I believe this means that this is some sort of global information that OpenSSL is hanging on to and not something in my SSL connection structure.
>>
>> Specifically I get 20 blocks totaling 253 bytes. I have stack traces of where each block is allocated but I cannot figure out how this memory should be cleaned up. Each of the 20 blocks filter down to 1 of 5 root stack traces. The stack traces are:
>>
>> Repeated 6 times:
>>

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  http://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Small memory leak on multithreaded server

Barbe, Charles
In reply to this post by Dr. Stephen Henson
It sort of makes sense that this is a stack of certs that is left over but what confuses me is that this memory is allocated by calls executed as a result of my calling SSL_accept(ssl). Also, we're talking 253 bytes... i would imagine that stack of certs would be larger than that, wouldn't it? And shouldn't all the memory allocated by SSL_accept be freed when I call SSL_shutdown and SSL_free?

What looks suspicious to me is the calls to "x509v3_cache_extensions" that are in the traces below. This implies to me that openssl is caching something. How do I ask it to clear that cache? If i need to augment the library to do so, where are the pointers to the objects it is caching so I can free it? I tried to trace the code but got lost... i'm sure I can trace it given enough time but thought I'd ask the community first.

I also note that I have no indication of any "left over memory" from my allocation of the certs that I am providing to my SSL_CTX... meaning that I believe I am cleaning my SSL_CTX and associated resources appropriately. The memory in question is definitely allocated by OpenSSL as a result of calling SSL_accept... I just need to figure out how to ask OpenSSL to free it.

Thanks so much for the help.


Charles A. Barbe
Senior Software Engineer
Allworx, a Windstream company
245 East Main St | Rochester NY | 14604
[hidden email] | 585.421.5565

________________________________________
From: [hidden email] [[hidden email]] on behalf of Dr. Stephen Henson [[hidden email]]
Sent: Thursday, November 20, 2014 9:50 PM
To: [hidden email]
Subject: Re: Small memory leak on multithreaded server

On Thu, Nov 20, 2014, Barbe, Charles wrote:

> Hello,
>
> I have noticed a small and consistent memory leak in my multithreaded openssl server and am wondering if somebody can help me figure out what I need to do to free it when my application closes. I am on OpenSSL version 1.0.1j. Here's how I reproduce the leak:
>
> 1) Start up my server
> 2) Load my homepage with Chrome
> 3) Load my homepage with IE
> 4) Load my homepage with Firefox
>
> I can do any combination of steps 2,3 and 4 above (ie. leave some of them out) and I always get the same amount of memory left over after I shut down my application. I believe this means that this is some sort of global information that OpenSSL is hanging on to and not something in my SSL connection structure.
>
> Specifically I get 20 blocks totaling 253 bytes. I have stack traces of where each block is allocated but I cannot figure out how this memory should be cleaned up. Each of the 20 blocks filter down to 1 of 5 root stack traces. The stack traces are:
>
> Repeated 6 times:
> 0x085c519c: asn1_item_ex_combine_new at tasn_new.c:164
> 0x085c7fd8: ASN1_item_ex_d2i at tasn_dec.c:315 (discriminator 1)
> 0x085c81dc: asn1_template_noexp_d2i at tasn_dec.c:706
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
> 0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
> 0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
> 0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
> 0x08683698: x509v3_cache_extensions at v3_purp.c:722
>  (inlined by) X509_check_issued at v3_purp.c:706
> 0x0866fddc: check_issued at x509_vfy.c:417
> 0x086716d0: X509_verify_cert at x509_vfy.c:212
> 0x086967c0: ssl3_output_cert_chain at s3_both.c:387
> 0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
> 0x0868b864: ssl3_accept at s3_srvr.c:418
> 0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
> 0x08698bb8: ssl23_accept at s23_srvr.c:213
>
> Repeated 6 times:
> 0x085d12a4: ASN1_STRING_set at asn1_lib.c:389
> 0x085c715c: asn1_ex_c2i at tasn_dec.c:1056
> 0x085c7334: asn1_d2i_ex_primitive at tasn_dec.c:836
> 0x085c7fc8: ASN1_item_ex_d2i at tasn_dec.c:198
> 0x085c82a0: asn1_template_noexp_d2i at tasn_dec.c:734
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7a20: ASN1_item_ex_d2i at tasn_dec.c:331
> 0x085c81dc: asn1_template_noexp_d2i at tasn_dec.c:706
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
> 0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
> 0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
> 0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
> 0x08683698: x509v3_cache_extensions at v3_purp.c:722
>  (inlined by) X509_check_issued at v3_purp.c:706
> 0x0866fddc: check_issued at x509_vfy.c:417
> 0x086716d0: X509_verify_cert at x509_vfy.c:212
> 0x086967c0: ssl3_output_cert_chain at s3_both.c:387
> 0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
> 0x0868b864: ssl3_accept at s3_srvr.c:418
> 0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
> 0x08698bb8: ssl23_accept at s23_srvr.c:213
>
> Repeated 6 times:
> 0x085d13bc: ASN1_STRING_type_new at asn1_lib.c:429
> 0x085c71f4: asn1_ex_c2i at tasn_dec.c:1032
> 0x085c7334: asn1_d2i_ex_primitive at tasn_dec.c:836
> 0x085c7fc8: ASN1_item_ex_d2i at tasn_dec.c:198
> 0x085c82a0: asn1_template_noexp_d2i at tasn_dec.c:734
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7a20: ASN1_item_ex_d2i at tasn_dec.c:331
> 0x085c81dc: asn1_template_noexp_d2i at tasn_dec.c:706
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
> 0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
> 0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
> 0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
> 0x08683698: x509v3_cache_extensions at v3_purp.c:722
>  (inlined by) X509_check_issued at v3_purp.c:706
> 0x0866fddc: check_issued at x509_vfy.c:417
> 0x086716d0: X509_verify_cert at x509_vfy.c:212
> 0x086967c0: ssl3_output_cert_chain at s3_both.c:387
> 0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
> 0x0868b864: ssl3_accept at s3_srvr.c:418
> 0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
> 0x08698bb8: ssl23_accept at s23_srvr.c:213
>
> Once:
> 0x0866380c: sk_new at stack.c:125
> 0x0866380c: sk_new at stack.c:125
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
> 0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
> 0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
> 0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
> 0x08683698: x509v3_cache_extensions at v3_purp.c:722
>  (inlined by) X509_check_issued at v3_purp.c:706
> 0x0866fddc: check_issued at x509_vfy.c:417
> 0x086716d0: X509_verify_cert at x509_vfy.c:212
> 0x086967c0: ssl3_output_cert_chain at s3_both.c:387
> 0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
> 0x0868b864: ssl3_accept at s3_srvr.c:418
> 0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
> 0x08698bb8: ssl23_accept at s23_srvr.c:213
>
> Once:
> 0x08663990: sk_insert at stack.c:151
> 0x085c8204: asn1_template_noexp_d2i at tasn_dec.c:715
> 0x085c85a0: asn1_template_ex_d2i at tasn_dec.c:616
> 0x085c7d90: ASN1_item_ex_d2i at tasn_dec.c:195
> 0x085c80a8: ASN1_item_d2i at tasn_dec.c:136
> 0x0867b368: X509V3_EXT_d2i at v3_lib.c:177
> 0x08682f3c: x509v3_cache_extensions at v3_purp.c:464
> 0x08683698: x509v3_cache_extensions at v3_purp.c:722
>  (inlined by) X509_check_issued at v3_purp.c:706
> 0x0866fddc: check_issued at x509_vfy.c:417
> 0x086716d0: X509_verify_cert at x509_vfy.c:212
> 0x086967c0: ssl3_output_cert_chain at s3_both.c:387
> 0x0868ac68: ssl3_send_server_certificate at s3_srvr.c:3365
> 0x0868b864: ssl3_accept at s3_srvr.c:418
> 0x086982f4: ssl23_get_client_hello at s23_srvr.c:657
> 0x08698bb8: ssl23_accept at s23_srvr.c:213
>
> Any help would be appreciated.
>

Not possible to be sure from that trace but at a guess I'd say a STACK of
certificates hasn't been freed up.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Small memory leak on multithreaded server

Viktor Dukhovni
On Fri, Nov 21, 2014 at 03:21:47PM +0000, Barbe, Charles wrote:

> What looks suspicious to me is the calls to "x509v3_cache_extensions"
> that are in the traces below. This implies to me that openssl is
> caching something. How do I ask it to clear that cache? If i need
> to augment the library to do so, where are the pointers to the
> objects it is caching so I can free it? I tried to trace the code
> but got lost... i'm sure I can trace it given enough time but
> thought I'd ask the community first.

This is not a separate cache,  the extensions are "cached" in each
certificate structure, and should be freed with that certificate.
It just avoids having to decode extensions all over the place,
by computing and saving pre-decoded versions.

--
        Viktor.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Small memory leak on multithreaded server

Dr. Stephen Henson
In reply to this post by Barbe, Charles
On Fri, Nov 21, 2014, Barbe, Charles wrote:

> Thanks for the response... here is the code that runs when my connection closes:
>
> void OpensslConnection::cleanup()
> {
>     if(ssl != NULL)
>     {
>         if(isConnected)
>         {
>         while(SSL_shutdown(ssl) == 0)
>            ;
>         }
>         SSL_free(ssl);
>         ERR_remove_state(0);
>         ssl = NULL;
>     }
>
>     isConnected = false;
> }
>
> And here is the code that runs to shut down my SSL library:
>
> static void
> openSslShutdown ()
> {
> CONF_modules_free();
> ERR_remove_state(0);
> CONF_modules_unload(1);
> ERR_free_strings();
> EVP_cleanup();
> CRYPTO_cleanup_all_ex_data();
>
> if (opensslLocks != NULL)
>     {
>         for(int i = 0; i < CRYPTO_num_locks(); i++)
>         {
>             PAL_mutexDestroy (opensslLocks[i]);
>         }
>
>         IST_FREE (opensslLocks);
>     }
> }
>
> Also, I have numerous worker threads handling connections and they all do the following before they exit:
>
>   ERR_remove_thread_state(0);
>

You are calling SSL_CTX_free aren't you?

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Small memory leak on multithreaded server

Barbe, Charles
Yes... sorry, forgot to include this part of my shutdown sequence. One thing I am noticing is that I do not call X409_free on my certs. I even have a comment in my code saying that I am not freeing them because I think they will be freed when the SSL_CTX is freed. Is that a correct assumption or should I be calling X509 free on them explicitly?

Here's the extra code that I forgot to include:

STATUS
SSL_shutDown()
{
    // Client
    if (clientCtxt != NULL)
    {
        SSL_CTX_free (clientCtxt);
        clientCtxt = NULL;
    }

    // View
    clearCertsVector (getCertCachePointer (SSL_TYPE_VIEW));

    if (pViewKeyCache != NULL)
    {
        EVP_PKEY_free (pViewKeyCache);
        pViewKeyCache = NULL;
    }

    if (viewServerCtxt != NULL)
    {
        SSL_CTX_free (viewServerCtxt);
        viewServerCtxt = NULL;
    }

    // Web - only if server supports HTTPS
    if (supportHTTPS)
    {
        clearCertsVector (getCertCachePointer (SSL_TYPE_WEB));

        if (pPendingWebKeyCache != NULL)
        {
            EVP_PKEY_free (pPendingWebKeyCache);
            pPendingWebKeyCache = NULL;
        }
   
        if (pWebKeyCache != NULL)
        {
            EVP_PKEY_free (pWebKeyCache);
            pWebKeyCache = NULL;
        }

        if (webServerCtxt != NULL)
        {
            SSL_CTX_free (webServerCtxt);
            webServerCtxt = NULL;
        }
    }

    CHECK_911 (BIO_free (out) > 0);

    openSslShutdown ();

    return SUCCESS;
}

static void
clearCertsVector(CERTS_VECTOR* certs)
{
    //Not freeing certs because they will be
    // freed when context is destroyed
    if(certs != NULL)
    {
        certs->clear();
    }
}

Charles A. Barbe
Senior Software Engineer
Allworx, a Windstream company
245 East Main St | Rochester NY | 14604
[hidden email] | 585.421.5565

________________________________________
From: [hidden email] [[hidden email]] on behalf of Dr. Stephen Henson [[hidden email]]
Sent: Friday, November 21, 2014 12:26 PM
To: [hidden email]
Subject: Re: Small memory leak on multithreaded server

On Fri, Nov 21, 2014, Barbe, Charles wrote:

> Thanks for the response... here is the code that runs when my connection closes:
>
> void OpensslConnection::cleanup()
> {
>     if(ssl != NULL)
>     {
>         if(isConnected)
>         {
>               while(SSL_shutdown(ssl) == 0)
>                   ;
>         }
>         SSL_free(ssl);
>         ERR_remove_state(0);
>         ssl = NULL;
>     }
>
>     isConnected = false;
> }
>
> And here is the code that runs to shut down my SSL library:
>
> static void
> openSslShutdown ()
> {
>       CONF_modules_free();
>       ERR_remove_state(0);
>       CONF_modules_unload(1);
>       ERR_free_strings();
>       EVP_cleanup();
>       CRYPTO_cleanup_all_ex_data();
>
>       if (opensslLocks != NULL)
>     {
>         for(int i = 0; i < CRYPTO_num_locks(); i++)
>         {
>             PAL_mutexDestroy (opensslLocks[i]);
>         }
>
>         IST_FREE (opensslLocks);
>     }
> }
>
> Also, I have numerous worker threads handling connections and they all do the following before they exit:
>
>   ERR_remove_thread_state(0);
>

You are calling SSL_CTX_free aren't you?

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Small memory leak on multithreaded server

Dr. Stephen Henson
On Fri, Nov 21, 2014, Barbe, Charles wrote:

> Yes... sorry, forgot to include this part of my shutdown sequence. One thing
> I am noticing is that I do not call X409_free on my certs. I even have a
> comment in my code saying that I am not freeing them because I think they
> will be freed when the SSL_CTX is freed. Is that a correct assumption or
> should I be calling X509 free on them explicitly?
>

If you have an explicit X509 structure and you call SSL_CTX_use_certificate
then the reference count is increased and you have to free up the certificate.

However I'm a bit confused by the output. It *looks* like it is associated
with a certificate verification operation which could be cached certificates
in a store. Do you perform any operations with an X509_STORE structure?

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Small memory leak on multithreaded server

Jeffrey Walton-3
In reply to this post by Viktor Dukhovni
> A fixed amount of memory that is not deallocated and is independent
> of the number of operations performed, is NOT a memory leak.
Languages like Java and C# can continuously load and unload a library.
You will see a growth in memory usage because the memory is not
reclaimed.

> Librariers to allocate memory for the lifetime of the process during
> one time initialization or first use of a function.  This is normal.
I think this may be an incorrect assumption.

> Tracking this down is a waste of time IMHO.
For a OpenSSL user, its hard to determine the "bad" leaks (those
caused by the user) versus the "good" leaks (those incumbent to
OpenSSL). The good leaks mask out the bad leaks.

For the 2561 bug, there's no need to track it down. You've been told
where the leak is.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Small memory leak on multithreaded server

Viktor Dukhovni
On Fri, Nov 21, 2014 at 04:13:58PM -0500, Jeffrey Walton wrote:
> > A fixed amount of memory that is not deallocated and is independent
> > of the number of operations performed, is NOT a memory leak.
>
> Languages like Java and C# can continuously load and unload a library.
> You will see a growth in memory usage because the memory is not
> reclaimed.

Unloading of shared libraries is generally unsafe.  Loading and
unloading of pure of Java packages may work well enough, but I
would expect a Java runtime that unloads native libraries to stay
running for very long.

--
        Viktor.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Small memory leak on multithreaded server

Barbe, Charles
In reply to this post by Dr. Stephen Henson
I did a search through my application code and see no occurrence of the string X509_STORE so I don't believe I'm doing any operations with that type of structure.

I am using SSL_CTX_use_certificate and was possibly under the incorrect assumption that my code did not have to explicitly free the X509* that I pass to that argument if I am subsequently calling SSL_CTX_free on the CTX. In retrospect that doesn't sound correct. I will fix my code to free the X509s as well as the CTX and see if that is my issue.

Thank you everybody for all of the help!


Charles A. Barbe
Senior Software Engineer
Allworx, a Windstream company
245 East Main St | Rochester NY | 14604
[hidden email] | 585.421.5565

________________________________________
From: [hidden email] [[hidden email]] on behalf of Dr. Stephen Henson [[hidden email]]
Sent: Friday, November 21, 2014 1:40 PM
To: [hidden email]
Subject: Re: Small memory leak on multithreaded server

On Fri, Nov 21, 2014, Barbe, Charles wrote:

> Yes... sorry, forgot to include this part of my shutdown sequence. One thing
> I am noticing is that I do not call X409_free on my certs. I even have a
> comment in my code saying that I am not freeing them because I think they
> will be freed when the SSL_CTX is freed. Is that a correct assumption or
> should I be calling X509 free on them explicitly?
>

If you have an explicit X509 structure and you call SSL_CTX_use_certificate
then the reference count is increased and you have to free up the certificate.

However I'm a bit confused by the output. It *looks* like it is associated
with a certificate verification operation which could be cached certificates
in a store. Do you perform any operations with an X509_STORE structure?

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Small memory leak on multithreaded server

Jakob Bohm-7
In reply to this post by Viktor Dukhovni
On 21-11-2014 23:23, Viktor Dukhovni wrote:

> On Fri, Nov 21, 2014 at 04:13:58PM -0500, Jeffrey Walton wrote:
>>> A fixed amount of memory that is not deallocated and is independent
>>> of the number of operations performed, is NOT a memory leak.
>> Languages like Java and C# can continuously load and unload a library.
>> You will see a growth in memory usage because the memory is not
>> reclaimed.
> Unloading of shared libraries is generally unsafe.  Loading and
> unloading of pure of Java packages may work well enough, but I
> would expect a Java runtime that unloads native libraries to stay
> running for very long.

That is horribly outdated information and an assumption that no
competent library author should make or rely on others to make.

On modern systems, unloading of shared libraries that are used
as plugins, and by extension any shared libraries that might be
referenced by plugins without being referenced by the plugin-using
application core, is a normal and frequent operation supported
by the core shared library loader and most shared libraries.

If a library contains code that needs to be automatically called
when it is loaded or unloaded without that being an exposed API
level init/cleanup function, then the library porter needs to do
the target specific gymnastics to get called by the (C) runtime
at the appropriate times, and it needs to deal with common
restrictions on what such calls from the (C) runtime are not
allowed to do (one of which is recursive calls to the dynamic
loader API).  For libraries written in C++, the static constructor
and destructor language mechanisms are treated this way
automatically and thus subject to the same limitations on
permitted operations.

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  http://www.wisemo.com
Transformervej 29, 2730 Herlev, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Small memory leak on multithreaded server

Viktor Dukhovni
On Sun, Nov 23, 2014 at 10:56:41PM +0100, Jakob Bohm wrote:

> >Unloading of shared libraries is generally unsafe.  Loading and
> >unloading of pure of Java packages may work well enough, but I
> >would [not] expect a Java runtime that unloads native libraries to stay
> >running for very long.
>
> That is horribly outdated information and an assumption that no
> competent library author should make or rely on others to make.

In an ideal world you'd be right.  In practice I think unloading
of libraries that are not specifically designed as plugins is often
problematic.

--
        Viktor.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Small memory leak on multithreaded server

Chris Gray-4
In reply to this post by Barbe, Charles
As the maintainer of an "alternative" JavaVM I can confirm that we
absolutely had to support library unloading because one customer was using
it heavily - and that was quite a few years ago. Early Sun VMs didn't
support library unloading, but then those VMs also did not garbage-collect
obsolete classes either.


> On 21-11-2014 23:23, Viktor Dukhovni wrote:
>> On Fri, Nov 21, 2014 at 04:13:58PM -0500, Jeffrey Walton wrote:
>>>> A fixed amount of memory that is not deallocated and is independent
>>>> of the number of operations performed, is NOT a memory leak.
>>> Languages like Java and C# can continuously load and unload a library.
>>> You will see a growth in memory usage because the memory is not
>>> reclaimed.
>> Unloading of shared libraries is generally unsafe.  Loading and
>> unloading of pure of Java packages may work well enough, but I
>> would expect a Java runtime that unloads native libraries to stay
>> running for very long.
>
> That is horribly outdated information and an assumption that no
> competent library author should make or rely on others to make.
>
> On modern systems, unloading of shared libraries that are used
> as plugins, and by extension any shared libraries that might be
> referenced by plugins without being referenced by the plugin-using
> application core, is a normal and frequent operation supported
> by the core shared library loader and most shared libraries.
>
> If a library contains code that needs to be automatically called
> when it is loaded or unloaded without that being an exposed API
> level init/cleanup function, then the library porter needs to do
> the target specific gymnastics to get called by the (C) runtime
> at the appropriate times, and it needs to deal with common
> restrictions on what such calls from the (C) runtime are not
> allowed to do (one of which is recursive calls to the dynamic
> loader API).  For libraries written in C++, the static constructor
> and destructor language mechanisms are treated this way
> automatically and thus subject to the same limitations on
> permitted operations.
>
> Enjoy
>
> Jakob
> --
> Jakob Bohm, CIO, Partner, WiseMo A/S.  http://www.wisemo.com
> Transformervej 29, 2730 Herlev, Denmark.  Direct +45 31 13 16 10
> This public discussion message is non-binding and may contain errors.
> WiseMo - Remote Service Management for PCs, Phones and Embedded
> ______________________________________________________________________
> OpenSSL Project                                 http://www.openssl.org
> User Support Mailing List                    [hidden email]
> Automated List Manager                           [hidden email]
>


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Small memory leak on multithreaded server

Barbe, Charles
In reply to this post by Dr. Stephen Henson
Thank you all for your help. My problem did end up being my misunderstanding of the reference counting done by OpenSSL I was expecting that my X509*'s would get freed when I freed my SSL_CTX but that was an incorrect expectation because I still had a reference to those X509*'s in a vector.

This conversation also helped me find some other places where I wasn't properly freeing reference counted OpenSSL structures.

Thanks for the help!

Charles A. Barbe
Senior Software Engineer
Allworx, a Windstream company
245 East Main St | Rochester NY | 14604
[hidden email] | 585.421.5565

________________________________________
From: [hidden email] [[hidden email]] on behalf of Dr. Stephen Henson [[hidden email]]
Sent: Friday, November 21, 2014 1:40 PM
To: [hidden email]
Subject: Re: Small memory leak on multithreaded server

On Fri, Nov 21, 2014, Barbe, Charles wrote:

> Yes... sorry, forgot to include this part of my shutdown sequence. One thing
> I am noticing is that I do not call X409_free on my certs. I even have a
> comment in my code saying that I am not freeing them because I think they
> will be freed when the SSL_CTX is freed. Is that a correct assumption or
> should I be calling X509 free on them explicitly?
>

If you have an explicit X509 structure and you call SSL_CTX_use_certificate
then the reference count is increased and you have to free up the certificate.

However I'm a bit confused by the output. It *looks* like it is associated
with a certificate verification operation which could be cached certificates
in a store. Do you perform any operations with an X509_STORE structure?

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]