libssl 1.1 blocking with multi-forking application

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

libssl 1.1 blocking with multi-forking application

Daniel-Constantin Mierla

Hello,

we have been used libssl 0.9.x/1.0.x with no issue for more than 15 years in a multi-forking application, respectively Kamailio SIP server. The application is initializing in the main process (including loading data from database like mysql or postgress) then creates a pool of processes. We are setting our memory management functions as well as locking functions for 0.9.x/1.0.x.

With the internal changes in the libssl 1.1, the locking functions cannot be set any more. However, it appears that it ends up in a deadlock, or at least long time blocking. I received several such reports from different people running Debian stable. Few of them switched back to compile against libssl 1.0.x and no such issue happened again.

Has anyone else here experienced something similar?

For a better understanding of how we use libssl, here is what happens:

  1) kamailio starts and sets the memory management functions for libssl
  2) it loads data from backend/database into memory -- this can create connections to database servers using external libs (e.g., mysql) that may use libssl
  3) once the initialization is done (connections to backends should be also closed), it knows how many child processes will be forked and creates a dedicated SSL_CTX for each of them
  4) kamailio forks and each process is using its own SSL_CTX structure for accepting or connecting over tls -- each child process will also reconnect to backend, if it needs it for runtime
  5) after a while, several processes try to acquire same mutex inside libssl/libcrypto, but that seems to be already acquired -- I paste next two partial backtraces from different processes taken when blocking happen, the others are similar and the rest of processes are waiting for traffic on the network, their backtrace don't show they do anything with libssl at that moment -- see all at https://sip.antisip.com/kamailio-trap-tcp-down.txt

Note that we do not create threads in our application, but we cannot control if an used external library (e.g., mysql client) does it. Also, the tls connection can be used by different child processes.

Digging a bit in the libssl code, my first thoughts of a possible issue went to the type of locks created by libcrypto, because they are not process shared. Normally, operations to read/write for a connections should happen in one process, then when no traffic, the child process can move to another connection and handle traffic on it and other process can get to the previous connection once it has new traffic to handle.

Anyone having hints about what can be wrong there? Is libssl 1.1 supposed to be initialized/used in a different way for a multi-forking application with use of a tls connection between child processes?

Thanks,
Daniel

#0  0x00007ff8eedb7470 in futex_wait (private=<optimized out>, expected=18780, futex_word=0x7ff8de86130c) at ../sysdeps/unix/sysv/linux/futex-internal.h:61
        __ret = -512
        err = <optimized out>
#1  futex_wait_simple (private=<optimized out>, expected=18780, futex_word=0x7ff8de86130c) at ../sysdeps/nptl/futex-internal.h:135
No locals.
#2  __pthread_rwlock_wrlock_slow (rwlock=0x7ff8de861300) at pthread_rwlock_wrlock.c:67
        waitval = 18780
        result = 0
        futex_shared = <optimized out>
#3  0x00007ff8ef189ee9 in CRYPTO_THREAD_write_lock () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#4  0x00007ff8ef158c08 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#5  0x00007ff8ef4a3caf in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
No symbol table info available.
#6  0x00007ff8ef4994ff in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
No symbol table info available.
#7  0x00007ff8ef491f61 in SSL_do_handshake () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1


#0  0x00007ff8eedb7470 in futex_wait (private=<optimized out>, expected=918, futex_word=0x7ff8de86130c) at ../sysdeps/unix/sysv/linux/futex-internal.h:61
        __ret = -512
        err = <optimized out>
#1  futex_wait_simple (private=<optimized out>, expected=918, futex_word=0x7ff8de86130c) at ../sysdeps/nptl/futex-internal.h:135
No locals.
#2  __pthread_rwlock_wrlock_slow (rwlock=0x7ff8de861300) at pthread_rwlock_wrlock.c:67
        waitval = 918
        result = 0
        futex_shared = <optimized out>
#3  0x00007ff8ef189ee9 in CRYPTO_THREAD_write_lock () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#4  0x00007ff8ef158756 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#5  0x00007ff8ef4a7465 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
No symbol table info available.
#6  0x00007ff8ef4997ee in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
No symbol table info available.
#7  0x00007ff8ef491f61 in SSL_do_handshake () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
No symbol table info available.

-- 
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Reply | Threaded
Open this post in threaded view
|

RE: libssl 1.1 blocking with multi-forking application

Michael Wojcik
> From: openssl-users [mailto:[hidden email]] On Behalf Of Daniel-Constantin Mierla
> Sent: Monday, April 01, 2019 03:41
>
>  4) kamailio forks and each process is using its own SSL_CTX structure for accepting or
> connecting over tls -- each child process will also reconnect to backend, if it needs it for
> runtime

Unfortunately for your purposes, OpenSSL does not guarantee correct operation when called from the child side of a fork (prior to exec). See:

https://wiki.openssl.org/index.php/Libcrypto_API#Fork_Safety

That said, using OpenSSL in a non-exec'd child process is common enough that the wiki also has a page on reseeding the PRNG after a fork - no longer relevant with 1.1, with the PRNG rewrite, but it suggests that the issue was relatively common.

Based on the evidence (which isn't much), it appears at least one of the locks (implemented as pthread mutexes on Linux) was locked when you forked. Since libcrypto and libssl aren't built with symbols, we can't guess which lock.

I suggest you build OpenSSL for debugging so you can get some useful information at the point of failure, then try to trace it back to confirm what was locked after the fork (put a breakpoint on the child side of the fork, then inspect the mutex object at that point). Then you can try to figure out why it was locked when you forked.

OpenSSL 1.1.1b doesn't seem to provide a way to reinitialize the mutexes programmatically after a fork.

--
Michael Wojcik
Distinguished Engineer, Micro Focus