ssh-keygen freezes if you force use of engine

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

ssh-keygen freezes if you force use of engine

Frederick Gotham

I have edited the source code for OpenSSL in the init functions so that the
config file always gets loaded. My config file specifies a custom engine, so
this means that every process that links with "libssl.so" or "libcrypto.so"
will use my custom engine.

The problem I'm having is that my device won't boot up. It gets as far as
running "ssh-keygen" and then it freezes.

In order to troubleshoot this, I recompiled "ssh-keygen" with the flag "-
rdynamic" so that I can see function names in a stack trace. Next I enabled
core dumps with "ulimit -c unlimited". Then I ran "ssh-keygen", and it froze
as expected, but this time I pressed "Ctrl + \", and so it produced a core
dump file.

I used GDB on the core file and saw the following backtrace:

Core was generated by `/home/service/ssh-keygen'.
Program terminated with signal SIGQUIT, Quit.
#0  0x00007f656e35e3c7 in __pthread_once_slow () from /lib/libpthread.so.0
(gdb) bt
#0  0x00007f656e35e3c7 in __pthread_once_slow () from /lib/libpthread.so.0
#1  0x00007f656f730fbe in CRYPTO_THREAD_run_once () from ./libcrypto.so.1.1
#2  0x00007f656f6d5c80 in OPENSSL_init_crypto () from ./libcrypto.so.1.1
#3  0x00007f656f66163b in openssl_config_int () from ./libcrypto.so.1.1
#4  0x00007f656f6d56cf in ossl_init_config_ossl_ () from ./libcrypto.so.1.1
#5  0x00007f656e35e407 in __pthread_once_slow () from /lib/libpthread.so.0
#6  0x00007f656f730fbe in CRYPTO_THREAD_run_once () from ./libcrypto.so.1.1
#7  0x00007f656f6d5c80 in OPENSSL_init_crypto () from ./libcrypto.so.1.1
#8  0x000055a8d125708b in main ()

Can anyone see what might be going on here?

Reply | Threaded
Open this post in threaded view
|

Re: ssh-keygen freezes if you force use of engine

Frederick Gotham


> I used GDB on the core file and saw the following backtrace:


I think I'm actually getting a stack overflow from "openssl_config_int"
calling "OPENSSL_init_crypto" because of the recursion.

Reply | Threaded
Open this post in threaded view
|

Re: ssh-keygen freezes if you force use of engine

Dmitry Belyavsky-3
Could you please reproduce this behavior without your patches?

On Wed, Nov 6, 2019 at 2:04 PM Frederick Gotham <[hidden email]> wrote:


> I used GDB on the core file and saw the following backtrace:


I think I'm actually getting a stack overflow from "openssl_config_int"
calling "OPENSSL_init_crypto" because of the recursion.



--
SY, Dmitry Belyavsky
Reply | Threaded
Open this post in threaded view
|

Re: ssh-keygen freezes if you force use of engine

Frederick Gotham
Dmitry Belyavsky <[hidden email]> wrote
in
news:CADqLbz+6WbCpday_1Gq-Zmst1y-17Hbc1S9OvYtE4=b=[hidden email]:

> Could you please reproduce this behavior without your patches?
>
> On Wed, Nov 6, 2019 at 2:04 PM Frederick Gotham
> <[hidden email]> wrote:
>
>>
>>
>> > I used GDB on the core file and saw the following backtrace:
>>
>>
>> I think I'm actually getting a stack overflow from "openssl_config_int"
>> calling "OPENSSL_init_crypto" because of the recursion.
>>
>>
>



My temporary solution is to link "ssh-keygen" statically with a normal
version of "libcrypto.a", and then have every other program link dynamically
with my altered version of "libcrypto.so".

I'll at least see if that works.

Reply | Threaded
Open this post in threaded view
|

Re: ssh-keygen freezes if you force use of engine

Frederick Gotham
 
> My temporary solution is to link "ssh-keygen" statically with a normal
> version of "libcrypto.a", and then have every other program link
> dynamically with my altered version of "libcrypto.so".
>
> I'll at least see if that works.


Yeah the problem is **definitely** that the function
"OPENSSL_init_crypto" is calling itself recursively.

I might have to put in a thread-duration variable to keep track of the
stack depth, and only tell it to load the config file on the first call.

I also tried playing around with OPENSSL_SSL_CLIENT_ENGINE_AUTO at compile
time, but I can't seem to get it to work.


Here's my latest stack trace when I link statically with libcrypto.a:

OPENSSL_init_crypto, with the following backtrace:
================== FUNCTION CALL STACK BACKTRACE ==================
/usr/lib/libcrypto.so.1.1(OPENSSL_init_crypto+0xb3) [0x7fce24d6ca6a]
/usr/lib/libcrypto.so.1.1(+0x143126) [0x7fce24d6a126]
/lib/libpthread.so.0(+0xf407) [0x7fce2460c407]
/usr/lib/libcrypto.so.1.1(CRYPTO_THREAD_run_once+0x9) [0x7fce24dc7fbe]
/usr/lib/libcrypto.so.1.1(+0x14317c) [0x7fce24d6a17c]
/usr/lib/libcrypto.so.1.1(CRYPTO_new_ex_data+0x1b) [0x7fce24d6a4d3]
/usr/lib/libcrypto.so.1.1(BIO_new+0x73) [0x7fce24cc8d37]
/usr/lib/libcrypto.so.1.1(BIO_new_fp+0x16) [0x7fce24ccca06]
openssl() [0x41fe6e]
/lib/libc.so.6(__libc_start_main+0xe7) [0x7fce2426ae77]
openssl() [0x42019a]
===================================================================
OPENSSL_init_crypto, with the following backtrace:
================== FUNCTION CALL STACK BACKTRACE ==================
/usr/lib/libcrypto.so.1.1(OPENSSL_init_crypto+0xb3) [0x7fce24d6ca6a]
/usr/lib/libcrypto.so.1.1(+0xd163b) [0x7fce24cf863b]
/usr/lib/libcrypto.so.1.1(+0x1456cf) [0x7fce24d6c6cf]
/lib/libpthread.so.0(+0xf407) [0x7fce2460c407]
/usr/lib/libcrypto.so.1.1(CRYPTO_THREAD_run_once+0x9) [0x7fce24dc7fbe]
/usr/lib/libcrypto.so.1.1(OPENSSL_init_crypto+0x2c9) [0x7fce24d6cc80]
/usr/lib/libcrypto.so.1.1(+0x143126) [0x7fce24d6a126]
/lib/libpthread.so.0(+0xf407) [0x7fce2460c407]
/usr/lib/libcrypto.so.1.1(CRYPTO_THREAD_run_once+0x9) [0x7fce24dc7fbe]
/usr/lib/libcrypto.so.1.1(+0x14317c) [0x7fce24d6a17c]
/usr/lib/libcrypto.so.1.1(CRYPTO_new_ex_data+0x1b) [0x7fce24d6a4d3]
/usr/lib/libcrypto.so.1.1(BIO_new+0x73) [0x7fce24cc8d37]
/usr/lib/libcrypto.so.1.1(BIO_new_fp+0x16) [0x7fce24ccca06]
openssl() [0x41fe6e]
/lib/libc.so.6(__libc_start_main+0xe7) [0x7fce2426ae77]
openssl() [0x42019a]




Reply | Threaded
Open this post in threaded view
|

Re: ssh-keygen freezes if you force use of engine

JordanBrown
In reply to this post by Frederick Gotham
On 11/6/2019 3:03 AM, Frederick Gotham wrote:
I used GDB on the core file and saw the following backtrace:

I think I'm actually getting a stack overflow from "openssl_config_int" 
calling "OPENSSL_init_crypto" because of the recursion.

No, a stack overflow would be much deeper.

#0  0x00007f656e35e3c7 in __pthread_once_slow () from /lib/libpthread.so.0
(gdb) bt
#0  0x00007f656e35e3c7 in __pthread_once_slow () from /lib/libpthread.so.0
#1  0x00007f656f730fbe in CRYPTO_THREAD_run_once () from ./libcrypto.so.1.1
#2  0x00007f656f6d5c80 in OPENSSL_init_crypto () from ./libcrypto.so.1.1
#3  0x00007f656f66163b in openssl_config_int () from ./libcrypto.so.1.1
#4  0x00007f656f6d56cf in ossl_init_config_ossl_ () from ./libcrypto.so.1.1
#5  0x00007f656e35e407 in __pthread_once_slow () from /lib/libpthread.so.0
#6  0x00007f656f730fbe in CRYPTO_THREAD_run_once () from ./libcrypto.so.1.1
#7  0x00007f656f6d5c80 in OPENSSL_init_crypto () from ./libcrypto.so.1.1
#8  0x000055a8d125708b in main ()

Note that __pthread_once_slow is in the stack twice, called from OPENSSL_init_crypto and CRYPTO_THREAD_run_once.

"once" functions ensure that they call their function argument exactly once, even if they are called multiple times in parallel while their function is running.  They do that by locking a mutex around the execution of the function.  The second call attempted to lock the mutex to wait for the first call to complete, and deadlocked.




-- 
Jordan Brown, Oracle ZFS Storage Appliance, Oracle Solaris
Reply | Threaded
Open this post in threaded view
|

Re: ssh-keygen freezes if you force use of engine

Frederick Gotham
Jordan Brown wrote:

> Note that __pthread_once_slow is in the stack twice, called from
> OPENSSL_init_crypto and CRYPTO_THREAD_run_once.
>
> "once" functions ensure that they call their function argument exactly
> once, even if they are called multiple times in parallel while their
> function is running.  They do that by locking a mutex around the
> execution of the function.  The second call attempted to lock the mutex
> to wait for the first call to complete, and deadlocked.



I still don't know what to do about this.

For them moment I'm continuing with my idea in the thread "Latest idea to
force usage of custom engine".

Reply | Threaded
Open this post in threaded view
|

Re: ssh-keygen freezes if you force use of engine

JordanBrown
On 11/6/2019 11:42 PM, Frederick Gotham wrote:
Jordan Brown wrote:

Note that __pthread_once_slow is in the stack twice, called from
OPENSSL_init_crypto and CRYPTO_THREAD_run_once.

"once" functions ensure that they call their function argument exactly
once, even if they are called multiple times in parallel while their
function is running.  They do that by locking a mutex around the
execution of the function.  The second call attempted to lock the mutex
to wait for the first call to complete, and deadlocked.


I still don't know what to do about this.

For them moment I'm continuing with my idea in the thread "Latest idea to 
force usage of custom engine".

Ensure that you don't end up calling the initialization functions recursively.

I'm not really in a position to help in any detail, either in terms of expertise or time.

-- 
Jordan Brown, Oracle ZFS Storage Appliance, Oracle Solaris