1.1.1a: crash in CRYPTO_THREAD_lock_free

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

1.1.1a: crash in CRYPTO_THREAD_lock_free

Claus Assmann
I'm trying OpenSSL 1.1.1a on FreeBSD 11.2-RELEASE-p4 and got the following
crash in one of my test programs (I compiled OpenSSL with -g after the
first time this happened to get at least some debug info):

#0  __je_huge_salloc (tsdn=<value optimized out>, ptr=<value optimized out>) at extent.h:114
#1  0x000000080122d01e in ifree (tsd=<value optimized out>) at arena.h:141468
#2  0x000000080122d5b1 in __free (ptr=0x800000000) at tsd.h:716
#3  0x0000000801535abb in _pthread_rwlock_destroy (rwlock=<value optimized out>)
   4at /usr/src/lib/libthr/thread/thr_rwlock.c:127
#4  0x0000000800e67c28 in CRYPTO_THREAD_lock_free (lock=0x801a27298) at crypto/threads_pthread.c:102
#5  0x0000000800dcb760 in EVP_PKEY_free (x=0x801a7b370) at crypto/evp/p_lib.c:601
#6  0x00000008008affce in ssl3_free (s=0x801bbd000) at ssl/s3_lib.c:3321
#7  0x0000000800904c91 in tls1_free (s=0x801bbd000) at ssl/t1_lib.c:115
#8  0x00000008008c085c in SSL_free (s=0x801bbd000) at ssl/ssl_lib.c:1204
#9  0x00000000004133d3 in sm_tlsbio_close (fp=0x6612e0, flags=0) at ../../mta/libmta/tlsbio.c:391
...
(gdb) p (pthread_rwlock_t)0x801a27298
$2 = (struct pthread_rwlock *) 0x801a27298
(gdb) p *$2
$3 = {lock = {rw_state = 1, rw_flags = 0, rw_blocked_readers = 1, rw_blocked_writers = 0, rw_spare = 0x801a272a8},
  owner = 27402512}


Since my program doesn't use pthreads, I compiled OpenSSL with no-threads
and the crash doesn't happen (surprise...).
Is this a bug in my application or in OpenSSL? I'm not sure how to debug
this any further (without going into the details of pthreads on that OS).
--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: 1.1.1a: crash in CRYPTO_THREAD_lock_free

Viktor Dukhovni
On Tue, Nov 27, 2018 at 04:11:06AM -0800, Claus Assmann wrote:

> I'm trying OpenSSL 1.1.1a on FreeBSD 11.2-RELEASE-p4 and got the following
> crash in one of my test programs (I compiled OpenSSL with -g after the
> first time this happened to get at least some debug info):
>
> #0  __je_huge_salloc (tsdn=<value optimized out>, ptr=<value optimized out>) at extent.h:114
> #1  0x000000080122d01e in ifree (tsd=<value optimized out>) at arena.h:141468
> #2  0x000000080122d5b1 in __free (ptr=0x800000000) at tsd.h:716
> #3  0x0000000801535abb in _pthread_rwlock_destroy (rwlock=<value optimized out>)
>    4at /usr/src/lib/libthr/thread/thr_rwlock.c:127
> #4  0x0000000800e67c28 in CRYPTO_THREAD_lock_free (lock=0x801a27298) at crypto/threads_pthread.c:102
> #5  0x0000000800dcb760 in EVP_PKEY_free (x=0x801a7b370) at crypto/evp/p_lib.c:601
> #6  0x00000008008affce in ssl3_free (s=0x801bbd000) at ssl/s3_lib.c:3321
> #7  0x0000000800904c91 in tls1_free (s=0x801bbd000) at ssl/t1_lib.c:115
> #8  0x00000008008c085c in SSL_free (s=0x801bbd000) at ssl/ssl_lib.c:1204
> #9  0x00000000004133d3 in sm_tlsbio_close (fp=0x6612e0, flags=0) at ../../mta/libmta/tlsbio.c:391
> ...
> (gdb) p (pthread_rwlock_t)0x801a27298
> $2 = (struct pthread_rwlock *) 0x801a27298
> (gdb) p *$2
> $3 = {lock = {rw_state = 1, rw_flags = 0, rw_blocked_readers = 1, rw_blocked_writers = 0, rw_spare = 0x801a272a8},
>   owner = 27402512}

I also use FreeBSD 11.2-RELEASE-p4, and have a /usr/src tree:

    110 int
    111 _pthread_rwlock_destroy (pthread_rwlock_t *rwlock)
    112 {
    113         pthread_rwlock_t prwlock;
    114         int ret;
    115
    116         prwlock = *rwlock;
    117         if (prwlock == THR_RWLOCK_INITIALIZER)
    118                 ret = 0;
    119         else if (prwlock == THR_RWLOCK_DESTROYED)
    120                 ret = EINVAL;
    121         else if (prwlock == THR_PSHARED_PTR) {
    122                 *rwlock = THR_RWLOCK_DESTROYED;
    123                 __thr_pshared_destroy(rwlock);
    124                 ret = 0;
    125         } else {
    126                 *rwlock = THR_RWLOCK_DESTROYED;
    127                 free(prwlock);
    128                 ret = 0;
    129         }
    130         return (ret);
    131 }

The first thing to note is that the function tkaes a *pointer* to
a pthread_rwlock_t, but the thing pointed to is itself a pointer.
It has three "magic" values, and otherwise points to allocated
storage, freed on line 127 (matching your stack trace).

The magic values are:

    #define   THR_RWLOCK_INITIALIZER          ((struct pthread_rwlock *)NULL)
    #define   THR_RWLOCK_DESTROYED            ((struct pthread_rwlock *)1)
    #define   THR_PSHARED_PTR                                         \
        ((void *)(uintptr_t)((1ULL << (NBBY * sizeof(long) - 1)) | 1))

But I think that your gdb commands to display the object are not
right.  We have

 lock = (pthread_rwlock_t *)0x801a27298
 *lock = prwlock = (struct pthread_rwlock_t *) 0x80000000

So you'd need to look at (**lock) not (*lock) to see the underlying
structure, but the address may be invalid.  I don't know how it
came to be 0x800000000 (2^35)...  I seems that something zeroed
the low 32 bits of the pointer.

More interesting that the lock might be the content of the EVP_PKEY
in frame #5:

    #5  0x0000000800dcb760 in EVP_PKEY_free (x=0x801a7b370) at crypto/evp/p_lib.c:601

    p ((struct evp_pkey_st *)0x801a7b370)[0]

Knowing what type of key this is might help to narrow the search.

--
        Viktor.
--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: 1.1.1a: crash in CRYPTO_THREAD_lock_free

Claus Assmann
Thanks for the reply, it helped me adding some more debugging
statements to various places to track down the problem:
it is due to a change in TLS session handling in TLSv1.3.
It seems there are multiple SSL_SESSION structures for a single SSL
connection (SMTP session).  The callback installed using
SSL_CTX_sess_set_new_cb() was called twice for the same SSL connection
and the code was written to handle only one callback per connection.
This resulted in a "use after free" situation. Sorry for the false
alarm.

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: 1.1.1a: crash in CRYPTO_THREAD_lock_free

Viktor Dukhovni
> On Nov 28, 2018, at 6:31 PM, Claus Assmann <[hidden email]> wrote:
>
> Thanks for the reply, it helped me adding some more debugging
> statements to various places to track down the problem:
> it is due to a change in TLS session handling in TLSv1.3.

Thanks for following up, much appreciated.

> It seems there are multiple SSL_SESSION structures for a single SSL
> connection (SMTP session).  The callback installed using
> SSL_CTX_sess_set_new_cb() was called twice for the same SSL connection
> and the code was written to handle only one callback per connection.

Yes, by default the OpenSSL library now issues two session tickets
for each full (non-resumed handshake), and a new session ticket
after each resumption.

Recent versions of the Postfix SMTP server, when linked against OpenSSL
1.1.1 ask the library to issue just one session ticket after a full
handshake:

  https://github.com/vdukhovni/postfix/blame/postfix-3.3/postfix/src/tls/tls_server.c#L507-L521

and none on resumption if the current ticket is still valid (existing
code from Postfix 2.11, which implements session ticket key sharing
and rotation for a pool of Postfix SMTP servers on a single host):

  https://github.com/vdukhovni/postfix/blame/postfix-3.3/postfix/src/tls/tls_server.c#L507-L521

Linkability of sessions is not in my view a concern for SMTP, and SMTP
clients either don't cache session tickets at all, or cache at most one,
so issuing two initially and avoiding re-use is largely wasteful.

> This resulted in a "use after free" situation. Sorry for the false
> alarm.

Makes sense.  I did look briefly for potential issues in the library,
that matched your stack trace (freeing the peer's DH public key), but
did not find any smoking gun.  I might however add belt-and-suspenders
safety in one code path were I think that the current safe behaviour
could prove fragile as the code evolves.  So it has not been entirely
unproductive.

--
        Viktor.

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users