Re: [openssl.org #2778] [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [openssl.org #2778] [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0

Rich Salz via RT
Don't know if this is related or not, but I'm also running a very similar test that uses TLS instead of DTLS, (same scenario, OpenSSL 1.0.1 with 1.0.0 Cipher Suites selected). That works fine, except that the 64 bit version of the test looks like it doesn't include the "Empty Fragments" security countermeasure, (at least the telltale 32 byte record at the start of each packet isn't there).

My config command is the same for 32/64 bits:

./config --prefix=../../ssl --openssldir=../../ssl/openssl  no-threads  no-shared  no-sse2 no-idea no-mdc2 no-rc5 $NO_HEARTBEATS

... where NO_HEARTBEATS="no-heartbeats" for OpenSSL 1.0.1 only, (option didn't exist for 1.0.0).

My understanding was that the "Empty Fragments" countermeasure should be "on" by default for both builds??



________________________________
 From: John Fitzgibbon <[hidden email]>
To: OpenSSL Response Team <[hidden email]>
Sent: Wednesday, March 28, 2012 2:37 PM
Subject: Re: [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0
 

Geez, not my best day... this *is* still a problem. I forgot to reset my test cases to use the 1.0.0 list of Cipher Suites. The alignment bug had nothing whatsoever to do with this particular OpenSSL failure, and it still fails even with my now-squeaky-clean test harness.


________________________________
 From: John Fitzgibbon <[hidden email]>
To: OpenSSL Response Team <[hidden email]>
Sent: Wednesday, March 28, 2012 2:29 PM
Subject: Re: [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0
 

Never mind... found a 64 bit memory alignment error in the test harness. I'm not entirely clear how/why the alignment problem was impacting OpenSSL, but with that bug fixed, the DTLS problem goes away.

Apologies for the false alarm,
John Fitzgibbon



________________________________
 From: John Fitzgibbon <[hidden email]>
To: OpenSSL Response Team <[hidden email]>
Sent: Wednesday, March 28, 2012 12:42 PM
Subject: [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0
 

Hi,
I'm trying to run a simple DTLS client/server test using OpenSSL 1.0.1, but with the same Cipher Suites that OpenSSL 1.0.0 uses, (to compare the two handshakes). This works fine with a 32 bit, (i686), build, but fails on 64 bit, (x86_64) with the following error:

d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0

If I do *not* override the default Cipher Suites, the 64 bit test works.

I've attached packet captures from the various tests:

dtls-assert-fail-x86_64.pcap -- failing handshake, (64 bit OpenSSL 1.0.1, using 1.0.0 Ciphers)
dtls-assert-ok-i686.pcap -- working handshake, (32 bit OpenSSL 1.0.1, using 1.0.0 Ciphers)
dtls-assert-ok-x86_64.pcap -- working handshake, (64
 bit OpenSSL 1.0.1, using 1.0.1 Ciphers)

Looking at the
 working/failing x86_64 pcaps, the MD differs for the chosen suite:
TLS_RSA_WITH_AES_256_CBC_SHA256 (works)
vs.
TLS_RSA_WITH_AES_256_CBC_SHA (fails)

My OpenSSL 1.0.1 code is built from the original sources, and linked directly into a test harness, with patches/overrides for the following:
1) The random number generator is controlled by the test harness
2) Time-of-day is controlled by the test harness
3) Memory allocation/freeing is handled by the test harness

The 64 bit code is built on Fedora 16 using:
gcc version 4.6.3 20120306 (Red Hat 4.6.3-2) (GCC)

I'd be interested to hear if other people experience the same problem with OpenSSL 1.0.1 x86_64 DTLS using TLS_RSA_WITH_AES_256_CBC_SHA, (or am I on my own here).

Thanks,
John Fitzgibbon

Don't know if this is related or not, but I'm also running a very similar test that uses TLS instead of DTLS, (same scenario, OpenSSL 1.0.1 with 1.0.0 Cipher Suites selected). That works fine, except that the 64 bit version of the test looks like it doesn't include the "Empty Fragments" security countermeasure, (at least the telltale 32 byte record at the start of each packet isn't there).

My config command is the same for 32/64 bits:

./config --prefix=../../ssl --openssldir=../../ssl/openssl  no-threads  no-shared  no-sse2 no-idea no-mdc2 no-rc5 $NO_HEARTBEATS

... where NO_HEARTBEATS="no-heartbeats" for OpenSSL 1.0.1 only, (option didn't exist for 1.0.0).

My understanding was that the "Empty Fragments" countermeasure should be "on" by default for both builds??



From: John Fitzgibbon <[hidden email]>
To: OpenSSL Response Team <[hidden email]>
Sent: Wednesday, March 28, 2012 2:37 PM
Subject: Re: [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0

Geez, not my best day... this *is* still a problem. I forgot to reset my test cases to use the 1.0.0 list of Cipher Suites. The alignment bug had nothing whatsoever to do with this particular OpenSSL failure, and it still fails even with my now-squeaky-clean test harness.


From: John Fitzgibbon <[hidden email]>
To: OpenSSL Response Team <[hidden email]>
Sent: Wednesday, March 28, 2012 2:29 PM
Subject: Re: [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0

Never mind... found a 64 bit memory alignment error in the test harness. I'm not entirely clear how/why the alignment problem was impacting OpenSSL, but with that bug fixed, the DTLS problem goes away.

Apologies for the false alarm,
John Fitzgibbon


From: John Fitzgibbon <[hidden email]>
To: OpenSSL Response Team <[hidden email]>
Sent: Wednesday, March 28, 2012 12:42 PM
Subject: [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0

Hi,
I'm trying to run a simple DTLS client/server test using OpenSSL 1.0.1, but with the same Cipher Suites that OpenSSL 1.0.0 uses, (to compare the two handshakes). This works fine with a 32 bit, (i686), build, but fails on 64 bit, (x86_64) with the following error:

d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0

If I do *not* override the default Cipher Suites, the 64 bit test works.

I've attached packet captures from the various tests:

dtls-assert-fail-x86_64.pcap -- failing handshake, (64 bit OpenSSL 1.0.1, using 1.0.0 Ciphers)
dtls-assert-ok-i686.pcap -- working handshake, (32 bit OpenSSL 1.0.1, using 1.0.0 Ciphers)
dtls-assert-ok-x86_64.pcap -- working handshake, (64 bit OpenSSL 1.0.1, using 1.0.1 Ciphers)

Looking at the working/failing x86_64 pcaps, the MD differs for the chosen suite:
TLS_RSA_WITH_AES_256_CBC_SHA256 (works)
vs.
TLS_RSA_WITH_AES_256_CBC_SHA (fails)

My OpenSSL 1.0.1 code is built from the original sources, and linked directly into a test harness, with patches/overrides for the following:
1) The random number generator is controlled by the test harness
2) Time-of-day is controlled by the test harness
3) Memory allocation/freeing is handled by the test harness

The 64 bit code is built on Fedora 16 using:
gcc version 4.6.3 20120306 (Red Hat 4.6.3-2) (GCC)

I'd be interested to hear if other people experience the same problem with OpenSSL 1.0.1 x86_64 DTLS using TLS_RSA_WITH_AES_256_CBC_SHA, (or am I on my own here).

Thanks,
John Fitzgibbon






Reply | Threaded
Open this post in threaded view
|

[openssl.org #2778] [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0

Rich Salz via RT
> [[hidden email] - Fri Mar 30 09:21:50 2012]:
>
> Don't know if this is related or not, but I'm also running a very
>    similar test that uses TLS instead of DTLS, (same scenario, OpenSSL
>    1.0.1 with 1.0.0 Cipher Suites selected). That works fine, except
>    that the 64 bit version of the test looks like it doesn't include
>    the "Empty Fragments" security countermeasure, (at least the
>    telltale 32 byte record at the start of each packet isn't there).
>

If you're using TLS v1.1 or 1.2 then you shouldn't encounter empty
fragments on any version as they are not required any more as CBC mode
includes an explicit IV.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [openssl.org #2778] [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0

john_fitzgibbon (Bugzilla)
>> the 64 bit version of the test looks like it doesn't include
>> the "Empty Fragments" security countermeasure >
> If you're using TLS v1.1 or 1.2 then you shouldn't encounter empty
> fragments on any version as they are not required any more as CBC mode
> includes an explicit IV.
The TLS tests are 1.0. The same code built on a 32 bit platform gives
different results, though the initial negotiation looks the same. I've
attached the 32/64 bit pcaps.

The tests that give problems, (DTLS and TLS), have three things in common:

1) They use OpenSSL 1.0.1
2) They are built on x86_64
3) They set the cipher list with a call to:
    SSL_CTX_set_cipher_list(ctx, [a very long cipher list])

The "very long cipher list" in question is a copy of this string:

static const char ssl_1_0_0_ciphers[]=
"ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:"
"DHE-RSA-CAMELLIA256-SHA:DHE-DSS-CAMELLIA256-SHA:ECDH-RSA-AES256-SHA:ECDH-ECDSA-AES256-SHA:"
"AES256-SHA:CAMELLIA256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:"
"EDH-RSA-DES-CBC3-SHA:EDH-DSS-DES-CBC3-SHA:ECDH-RSA-DES-CBC3-SHA:ECDH-ECDSA-DES-CBC3-SHA:"
"DES-CBC3-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:DHE-RSA-AES128-SHA:"
"DHE-DSS-AES128-SHA:DHE-RSA-SEED-SHA:DHE-DSS-SEED-SHA:DHE-RSA-CAMELLIA128-SHA:"
"DHE-DSS-CAMELLIA128-SHA:ECDH-RSA-AES128-SHA:ECDH-ECDSA-AES128-SHA:AES128-SHA:"
"SEED-SHA:CAMELLIA128-SHA:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:ECDH-RSA-RC4-SHA:"
"ECDH-ECDSA-RC4-SHA:RC4-SHA:RC4-MD5:EDH-RSA-DES-CBC-SHA:EDH-DSS-DES-CBC-SHA:DES-CBC-SHA:"
"EXP-EDH-RSA-DES-CBC-SHA:EXP-EDH-DSS-DES-CBC-SHA:EXP-DES-CBC-SHA:EXP-RC2-CBC-MD5:EXP-RC4-MD5";

If I do any of the following the tests work as expected:
- revert to OpenSSL 1.0.0h
- build 32 bit
- remove the cipher list, (use defaults)

tls-1.0-i686.pcap (10K) Download Attachment
tls-1.0-x86_64.pcap (9K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [openssl.org #2778] [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0

Rich Salz via RT
>> the 64 bit version of the test looks like it doesn't include
>> the "Empty Fragments" security countermeasure >
> If you're using TLS v1.1 or 1.2 then you shouldn't encounter empty
> fragments on any version as they are not required any more as CBC mode
> includes an explicit IV.
The TLS tests are 1.0. The same code built on a 32 bit platform gives
different results, though the initial negotiation looks the same. I've
attached the 32/64 bit pcaps.

The tests that give problems, (DTLS and TLS), have three things in common:

1) They use OpenSSL 1.0.1
2) They are built on x86_64
3) They set the cipher list with a call to:
    SSL_CTX_set_cipher_list(ctx, [a very long cipher list])

The "very long cipher list" in question is a copy of this string:

static const char ssl_1_0_0_ciphers[]=
"ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:"
"DHE-RSA-CAMELLIA256-SHA:DHE-DSS-CAMELLIA256-SHA:ECDH-RSA-AES256-SHA:ECDH-ECDSA-AES256-SHA:"
"AES256-SHA:CAMELLIA256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:"
"EDH-RSA-DES-CBC3-SHA:EDH-DSS-DES-CBC3-SHA:ECDH-RSA-DES-CBC3-SHA:ECDH-ECDSA-DES-CBC3-SHA:"
"DES-CBC3-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:DHE-RSA-AES128-SHA:"
"DHE-DSS-AES128-SHA:DHE-RSA-SEED-SHA:DHE-DSS-SEED-SHA:DHE-RSA-CAMELLIA128-SHA:"
"DHE-DSS-CAMELLIA128-SHA:ECDH-RSA-AES128-SHA:ECDH-ECDSA-AES128-SHA:AES128-SHA:"
"SEED-SHA:CAMELLIA128-SHA:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:ECDH-RSA-RC4-SHA:"
"ECDH-ECDSA-RC4-SHA:RC4-SHA:RC4-MD5:EDH-RSA-DES-CBC-SHA:EDH-DSS-DES-CBC-SHA:DES-CBC-SHA:"
"EXP-EDH-RSA-DES-CBC-SHA:EXP-EDH-DSS-DES-CBC-SHA:EXP-DES-CBC-SHA:EXP-RC2-CBC-MD5:EXP-RC4-MD5";

If I do any of the following the tests work as expected:
- revert to OpenSSL 1.0.0h
- build 32 bit
- remove the cipher list, (use defaults)


tls-1.0-i686.pcap (9K) Download Attachment
tls-1.0-x86_64.pcap (9K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [openssl.org #2778] [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0

john_fitzgibbon (Bugzilla)
In reply to this post by john_fitzgibbon (Bugzilla)
Some interesting observations:

1) Changed the cipher lists to much simpler values:
ciphers = "AES256-SHA256"   => works
ciphers = "AES256-SHA"   => fails

2) On a hunch, I tried adding "no-asm" to the config line:  2.1) TLS test now works and yields a perfect match with the 32 bit test
  2.2) DTLS test "works", but there is an interesting side-effect...

... the "random bytes" field differs in the server hello. There should be
no difference because the test harness is supplying a non-random PRG,
(and time-of-day is also strictly controlled).

I'm guessing that OpenSSL uses a SHA-based PRF on random bytes?

If so, then it looks like something might be up with 64-bit SHA, (or PRF??) in 1.0.1.

I've attached the "no-asm-64bit" pkt cap and the reference 32 bit equivalent.

dtls-no-asm-x86_64.pcap (18K) Download Attachment
dtls-i686.pcap (18K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [openssl.org #2778] [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0

Rich Salz via RT
Some interesting observations:

1) Changed the cipher lists to much simpler values:
ciphers = "AES256-SHA256"   => works
ciphers = "AES256-SHA"   => fails

2) On a hunch, I tried adding "no-asm" to the config line:  2.1) TLS test now works and yields a perfect match with the 32 bit test
  2.2) DTLS test "works", but there is an interesting side-effect...

... the "random bytes" field differs in the server hello. There should be
no difference because the test harness is supplying a non-random PRG,
(and time-of-day is also strictly controlled).

I'm guessing that OpenSSL uses a SHA-based PRF on random bytes?

If so, then it looks like something might be up with 64-bit SHA, (or PRF??) in 1.0.1.

I've attached the "no-asm-64bit" pkt cap and the reference 32 bit equivalent.

dtls-no-asm-x86_64.pcap (17K) Download Attachment
dtls-i686.pcap (17K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [openssl.org #2778] [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0

john_fitzgibbon (Bugzilla)
In reply to this post by john_fitzgibbon (Bugzilla)
> DTLS test "works", but the "random bytes" field differs in the server hello. There should be
> no difference because the test harness is supplying a non-random PRNG.

This is happening because of the following, (which looks like a bug), in ssl/d1_srvr.c, line 923:

                Time=(unsigned long)time(NULL);                 /* Time */
                l2n(Time,p);
                RAND_pseudo_bytes(p,SSL3_RANDOM_SIZE-sizeof(Time));


sizeof(Time) is 8 bytes in x86_84, but l2n() has only advanced the pointer 4 bytes.


This leaves 4 bytes of uninitialized data exposed in the random bytes field for x68_64.

Unless l2n() can do something different on other platforms, line 923 should be more explicit:

                RAND_pseudo_bytes(p,SSL3_RANDOM_SIZE-4);

With this fix, my "no-asm" 64bit DTLS test yields a pcap that matches the 32 bit build.

... that isolates the problem to the 64 bit assember changes between 1.0.0 and 1.0.1,
specifically, (it would seem), for SHA-1.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [openssl.org #2778] [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0

Rich Salz via RT
> DTLS test "works", but the "random bytes" field differs in the server hello. There should be
> no difference because the test harness is supplying a non-random PRNG.

This is happening because of the following, (which looks like a bug), in ssl/d1_srvr.c, line 923:

                Time=(unsigned long)time(NULL);                 /* Time */
                l2n(Time,p);
                RAND_pseudo_bytes(p,SSL3_RANDOM_SIZE-sizeof(Time));


sizeof(Time) is 8 bytes in x86_84, but l2n() has only advanced the pointer 4 bytes.


This leaves 4 bytes of uninitialized data exposed in the random bytes field for x68_64.

Unless l2n() can do something different on other platforms, line 923 should be more explicit:

                RAND_pseudo_bytes(p,SSL3_RANDOM_SIZE-4);

With this fix, my "no-asm" 64bit DTLS test yields a pcap that matches the 32 bit build.

... that isolates the problem to the 64 bit assember changes between 1.0.0 and 1.0.1,
specifically, (it would seem), for SHA-1.


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

[openssl.org #2778] [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0

Rich Salz via RT
> [[hidden email] - Sat Mar 31 07:50:09 2012]:
>
> This is happening because of the following, (which looks like a bug),
> in ssl/d1_srvr.c, line 923:
>
>                 Time=(unsigned long)time(NULL);                 /*
> Time */
>                 l2n(Time,p);
>                 RAND_pseudo_bytes(p,SSL3_RANDOM_SIZE-sizeof(Time));
>
>
> sizeof(Time) is 8 bytes in x86_84, but l2n() has only advanced the
> pointer 4 bytes.
>
>
> This leaves 4 bytes of uninitialized data exposed in the random bytes
> field for x68_64.
>
> Unless l2n() can do something different on other platforms, line 923
> should be more explicit:
>
>                 RAND_pseudo_bytes(p,SSL3_RANDOM_SIZE-4);
>

Fixed now, thanks for the report.

> With this fix, my "no-asm" 64bit DTLS test yields a pcap that matches
> the 32 bit build.
>
> ... that isolates the problem to the 64 bit assember changes between
> 1.0.0 and 1.0.1,
> specifically, (it would seem), for SHA-1.
>
>

Andy has made some recent fixes to the AES code too which may be
relevant. Please check the next snapshot to see if you still have problems.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [openssl.org #2778] [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0

john_fitzgibbon (Bugzilla)
> Andy has made some recent fixes to the AES code too which may be

> relevant. Please check the next snapshot to see if you still have problems.

I get the same results with openssl-1.0.1-stable-SNAP-20120403.tar.gz

To narrow down the problem, I built the "no-asm" version, (which works),
saved off the .o files from crypto/sha, then rebuilt with asm enabled.
I then replaced the sha object files with the no-asm flavor, edited
and rebuilt evp/e_aes_cbc_hmac_sha1.c, (to not use asm), and then
linked all that back together... in other words only SHA was non-asm.

That worked, which (again) points to the x86_64 sha-1 assembler.

Regards,
John

PS. For what it's worth, here are my CPU details:

[    0.002720] CPU: Physical Processor ID: 0
[    0.002721] CPU: Processor Core ID: 0
[    0.002724] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[    0.002725] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
[    0.002727] mce: CPU supports 9 MCE banks
[    0.002737] CPU0: Thermal monitoring enabled (TM1)
[    0.002744] using mwait in idle threads.
[    0.003826] ACPI: Core revision 20120111
[    0.006109] ftrace: allocating 22586 entries in 89 pages
[    0.016336] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.026335] CPU0: Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz stepping 07
[    0.127985] Performance Events: PEBS fmt1+, SandyBridge events, Intel PMU driver.
[    0.127989] PEBS disabled due to CPU errata.
[    0.127991] ... version:                3
[    0.127991] ... bit width:              48
[    0.127992] ... generic registers:      8
[    0.127993] ... value mask:             0000ffffffffffff
[    0.127994] ... max period:             000000007fffffff
[    0.127995] ... fixed-purpose events:   3
[    0.127996] ... event mask:             00000007000000ff
[    0.128088] NMI watchdog enabled, takes one hw-pmu counter.
[    0.128142] Booting Node   0, Processors  #1
[    0.128143] smpboot cpu 1: start_ip = 95000
[    0.159370] NMI watchdog enabled, takes one hw-pmu counter.
[    0.159433]  #2
[    0.159434] smpboot cpu 2: start_ip = 95000
[    0.190654] NMI watchdog enabled, takes one hw-pmu counter.
[    0.190714]  #3 Ok.
[    0.190715] smpboot cpu 3: start_ip = 95000
[    0.221937] NMI watchdog enabled, takes one hw-pmu counter.
[    0.221962] Brought up 4 CPUs
[    0.221964] Total of 4 processors activated (24741.86 BogoMIPS).
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [openssl.org #2778] [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0

Rich Salz via RT
> Andy has made some recent fixes to the AES code too which may be

> relevant. Please check the next snapshot to see if you still have problems.

I get the same results with openssl-1.0.1-stable-SNAP-20120403.tar.gz

To narrow down the problem, I built the "no-asm" version, (which works),
saved off the .o files from crypto/sha, then rebuilt with asm enabled.
I then replaced the sha object files with the no-asm flavor, edited
and rebuilt evp/e_aes_cbc_hmac_sha1.c, (to not use asm), and then
linked all that back together... in other words only SHA was non-asm.

That worked, which (again) points to the x86_64 sha-1 assembler.

Regards,
John

PS. For what it's worth, here are my CPU details:

[    0.002720] CPU: Physical Processor ID: 0
[    0.002721] CPU: Processor Core ID: 0
[    0.002724] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[    0.002725] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
[    0.002727] mce: CPU supports 9 MCE banks
[    0.002737] CPU0: Thermal monitoring enabled (TM1)
[    0.002744] using mwait in idle threads.
[    0.003826] ACPI: Core revision 20120111
[    0.006109] ftrace: allocating 22586 entries in 89 pages
[    0.016336] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.026335] CPU0: Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz stepping 07
[    0.127985] Performance Events: PEBS fmt1+, SandyBridge events, Intel PMU driver.
[    0.127989] PEBS disabled due to CPU errata.
[    0.127991] ... version:                3
[    0.127991] ... bit width:              48
[    0.127992] ... generic registers:      8
[    0.127993] ... value mask:             0000ffffffffffff
[    0.127994] ... max period:             000000007fffffff
[    0.127995] ... fixed-purpose events:   3
[    0.127996] ... event mask:             00000007000000ff
[    0.128088] NMI watchdog enabled, takes one hw-pmu counter.
[    0.128142] Booting Node   0, Processors  #1
[    0.128143] smpboot cpu 1: start_ip = 95000
[    0.159370] NMI watchdog enabled, takes one hw-pmu counter.
[    0.159433]  #2
[    0.159434] smpboot cpu 2: start_ip = 95000
[    0.190654] NMI watchdog enabled, takes one hw-pmu counter.
[    0.190714]  #3 Ok.
[    0.190715] smpboot cpu 3: start_ip = 95000
[    0.221937] NMI watchdog enabled, takes one hw-pmu counter.
[    0.221962] Brought up 4 CPUs
[    0.221964] Total of 4 processors activated (24741.86 BogoMIPS).


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

[openssl.org #2778] [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0

Rich Salz via RT
> [[hidden email] - Tue Apr 03 23:24:21 2012]:
>
> > Andy has made some recent fixes to the AES code too which may be
>
> > relevant. Please check the next snapshot to see if you still have
> problems.
>
> I get the same results with openssl-1.0.1-stable-SNAP-20120403.tar.gz
>
> To narrow down the problem, I built the "no-asm" version, (which
> works),
> saved off the .o files from crypto/sha, then rebuilt with asm enabled.
> I then replaced the sha object files with the no-asm flavor, edited
> and rebuilt evp/e_aes_cbc_hmac_sha1.c, (to not use asm), and then
> linked all that back together... in other words only SHA was non-asm.
>
> That worked, which (again) points to the x86_64 sha-1 assembler.
>

Please try setting the OPENSSL_ia32cap environment variable to 0 and see
if you still get the problem.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [openssl.org #2778] [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0

john_fitzgibbon (Bugzilla)
> Please try setting the OPENSSL_ia32cap environment variable to 0 and see
> if you still get the problem.

That worked.

I added code to the test harness to show the capability flags:

OPENSSL_ia32cap=[ffebffff:1fbae3ff]


I then added a command-line switch to change the flags on-the-fly.

Turning off the AES Instruction Set did the trick, (at least I think this is the AES bit):

-O 0xffebffff:0x1dbae3ff
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [openssl.org #2778] [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0

Rich Salz via RT
> Please try setting the OPENSSL_ia32cap environment variable to 0 and see
> if you still get the problem.

That worked.

I added code to the test harness to show the capability flags:

OPENSSL_ia32cap=[ffebffff:1fbae3ff]


I then added a command-line switch to change the flags on-the-fly.

Turning off the AES Instruction Set did the trick, (at least I think this is the AES bit):

-O 0xffebffff:0x1dbae3ff


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [openssl.org #2778] [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0

Rich Salz via RT
>> Please try setting the OPENSSL_ia32cap environment variable to 0 and see
>> if you still get the problem.
>
> That worked.

http://cvs.openssl.org/chngview?cn=22334 is interim solution, proper
solution will be provided at later point (if found appropriate).


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [openssl.org #2778] [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0

john_fitzgibbon (Bugzilla)
> http://cvs.openssl.org/chngview?cn=22334 is interim solution,
> proper solution will be provided at later point (if found appropriate).

Thanks, this circumvents the DTLS issue.

The TLS empty fragments issue remains, but this patch hints at
the cause. I think the problem is here, (s3_pkt.c, circa line 664):

if (    (sess == NULL) ||
        (s->enc_write_ctx == NULL) ||
        (EVP_MD_CTX_md(s->write_hash) == NULL))
        clear=1;

if (clear)
        mac_size=0;
else
        {
        mac_size=EVP_MD_CTX_size(s->write_hash);
        if (mac_size < 0)
                goto err;
        }

/* 'create_empty_fragment' is true only when this function calls itself */
if (!clear && !create_empty_fragment && !s->s3->empty_fragment_done)
        {
        /* countermeasure against known-IV weakness in CBC ciphersuites
          * (see http://www.openssl.org/~bodo/tls-cbc.txt) */


... If I'm reading things correctly, the cipher workarounds mean
EVP_MD_CTX_md(s->write_hash) is always NULL so this code skips the
empty fragments countermeasure. Debug printfs verify that "clear"
differs in good/bad test runs.

I'm guessing this test is here to prevent unwanted empty fragments
before the handshake is complete, but it looks like the logic is
flawed.

I notice similar logic in ssl3_get_record(), (unrelated to empty
fragments). That may be broken also.

Regards,
John
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [openssl.org #2778] [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0

Rich Salz via RT
> http://cvs.openssl.org/chngview?cn=22334 is interim solution,
> proper solution will be provided at later point (if found appropriate).

Thanks, this circumvents the DTLS issue.

The TLS empty fragments issue remains, but this patch hints at
the cause. I think the problem is here, (s3_pkt.c, circa line 664):

if (    (sess == NULL) ||
        (s->enc_write_ctx == NULL) ||
        (EVP_MD_CTX_md(s->write_hash) == NULL))
        clear=1;

if (clear)
        mac_size=0;
else
        {
        mac_size=EVP_MD_CTX_size(s->write_hash);
        if (mac_size < 0)
                goto err;
        }

/* 'create_empty_fragment' is true only when this function calls itself */
if (!clear && !create_empty_fragment && !s->s3->empty_fragment_done)
        {
        /* countermeasure against known-IV weakness in CBC ciphersuites
          * (see http://www.openssl.org/~bodo/tls-cbc.txt) */


... If I'm reading things correctly, the cipher workarounds mean
EVP_MD_CTX_md(s->write_hash) is always NULL so this code skips the
empty fragments countermeasure. Debug printfs verify that "clear"
differs in good/bad test runs.

I'm guessing this test is here to prevent unwanted empty fragments
before the handshake is complete, but it looks like the logic is
flawed.

I notice similar logic in ssl3_get_record(), (unrelated to empty
fragments). That may be broken also.

Regards,
John


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [openssl.org #2778] [BUG:] OpenSSL 1.0.1 x86_64: d1_pkt.c(444): OpenSSL internal error, assertion failed: t >= 0

Rich Salz via RT
>> http://cvs.openssl.org/chngview?cn=22334 is interim solution,
>> proper solution will be provided at later point (if found appropriate).
>
> Thanks, this circumvents the DTLS issue.
>
> The TLS empty fragments issue remains,

http://cvs.openssl.org/chngview?cn=22390


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]