OpenSSL 1.1.1g test failures

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

OpenSSL 1.1.1g test failures

David Harris
Environment: Windows 7 (I know, I know - I just hate Windows 10).
Compiler: Visual Studio, have tried both VS2008 Pro and VS2019 Pro
OpenSSL Build: 1.1.1g, retrieved from OpenSSL.org last night

I've been attempting to build OpenSSL 1.1.x since it came out, but each time I do so,
I find that, while it compiles and links cleanly, it fails about 50% of its self tests when
I perform "nmake test". It has been this way for several releases. By "fail" I mean
that there's a stream of "dubious..." outputs that look like this excerpt:

-------------------------- Cut here ----------------------------
...
test\recipes\03-test_internal_siphash.t ......... ok
test\recipes\03-test_internal_sm2.t ............. ok
test\recipes\03-test_internal_sm4.t ............. ok
test\recipes\03-test_internal_ssl_cert_table.t ..
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests
test\recipes\03-test_internal_x509.t ............ ok
test\recipes\03-test_ui.t ....................... ok
test\recipes\04-test_asn1_decode.t ..............
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests
test\recipes\04-test_asn1_encode.t ..............
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests
...
-------------------------- Cut here ----------------------------

Each time I went through the process, I saw the long string of self-test failures and
decided I'd put off migrating to 1.1.1 until it was sorted out, but it's been the same for
at least four releases now. I finally decided I needed to track down what was going
on, so I extrapolated how to run the failing tests manually with more verbose output
from the OpenSSL wiki pages (which are just a little out of date).

It appears that for at least the first twenty or thirty of these failures, the reason is
because the test application has been compiled without including the required
Applink code - a verbose output typically looks like this:

-------------------------- Cut here ----------------------------
O:\ >perl test\recipes\05-test_idea.t
1..1
OPENSSL_Uplink(5C790330,08): no OPENSSL_Applink
..\ideatest.exe => 1
not ok 1 - running ideatest
#   Failed test 'running ideatest'
#   at util/perl/OpenSSL/Test/Simple.pm line 77.
# Looks like you failed 1 test of 1.
-------------------------- Cut here ----------------------------

Is this just the way it is? I would have thought that 50% self-test failure would be
ringing alarm bells everywhere if it were common, so I can only conclude that there's
something odd about my environment, or that I'm doing something wrong, but this is
about as vanilla a build process as I can possibly make it. I follow the steps for
Win32 in INSTALL, and as I said at the start of this message, the nmake process
goes cleanly, not a single warning or error. The ONLY non-standard thing I do is
change the /MD switch (link to the DLL versions of the runtime libraries) to /MT
(static link the runtimes) because I don't want to have external dependencies in my
production environments (I lived in "DLL Hell" for so long that I'm now quite paranoid
about that). This change has never caused problems in the past, and doesn't seem
to be relevant to the problems I'm seeing.

I've been building OpenSSL myself for a number of years, most recently with the
end-of-life v1.0.2 builds, which always go without a hitch. As I remarked, I've been
putting off moving to v1.1.1 because I'm so uneasy about these self-test failures, but
I can't continue doing that any longer as TLS3 starts coming on stream.

Anyone have any insights into what I'm doing wrong, or what I can do about this? I'm
very reluctant to use the software in production if it can't pass its own self-test
regime, even if it appears to work normally otherwise.

Comments most welcome.

Cheers!

-- David --

Reply | Threaded
Open this post in threaded view
|

Re: OpenSSL 1.1.1g test failures

Matt Caswell-2


On 26/06/2020 10:45, David Harris wrote:

> Is this just the way it is? I would have thought that 50% self-test failure would be
> ringing alarm bells everywhere if it were common, so I can only conclude that there's
> something odd about my environment, or that I'm doing something wrong, but this is
> about as vanilla a build process as I can possibly make it.

No - this is not normal output. We would expect the self tests to pass
on Windows (and they do), e.g.

https://ci.appveyor.com/project/openssl/openssl/builds/33676970


> I follow the steps for
> Win32 in INSTALL, and as I said at the start of this message, the nmake process
> goes cleanly, not a single warning or error. The ONLY non-standard thing I do is
> change the /MD switch (link to the DLL versions of the runtime libraries) to /MT
> (static link the runtimes) because I don't want to have external dependencies in my
> production environments (I lived in "DLL Hell" for so long that I'm now quite paranoid
> about that). This change has never caused problems in the past, and doesn't seem
> to be relevant to the problems I'm seeing.
>

How exactly do you make this change? By editing the Makefile? Have you
tried it without doing this? My guess is that this is exactly the cause
of the problem. AppLink is all about dealing with differences in MS
runtimes.

Matt
Reply | Threaded
Open this post in threaded view
|

Re: OpenSSL 1.1.1g test failures

David Harris
On 26 Jun 2020 at 11:55, Matt Caswell wrote:

> No - this is not normal output. We would expect the self tests to pass
> on Windows

> >  The ONLY
> > non-standard thing I do is change the /MD switch (link to the DLL
> > versions of the runtime libraries) to /MT (static link the runtimes)
> > because I don't want to have external dependencies in my production
> > environments
>
> How exactly do you make this change? By editing the Makefile? Have you
> tried it without doing this? My guess is that this is exactly the
> cause of the problem. AppLink is all about dealing with differences in
> MS runtimes.

Assumption, as they say, is the mother of all fu??ups...

In this case, the failed assumption was that a non-standard modification I had
been making for many years would continue to work simply because it had in
the past.

Matt is, of course, quite right. When I changed the "/MT" back to "/MD" in
CNF_CFLAGS and rebuilt everything, it all worked like clockwork.

My thanks to you Matt - you've solved my problem.

Is there a standard (i.e, approved) way of using the static RTLs instead of the
DLL ones? Or is my only option to modify the applink code so that it checks its
environment in a different way? The problem with the dynamic RTLs is that my
application is often used in environments where the user may not have sufficient
rights to install the redistributables - whereas, if I use the static versions, the
code is a little bigger, but there's no redistributable installation required and I
never run into rights issues.

Again, thank you for the assistance, Matt - I appreciate it.

Cheers!

-- David --

Reply | Threaded
Open this post in threaded view
|

Re: OpenSSL 1.1.1g test failures

Richard Levitte - VMS Whacker-2
On Fri, 26 Jun 2020 16:01:09 +0200,
David Harris wrote:
> Is there a standard (i.e, approved) way of using the static RTLs instead of the
> DLL ones? Or is my only option to modify the applink code so that it checks its
> environment in a different way? The problem with the dynamic RTLs is that my
> application is often used in environments where the user may not have sufficient
> rights to install the redistributables - whereas, if I use the static versions, the
> code is a little bigger, but there's no redistributable installation required and I
> never run into rights issues.

The standard way of getting /MT is to configure 'no-shared', i.e. not
produce DLLs.

    perl Configure VC-WIN64A no-shared

However, if the main issue is *your* application, then the simplest
way is to link with the static library, regardless of configuration.
We do produce alongside static libraries alongside the DLLs since
1.1.1: libcrypto_static.lib.

Cheers,
Richard

--
Richard Levitte         [hidden email]
OpenSSL Project         http://www.openssl.org/~levitte/