Weird issue with 32 bit DLL involving FIPS

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Weird issue with 32 bit DLL involving FIPS

Glenn Smith
This post was updated on .
Ok, I've got a very weird issue involving OpenSSL 1.0.1e with FIPS.  When I run my test application which uses a dynamic dll, every other attempt will fail with this error:

error:2D09F086:FIPS routines:FIPS_digestupdate:selftest failed

This happens with both checked and free builds.

I'm using VS 2010 and the configuration line for OpenSSL reads:

perl Configure no-asm VC-WIN32 fips

I get the same results if no-asm is removed.  These builds are always on a 32 bit machine.

Now, if I build the same set of DLLs for 64 bit using a 64 bit machine, they always work on the 64 bit machine.  If I use the DLLs from the 32 bit machine with a 32 bit version of the test application, they always report the same error:2D09F086:FIPS routines:FIPS_digestupdate:selftest failed error.  However, if I build the 32 bit DLLs on the 64 bit machine, they will always run with the 32 bit application on the 64 bit machine, but always fail with the same error:2D09F086:FIPS routines:FIPS_digestupdate:selftest failed error on the 32 bit machines.

Bascially, here's the pattern - the 32 bit app is the same for all 32 bit tests - I'm just swapping the libeay32.dll.

64 bit App, 64 bit DLL - built on 64 bit machine - Free and Checked builds always work
32 bit App, 32 bit DLL - build on 64 bit machine - Free and Checked builds always work on 64 bit machine, fail on 32 bit machine
32 bit App, 32 bit DLL - build on 32 bit machine - Free and Checked builds always work on 50% of the time on 32 bit machine, fail on 64 bit machine

Both build machines should have the same version of VS 2010.  Perl is different because the 64 bit machine has Active Perl 64 bit and not 32 bit.

Nasm is the same .exe files

The test app bascially does this:

                // Enable FIPS
                if (!FIPS_mode())
                {
                        if (!FIPS_mode_set(1))
                        {
                                m_ErrorCode = ERR_get_error();
                                m_Fips = false;
                                FIPS_mode_set(0); // turn FIPS off because of error
                        }
                        else m_Fips = true;
                }
                else m_Fips = true;


This situation has me baffled.