Fingerprint mismatch only for 32-bit DLL linked statically to FIPS Capable OpenSSL

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Fingerprint mismatch only for 32-bit DLL linked statically to FIPS Capable OpenSSL

Dipak B

Appreciate any help on the following.

  1. Built OpenSSL Fips Module and then 'static binaries' of FIPS capable OSSL which 'statically link to the windows run-time'. Thus, my application binary (FipsApp.exe) does not depend on OSSL DLLs.

  2. Consumed these static binaries namely (libeaycompat32.liblibeayfips32.lib and ssleay32.lib) into myapp.dll using msincore.pl.

  3. FipsApp.exe calls function foo() inside myapp.dll which executes FIPS_mode_set() which returns (100:error:2D06B06F:lib(45):func(107): reason (111):/FIPS/FIPS.c:232)

Result

  1. On executing 64-bit FipsApp.exe, the FIPS mode gets set and working with 64-bit myapp.dll

  2. But on executing 32-bit FipsApp.exe which uses 32-bit myapp.dll with same configuration, FIPS_mode_set() fails with reason 111 (Fingerprint mismatch)

Attempted

Since above 32-bit myapp.dll did not work, some additional configuration changes were made.

  1. ReBuilt 32-bit myapp.dll with above LFLAGS "/DynamicBase:No /Fixed". Here default base address gets used for myapp.dll

  2. ReBuilt 32-bit myapp.dll with base address of 0xFB00000. (OSSL does same thing for FIPS dlls)

  3. Checking out following http://openssl.6102.n7.nabble.com/FIPS-Static-Library-linked-into-Win32-Dll-builds-but-fails-self-test-td63011.html

But 32-bit myapp DLL does always fail with fingerprint mismatch.

Question

How do I get 32-bit myapp.dll working in FIPS mode? FIPS_mode_set() returns (100:error:2D06B06F:lib(45):func(107): reason (111):/FIPS/FIPS.c:232)

Thanks.

Reply | Threaded
Open this post in threaded view
|

Re: Fingerprint mismatch only for 32-bit DLL linked statically to FIPS Capable OpenSSL

Neptune
I ran into the same issue on my FIPS journey a few years ago. I'm assuming
you are building for windows in which case setting the /FIXED flag is the
right thing to do, however you cannot be guaranteed to get the address you
specify - it may already be occupied in which case the dll will be re-based
and then you're bound to get a fingerprint mismatch.
I used the Process Explorer tool to verify that when my dll loaded, the
address it wanted was already occupied. I would suggest you do the same as a
first step to make sure this is actually the problem you are experiencing.
Unfortunately, I don't believe there is any way to guarantee that your dll
will not be re-based using FIPS in 32-bit Windows dlls...it's a bit of a
game of chance.



--
Sent from: http://openssl.6102.n7.nabble.com/OpenSSL-User-f3.html