We are coding an engine to work with a crypto device we are developing.
Our crypto hardware work on a two level basis. We have the crypto
hardware direct connected to a soekris (net4801), running a custom
OpenBSD and a key management application. We already have a front end to
connect to this application.
The communication with the hardware is ok with an engine. All, the
server and the client were developed using openssl as a crypto library.
Our aim is to simulate a NetHSM environment. So the client connects to
the server using a SSL channel.
We are changing now our client application to adapt it to use with CA
applications already on market, principally those ones based on OpenSSL.
So we decided to write an engine to connect to our NetHSM. We started
basing our work on opensc and chil engines, due to the lack of
documentation on engine writing.
The connection between the engine and the NetHSM is based on a BIO SSL
structure. If we call this engine on OpenSSL command line as a dynaimc
one, it loads fine. We can even execute a "engine -t openhsmd" it
establishes the SSL conection by calling ENGINE_init and ENGINE_finish.
But when we call one of our functions, like random generation ("rand
-engine openhsmd 2"), the engine issues the following error: openssl
(lock_dbg_cb): already locked (mode=9, type=30) at eng_table.c:265. The
random function works ok on our client application.
We tried to find the error, but we don have an ideia of what is
happening. We noticed that this error occur when we call the
SSL_connect(). Porbably because we are trying to use ou engine to
connect to our engine, something like chicken-egg problem.
Does any one have an idea of what is really happening?