Plea for a new public OpenSSL RNG API

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
51 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Plea for a new public OpenSSL RNG API

Dr. Matthias St. Pierre
Hi everybody,

on the [openssl-dev] mailing list, there has been a long ongoing discussion about the new RAND_DRBG API and comparing it with the old RAND_METHOD API (see "[openssl-dev] Work on a new RNG for OpenSSL"). Two of the most controversal questions were:

 - Do we really need a new RNG API? Should the RAND_DRBG API be made public or kept private? (Currently, it's exported from libcrypto but only used internally by libssl.)
 - How much control should the user (programmer) be given over the reseeding process and/or should he be allowed to add his own additional randomness?

Many developers seem to be realizing the interesting possibilities of the DRBG API and are asking for public access to this new and promising API. One of the driving forces behind it is the question about how to do seeding and reseeding right. Among others, Uri Blumenthal asked for making the DRBG API public.

Currently, the OpenSSL core members seem to be reluctant to make the API public, at least at this early stage. I understand Rich Salz's viewpoint that this requires a thorough discussion, because a public interface can't be easily changed and wrong decisions in the early phase can become a heavy burdon.

Nevertheless, I agree with Uri Blumenthal that the DRBG API should be made public. So here comes my

======================================
Plea for a new public OpenSSL RNG API:
======================================

    The new RAND_DRBG is the superior API. It shouldn't be kept private and hidden behind the ancient RAND_METHOD API.
    The philosophy of the two APIs is not very well compatible, in particular when it comes to reseeding and adding
    additional unpredictable input. Hiding the RAND_DRBG behind the RAND_METHOD API only causes problems.
        Also, it will force people to patch their OpenSSL copy if they want to use the superior API.

    The RAND_DRBG API should become the new public OpenSSL RNG API and the old RAND_METHOD API should be deprecated
    in the long run. This transition does not need to be rushed, but it would be good if there would be early consent
    on the road map. I am thinking of a smooth transition with a phase of coexistence and a compatibility layer
    mapping the default RAND_METHOD to the default public RAND_DRBG instance. (This compatibility layer already exists,
    it's the 'RAND_OpenSSL()' method.)



Historical Background
=====================

As Rich already mentioned in his blog post, the RAND_DRBG isn't new. It's been a part of OpenSSL for a long time, hidden in the FIPS 2.0 Object Module.

I have been working with the FIPS DRBG for quite a while now, using a FIPS-capable OpenSSL 1.0.2x crypto library. The reason why our company switched to the FIPS DRBG is that one of our products runs on a small hardware device which does not have a reliable entropy source, but the product has to meet high security standards, in particular w.r.t. its RNG. So we decided to use the SmartCard RNG as primary entropy source for a deterministic AES-CTR based RNG and use /dev/urandom as additional input. Reseeding should occur on every generate request. Using the FIPS DRBG, these requirements were easily met, because the API gives such a fine grained control over reseeding and adding additional entropy.

The DRBG was well documented, its design in NIST SP800-90A (now: NIST SP800-90Ar1)  and its API in the OpenSSL FIPS 2.0 User Guide. The implementation was thoroughly tested and assessed during the FIPS certification process. So the only minor obstacle was that we had to patch the crypto library (not the FIPS object module) in order to get public access to the FIPS_drbg_*() methods.

I always considered the DRBG API more mature than the good old RAND_METHOD API and I wondered, why the DRBG code lay forgotten for so many years in the FIPS 2.0 object module sources and was never ported to master.

When in June of this year the thread "[openssl-dev] Work on a new RNG for OpenSSL" popped up (https://mta.openssl.org/pipermail/openssl-dev/2017-June), I closely watched the discussion, and when John Denker suggested having a look at NIST SP800-90A, I was electrified:

    > Constructive suggestion:  If you want to see what a RNG looks
    > like when designed by cryptographers, take a look at:
    >   Elaine Barker and John Kelsey,    >   "Recommendation for Random Number Generation Using Deterministic Random Bit Generators"
    >   http://csrc.nist.gov/publications/nistpubs/800-90A/SP800-90A.pdf
    >
    > That design may look complicated, but if you think you can
    > leave out some of the blocks in their diagram, proceed with
    > caution.  Every one of those blocks is there for a reason.

    <https://mta.openssl.org/pipermail/openssl-dev/2017-June/009423.html>

From his mail and the reaction to it, I had the impression that nobody seemed to remember the fact that the DRBG code was already present in the FIPS object module. The hidden treasure seemed forgotten! When I seized the opportunity and proposed to port the FIPS DRBG code into master, Rich Salz liked my idea and immediately started working on PR #3789:

       <https://mta.openssl.org/pipermail/openssl-dev/2017-June/009439.html>
       <https://mta.openssl.org/pipermail/openssl-dev/2017-June/009440.html>
       <https://github.com/openssl/openssl/pull/3789>

I am very grateful to Rich for picking up the idea so fast and and giving it so much speed and momentum. He has done a lot of work to convince others and defend the idea against initial scepticism by voices objecting to the seemingly new and unknown API. In the middle of August, when the first bulk of work was finished, Rich wrote a detailed blog post to advertise the new DRBG and explain his work <https://www.openssl.org/blog/blog/2017/08/12/random> and the discussion restarted:

     "[openssl-dev] Work on a new RNG for OpenSSL>  (see https://mta.openssl.org/pipermail/openssl-dev/2017-August)

It became quickly evident that users that the DRBG API had promising features, so they started asking for public access to this new API. The driving force was the question about how to do seeding and reseeding right. Among others, Uri Blumenthal was a dedicated advocate to make the DRBG API public (https://mta.openssl.org/pipermail/openssl-dev/2017-August/009594.html).

But the OpenSSL members currently seem to be reluctant to make the API public right away. I understand Rich's viewpoint that this decision requires a thorough discussion, because a public interface can't be easily changed and wrong decisions in the early phase can become a heavy burdon.


Nevertheless, I agree with Uri Blumenthal that the DRBG API should be made public. Here are some of my arguments for it.


The DRBG API supports multiple instances and chaining
=====================================================

The NIST DRBG standard had chaining of multiple DRBG instances in the mind from the very beginnning, see for example footnote [4] on page 25 of NIST SP800-90Ar1:

    > Entropy input may be obtained from an entropy source or an NRBG, both of which provide fresh entropy.
    > Entropy input could also be obtained from a DRBG that has access to an entropy source or NRBG.

The original OpenSSL FIPS DRBG implementation did not support chaining, but this support has been added by Rich during the DRBG port.



The DRBG API has a highly flexible concept for seeding and reseeding
====================================================================

As mentioned previously, the DRBG has a callback mechanism with callbacks like get_entropy() and get_adin() callbacks, which make it easy to fine tune the default instantiation process by either adding additional randomness input, or changing the entropy source entirely. The callbacks are primarily intended for obtaining randomness instantiation and reseeing. There is a clear concept for reseeding, which can be adjusted by changing the reseed_interval: Normally, the DRBG reseeds itself automatically, whenever the reseed_interval has been reached.

In addition, there is a way for the DRBG consumer to add his own unpredictability when requesting random bytes: by adding additional input 'adin' to the RAND_DRBG_generate() call:

    int RAND_DRBG_generate(RAND_DRBG *drbg, unsigned char *out, size_t outlen,
                           int prediction_resistance,
                           const unsigned char *adin, size_t adinlen)

So why are there so many ways to add randomness and additional input? And what is the difference?

* The get_entropy() and get_adin() callbacks are used by the DRBG itself to _pull_ unpredictable data from some backend entropy source (which can also be a chained DRBG which is connected to some entropy source) during instantiation or reseeding.
* The 'adin' argument can be used by the DRBG consumer to _push_ unpredictable input into the DRBG when generating random output.


Note that the DRBG consumer has no possibility to push out-of-band randomness into the DRBG. Adding randomness is always coupled with a generate or (re-)seed operation. This is a very import difference between the RAND_DRBG and the RAND_METHOD and one of the reasons why it's so hard to do reseeding right in both the RAND and RAND_DRBG API simultaneously. In fact, currently


The 'RAND_add()/RAND_bytes()' pattern is broken
===============================================

In OpenSSL, the classical way for the RNG consumer to add his own randomness is to call 'RAND_add()' before calling 'RAND_bytes()'. If the new 'RAND_OpenSSL()' method (the "compatibility layer" hiding the public RAND_DRBG instance)  is the default, then this does not work as expected anymore:

The reason is that a call to 'RAND_add()' adds the provided randomness only to a global buffer ('rand_bytes'), from which it will be pulled during the next reseed. But no reseed is triggered. So the next RAND_bytes() call will be unaffected from the RAND_add(), which is not what the consumer expected. (The same holds for 'RAND_seed()', since 'drbg_seed()' only calls into 'drbg_add()')

Reseeding of DRBGs occurs only at the following occasions:

* immediately after a 'fork()' (new)
* if the 'reseed_counter' exceeds the 'reseed_interval'
* if 'RAND_DRBG_generate()' is called requesting 'prediction_resistance'
* 'RAND_DRBG_reseed()' is called explicitely

*Note:* Currently it looks like the situation is even worse: if 'RAND_add()' is called multiple times before a reseed occurs, then the result of the previous call is overwritten.

Reseeding the 'DRBG' whenever the user calls 'RAND_add()' does not seem a good solution. It would be too expensive, in particular if system entropy is pulled for reseeding. Of course it is possible to fix this issue, but the DRBG provides for a much simpler solution: it lets the consumer contribute to the entropy of the internal state by providing additional input. If the user input contains entropy, that's fine, if it's "snake oil", no harm. The additional input is mixed into the internal state in just the same way as the entropy buffer using the 'ctr_df()' derivation function. One might think of the 'entropy' input as trusted randomness and 'adin' as untrusted randomness.

For this reason, I would like to see the 'RAND_add()/RAND_bytes()' pattern deprecated and the 'RAND_DRBG_generate() with additional input' pattern advertised instead.


The DRBG API supports different implementations
===============================================

Well, it _supported_ them, until recently. But that's not irreversible.

The DRBG concept, as layed out in the NIST standard, provides a generic framework for deterministic RNGs (the acronym DRBG stands for Deterministic Random Bit Generator). It deals with general questions like how to instantiate and reseed the RNG, where does it get the entropy from, etc.

The standard proposes three concrete implementations, Hash_DRBG, HMAC_DRBG, and CTR_DRBG. In the FIPS code, all three were implemented, and the genericity was achieved using a data union and a set of five function pointers, reminiscent of a vtable in C++:

    struct drbg_ctx_st
    {
        ...
       
        /* Implementation specific structures */
        union
            {
            DRBG_HASH_CTX hash;
            DRBG_HMAC_CTX hmac;
            DRBG_CTR_CTX  ctr;
            } d;
        /* Initialiase PRNG and setup callbacks below */
        int (*init)(DRBG_CTX *ctx, int nid, int security, unsigned int flags);
        /* Intantiate PRNG */
        int (*instantiate)(DRBG_CTX *ctx,
                    const unsigned char *ent, size_t entlen,
                    const unsigned char *nonce, size_t noncelen,
                    const unsigned char *pers, size_t perslen);
        /* reseed */
        int (*reseed)(DRBG_CTX *ctx,
                    const unsigned char *ent, size_t entlen,
                    const unsigned char *adin, size_t adinlen);
        /* generat output */
        int (*generate)(DRBG_CTX *ctx,
                    unsigned char *out, size_t outlen,
                    const unsigned char *adin, size_t adinlen);
        /* uninstantiate */
        int (*uninstantiate)(DRBG_CTX *ctx);

        ...
    };
   

This part of the code was removed during the DRBG port, because currently CTR_DRBG is the only impementation. I would like to suggest to restore this 'polymorphic' implementation, to ease adding new implementations CHACHA20_DRBG in the future


The DRBG API is well documented and tested
==========================================


The entire DRBG API is part of the OpenSSL FIPS 2.0 Module and as such is well tested and well documented. Most of the documentation for the manual pages still-to-be-written can be taken from the FIPS User Guide and converted into manual pages, starting with textual modifications like


        FIPS_drbg_xxx(...)  ->  RAND_DRBG_xxx(...)
        DRBG_CTX *dctx      ->  RAND_DRBG *dctx

and taking the new typedefs into consideration. Here is for example a comparison of the API function to install the callbacks:

    FIPS DRBG:
       
        int FIPS_drbg_set_callbacks(DRBG_CTX *dctx,
            size_t (*get_entropy)(DRBG_CTX *ctx,  <args> ),
            void (*cleanup_entropy)(DRBG_CTX *ctx, <args> ),
            size_t entropy_blocklen,
            size_t (*get_nonce)(DRBG_CTX *ctx, <args> ),
            void (*cleanup_nonce)(DRBG_CTX *ctx, <args>)
        );
   
   
    RAND_DRBG:
   
        typedef size_t (*RAND_DRBG_get_entropy_fn)(RAND_DRBG *ctx, <args> );
        typedef void (*RAND_DRBG_cleanup_entropy_fn)(RAND_DRBG *ctx, <args> );
        typedef size_t (*RAND_DRBG_get_nonce_fn)(RAND_DRBG *ctx,  <args> );
        typedef void (*RAND_DRBG_cleanup_nonce_fn)(RAND_DRBG *ctx,  <args> );
       
        int RAND_DRBG_set_callbacks(RAND_DRBG *dctx,
                                    RAND_DRBG_get_entropy_fn get_entropy,
                                    RAND_DRBG_cleanup_entropy_fn cleanup_entropy,
                                    RAND_DRBG_get_nonce_fn get_nonce,
                                    RAND_DRBG_cleanup_nonce_fn cleanup_nonce);
       




Conclusion
==========

I see no reason why the RAND_DRBG shouldn't be made public as soon as possible, keeping the API as close as possible to the original FIPS DRBG API (FIPS 3.0 is upcoming!). In a second step, the current compatibility binding from RAND_METHOD to RAND_DRBG could be deprecated and faded out smoothly.


Looking forward to receiving your comments. (But please be patient with me, I'm currently on physical rehab after a surgery.)


Matthias St. Pierre



--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Matt Caswell-2

On 29/08/17 10:45, Dr. Matthias St. Pierre wrote
> Currently, the OpenSSL core members seem to be reluctant to make the
> API public, at least at this early stage. I understand Rich Salz's
> viewpoint that this requires a thorough discussion, because a public
> interface can't be easily changed and wrong decisions in the early
> phase can become a heavy burdon.

FWIW, I am not against making the API public although I too understand
Rich's reluctance. However it seems inevitable to me that sooner or
later this API will have to be made accessible and I see little benefit
in delaying that and plenty of reasons for doing it now (it only
frustrates those that know the functionality is there but can't get at
it :-)). There is always a risk with every API added that you get it
wrong, and then you are stuck with it for ever more. The only real
answers to this are:

1) Make sure you "design" your API to start with and get it reviewed
2) Encourage people to try the API while it is still in dev i.e. so we
can still change it before the final release.

I'm not really convinced that keeping the API hidden now will mean that
it will be better in some future release. It just delays the point at
which we have the realisation of "if only we had designed it like *this*
it would have been soooo much better". Better is to make it public in
dev asap. If there are problems with it we can always revert it before
it makes it into a release.

> The 'RAND_add()/RAND_bytes()' pattern is broken
> ===============================================
>
> In OpenSSL, the classical way for the RNG consumer to add his own
> randomness is to call 'RAND_add()' before calling 'RAND_bytes()'. If
> the new 'RAND_OpenSSL()' method (the "compatibility layer" hiding the
> public RAND_DRBG instance)  is the default, then this does not work
> as expected anymore:
>
> The reason is that a call to 'RAND_add()' adds the provided
> randomness only to a global buffer ('rand_bytes'), from which it will
> be pulled during the next reseed. But no reseed is triggered. So the
> next RAND_bytes() call will be unaffected from the RAND_add(), which
> is not what the consumer expected. (The same holds for 'RAND_seed()',
> since 'drbg_seed()' only calls into 'drbg_add()')
>
> Reseeding of DRBGs occurs only at the following occasions:
>
> * immediately after a 'fork()' (new) * if the 'reseed_counter'
> exceeds the 'reseed_interval' * if 'RAND_DRBG_generate()' is called
> requesting 'prediction_resistance' * 'RAND_DRBG_reseed()' is called
> explicitely
>
> *Note:* Currently it looks like the situation is even worse: if
> 'RAND_add()' is called multiple times before a reseed occurs, then
> the result of the previous call is overwritten.
>
> Reseeding the 'DRBG' whenever the user calls 'RAND_add()' does not
> seem a good solution. It would be too expensive, in particular if
> system entropy is pulled for reseeding. Of course it is possible to
> fix this issue, but the DRBG provides for a much simpler solution: it
> lets the consumer contribute to the entropy of the internal state by
> providing additional input. If the user input contains entropy,
> that's fine, if it's "snake oil", no harm. The additional input is
> mixed into the internal state in just the same way as the entropy
> buffer using the 'ctr_df()' derivation function. One might think of
> the 'entropy' input as trusted randomness and 'adin' as untrusted
> randomness.
>
> For this reason, I would like to see the 'RAND_add()/RAND_bytes()'
> pattern deprecated and the 'RAND_DRBG_generate() with additional
> input' pattern advertised instead.

Deprecation itself would probably have to wait for a 1.2.0 release. We
have a binary/source compatibility guarantee for 1.1.0 which would not
be met for builds using "--api" or "no-deprecated".

Is there a potential security vulnerability here? Applications using the
"old" APIs expect RAND_add() to behave in a particular way. If we have
silently changed this behaviour in 1.1.1 are they exposed?

Matt
--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Dr. Matthias St. Pierre
> -----Ursprüngliche Nachricht-----

> Von: openssl-dev [mailto:[hidden email]] Im Auftrag von Matt Caswell
> Gesendet: Dienstag, 29. August 2017 12:17
> An: [hidden email]
> Betreff: Re: [openssl-dev] Plea for a new public OpenSSL RNG API
>
>
> On 29/08/17 10:45, Dr. Matthias St. Pierre wrote
> > ...
> > The 'RAND_add()/RAND_bytes()' pattern is broken
> > ===============================================
> >
> > In OpenSSL, the classical way for the RNG consumer to add his own
> > randomness is to call 'RAND_add()' before calling 'RAND_bytes()'. If
> > the new 'RAND_OpenSSL()' method (the "compatibility layer" hiding the
> > public RAND_DRBG instance)  is the default, then this does not work
> > as expected anymore:
> > ...
>
> Is there a potential security vulnerability here? Applications using the
> "old" APIs expect RAND_add() to behave in a particular way. If we have
> silently changed this behaviour in 1.1.1 are they exposed?
Don't worry, this issue is new, the global 'rand_bytes' buffer has only been introduced by the DRBG port to master in August. I don't think it's a big deal to fix it. The reason I mentioned it here was to emphasize, that it is really hard to get the different philosophies (push vs. pull) of the two APIs working together correctly. The code was reviewed by several people and nobody noticed it. By the way: the approach using the fixed size global 'rand_bytes' buffer has another issue, which I will try to write down on GitHub within the next days.

Matthias
 


--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Richard Levitte - VMS Whacker-2
In reply to this post by Dr. Matthias St. Pierre
I'm late in the game, having only followed the development very
superficially...

If I understand correctly, the RAND_DRBG API is really a completely
separate API that has nothing to do with the RAND_METHOD API pers se,
i.e. any association between the two is more or less "accidental"?

Frankly, I cannot see anything wrong with making that a public API.  I
wonder, though, if this is going to be an implementation that's kinda
sorta built-in only, or if other implementations of the basic stuff
will be possible...  or in other words, will we have a "method"
structure like we have with so many other APIs?  In saying this, I'm
counting in the possibility that some implementations could come in
the form of engines, is this something that's been thought about?  I
do notive the DRGB_CTX structure, but that doesn't quite follow the
usual "method" pattern we have...

Cheers,
Richard

In message <[hidden email]> on Tue, 29 Aug 2017 09:45:26 +0000, "Dr. Matthias St. Pierre" <[hidden email]> said:

Matthias.St.Pierre> Hi everybody,
Matthias.St.Pierre>
Matthias.St.Pierre> on the [openssl-dev] mailing list, there has been a long ongoing discussion about the new RAND_DRBG API and comparing it with the old RAND_METHOD API (see "[openssl-dev] Work on a new RNG for OpenSSL"). Two of the most controversal questions were:
Matthias.St.Pierre>
Matthias.St.Pierre>  - Do we really need a new RNG API? Should the RAND_DRBG API be made public or kept private? (Currently, it's exported from libcrypto but only used internally by libssl.)
Matthias.St.Pierre>  - How much control should the user (programmer) be given over the reseeding process and/or should he be allowed to add his own additional randomness?
Matthias.St.Pierre>
Matthias.St.Pierre> Many developers seem to be realizing the interesting possibilities of the DRBG API and are asking for public access to this new and promising API. One of the driving forces behind it is the question about how to do seeding and reseeding right. Among others, Uri Blumenthal asked for making the DRBG API public.
Matthias.St.Pierre>
Matthias.St.Pierre> Currently, the OpenSSL core members seem to be reluctant to make the API public, at least at this early stage. I understand Rich Salz's viewpoint that this requires a thorough discussion, because a public interface can't be easily changed and wrong decisions in the early phase can become a heavy burdon.
Matthias.St.Pierre>
Matthias.St.Pierre> Nevertheless, I agree with Uri Blumenthal that the DRBG API should be made public. So here comes my
Matthias.St.Pierre>
Matthias.St.Pierre> ======================================
Matthias.St.Pierre> Plea for a new public OpenSSL RNG API:
Matthias.St.Pierre> ======================================
Matthias.St.Pierre>
Matthias.St.Pierre>     The new RAND_DRBG is the superior API. It shouldn't be kept private and hidden behind the ancient RAND_METHOD API.
Matthias.St.Pierre>     The philosophy of the two APIs is not very well compatible, in particular when it comes to reseeding and adding
Matthias.St.Pierre>     additional unpredictable input. Hiding the RAND_DRBG behind the RAND_METHOD API only causes problems.
Matthias.St.Pierre> Also, it will force people to patch their OpenSSL copy if they want to use the superior API.
Matthias.St.Pierre>
Matthias.St.Pierre>     The RAND_DRBG API should become the new public OpenSSL RNG API and the old RAND_METHOD API should be deprecated
Matthias.St.Pierre>     in the long run. This transition does not need to be rushed, but it would be good if there would be early consent
Matthias.St.Pierre>     on the road map. I am thinking of a smooth transition with a phase of coexistence and a compatibility layer
Matthias.St.Pierre>     mapping the default RAND_METHOD to the default public RAND_DRBG instance. (This compatibility layer already exists,
Matthias.St.Pierre>     it's the 'RAND_OpenSSL()' method.)
Matthias.St.Pierre>
Matthias.St.Pierre>
Matthias.St.Pierre>
Matthias.St.Pierre> Historical Background
Matthias.St.Pierre> =====================
Matthias.St.Pierre>
Matthias.St.Pierre> As Rich already mentioned in his blog post, the RAND_DRBG isn't new. It's been a part of OpenSSL for a long time, hidden in the FIPS 2.0 Object Module.
Matthias.St.Pierre>
Matthias.St.Pierre> I have been working with the FIPS DRBG for quite a while now, using a FIPS-capable OpenSSL 1.0.2x crypto library. The reason why our company switched to the FIPS DRBG is that one of our products runs on a small hardware device which does not have a reliable entropy source, but the product has to meet high security standards, in particular w.r.t. its RNG. So we decided to use the SmartCard RNG as primary entropy source for a deterministic AES-CTR based RNG and use /dev/urandom as additional input. Reseeding should occur on every generate request. Using the FIPS DRBG, these requirements were easily met, because the API gives such a fine grained control over reseeding and adding additional entropy.
Matthias.St.Pierre>
Matthias.St.Pierre> The DRBG was well documented, its design in NIST SP800-90A (now: NIST SP800-90Ar1)  and its API in the OpenSSL FIPS 2.0 User Guide. The implementation was thoroughly tested and assessed during the FIPS certification process. So the only minor obstacle was that we had to patch the crypto library (not the FIPS object module) in order to get public access to the FIPS_drbg_*() methods.
Matthias.St.Pierre>
Matthias.St.Pierre> I always considered the DRBG API more mature than the good old RAND_METHOD API and I wondered, why the DRBG code lay forgotten for so many years in the FIPS 2.0 object module sources and was never ported to master.
Matthias.St.Pierre>
Matthias.St.Pierre> When in June of this year the thread "[openssl-dev] Work on a new RNG for OpenSSL" popped up (https://mta.openssl.org/pipermail/openssl-dev/2017-June), I closely watched the discussion, and when John Denker suggested having a look at NIST SP800-90A, I was electrified:
Matthias.St.Pierre>
Matthias.St.Pierre>     > Constructive suggestion:  If you want to see what a RNG looks
Matthias.St.Pierre>     > like when designed by cryptographers, take a look at:
Matthias.St.Pierre>     >   Elaine Barker and John Kelsey,    >   "Recommendation for Random Number Generation Using Deterministic Random Bit Generators"
Matthias.St.Pierre>     >   http://csrc.nist.gov/publications/nistpubs/800-90A/SP800-90A.pdf
Matthias.St.Pierre>     >
Matthias.St.Pierre>     > That design may look complicated, but if you think you can
Matthias.St.Pierre>     > leave out some of the blocks in their diagram, proceed with
Matthias.St.Pierre>     > caution.  Every one of those blocks is there for a reason.
Matthias.St.Pierre>
Matthias.St.Pierre>     <https://mta.openssl.org/pipermail/openssl-dev/2017-June/009423.html>
Matthias.St.Pierre>
Matthias.St.Pierre> From his mail and the reaction to it, I had the impression that nobody seemed to remember the fact that the DRBG code was already present in the FIPS object module. The hidden treasure seemed forgotten! When I seized the opportunity and proposed to port the FIPS DRBG code into master, Rich Salz liked my idea and immediately started working on PR #3789:
Matthias.St.Pierre>
Matthias.St.Pierre>        <https://mta.openssl.org/pipermail/openssl-dev/2017-June/009439.html>
Matthias.St.Pierre>        <https://mta.openssl.org/pipermail/openssl-dev/2017-June/009440.html>
Matthias.St.Pierre>        <https://github.com/openssl/openssl/pull/3789>
Matthias.St.Pierre>
Matthias.St.Pierre> I am very grateful to Rich for picking up the idea so fast and and giving it so much speed and momentum. He has done a lot of work to convince others and defend the idea against initial scepticism by voices objecting to the seemingly new and unknown API. In the middle of August, when the first bulk of work was finished, Rich wrote a detailed blog post to advertise the new DRBG and explain his work <https://www.openssl.org/blog/blog/2017/08/12/random> and the discussion restarted:
Matthias.St.Pierre>
Matthias.St.Pierre>      "[openssl-dev] Work on a new RNG for OpenSSL>  (see https://mta.openssl.org/pipermail/openssl-dev/2017-August)
Matthias.St.Pierre>
Matthias.St.Pierre> It became quickly evident that users that the DRBG API had promising features, so they started asking for public access to this new API. The driving force was the question about how to do seeding and reseeding right. Among others, Uri Blumenthal was a dedicated advocate to make the DRBG API public (https://mta.openssl.org/pipermail/openssl-dev/2017-August/009594.html).
Matthias.St.Pierre>
Matthias.St.Pierre> But the OpenSSL members currently seem to be reluctant to make the API public right away. I understand Rich's viewpoint that this decision requires a thorough discussion, because a public interface can't be easily changed and wrong decisions in the early phase can become a heavy burdon.
Matthias.St.Pierre>
Matthias.St.Pierre>
Matthias.St.Pierre> Nevertheless, I agree with Uri Blumenthal that the DRBG API should be made public. Here are some of my arguments for it.
Matthias.St.Pierre>
Matthias.St.Pierre>
Matthias.St.Pierre> The DRBG API supports multiple instances and chaining
Matthias.St.Pierre> =====================================================
Matthias.St.Pierre>
Matthias.St.Pierre> The NIST DRBG standard had chaining of multiple DRBG instances in the mind from the very beginnning, see for example footnote [4] on page 25 of NIST SP800-90Ar1:
Matthias.St.Pierre>
Matthias.St.Pierre>     > Entropy input may be obtained from an entropy source or an NRBG, both of which provide fresh entropy.
Matthias.St.Pierre>     > Entropy input could also be obtained from a DRBG that has access to an entropy source or NRBG.
Matthias.St.Pierre>
Matthias.St.Pierre> The original OpenSSL FIPS DRBG implementation did not support chaining, but this support has been added by Rich during the DRBG port.
Matthias.St.Pierre>
Matthias.St.Pierre>
Matthias.St.Pierre>
Matthias.St.Pierre> The DRBG API has a highly flexible concept for seeding and reseeding
Matthias.St.Pierre> ====================================================================
Matthias.St.Pierre>
Matthias.St.Pierre> As mentioned previously, the DRBG has a callback mechanism with callbacks like get_entropy() and get_adin() callbacks, which make it easy to fine tune the default instantiation process by either adding additional randomness input, or changing the entropy source entirely. The callbacks are primarily intended for obtaining randomness instantiation and reseeing. There is a clear concept for reseeding, which can be adjusted by changing the reseed_interval: Normally, the DRBG reseeds itself automatically, whenever the reseed_interval has been reached.
Matthias.St.Pierre>
Matthias.St.Pierre> In addition, there is a way for the DRBG consumer to add his own unpredictability when requesting random bytes: by adding additional input 'adin' to the RAND_DRBG_generate() call:
Matthias.St.Pierre>
Matthias.St.Pierre>     int RAND_DRBG_generate(RAND_DRBG *drbg, unsigned char *out, size_t outlen,
Matthias.St.Pierre>                            int prediction_resistance,
Matthias.St.Pierre>                            const unsigned char *adin, size_t adinlen)
Matthias.St.Pierre>
Matthias.St.Pierre> So why are there so many ways to add randomness and additional input? And what is the difference?
Matthias.St.Pierre>
Matthias.St.Pierre> * The get_entropy() and get_adin() callbacks are used by the DRBG itself to _pull_ unpredictable data from some backend entropy source (which can also be a chained DRBG which is connected to some entropy source) during instantiation or reseeding.
Matthias.St.Pierre> * The 'adin' argument can be used by the DRBG consumer to _push_ unpredictable input into the DRBG when generating random output.
Matthias.St.Pierre>
Matthias.St.Pierre>
Matthias.St.Pierre> Note that the DRBG consumer has no possibility to push out-of-band randomness into the DRBG. Adding randomness is always coupled with a generate or (re-)seed operation. This is a very import difference between the RAND_DRBG and the RAND_METHOD and one of the reasons why it's so hard to do reseeding right in both the RAND and RAND_DRBG API simultaneously. In fact, currently
Matthias.St.Pierre>
Matthias.St.Pierre>
Matthias.St.Pierre> The 'RAND_add()/RAND_bytes()' pattern is broken
Matthias.St.Pierre> ===============================================
Matthias.St.Pierre>
Matthias.St.Pierre> In OpenSSL, the classical way for the RNG consumer to add his own randomness is to call 'RAND_add()' before calling 'RAND_bytes()'. If the new 'RAND_OpenSSL()' method (the "compatibility layer" hiding the public RAND_DRBG instance)  is the default, then this does not work as expected anymore:
Matthias.St.Pierre>
Matthias.St.Pierre> The reason is that a call to 'RAND_add()' adds the provided randomness only to a global buffer ('rand_bytes'), from which it will be pulled during the next reseed. But no reseed is triggered. So the next RAND_bytes() call will be unaffected from the RAND_add(), which is not what the consumer expected. (The same holds for 'RAND_seed()', since 'drbg_seed()' only calls into 'drbg_add()')
Matthias.St.Pierre>
Matthias.St.Pierre> Reseeding of DRBGs occurs only at the following occasions:
Matthias.St.Pierre>
Matthias.St.Pierre> * immediately after a 'fork()' (new)
Matthias.St.Pierre> * if the 'reseed_counter' exceeds the 'reseed_interval'
Matthias.St.Pierre> * if 'RAND_DRBG_generate()' is called requesting 'prediction_resistance'
Matthias.St.Pierre> * 'RAND_DRBG_reseed()' is called explicitely
Matthias.St.Pierre>
Matthias.St.Pierre> *Note:* Currently it looks like the situation is even worse: if 'RAND_add()' is called multiple times before a reseed occurs, then the result of the previous call is overwritten.
Matthias.St.Pierre>
Matthias.St.Pierre> Reseeding the 'DRBG' whenever the user calls 'RAND_add()' does not seem a good solution. It would be too expensive, in particular if system entropy is pulled for reseeding. Of course it is possible to fix this issue, but the DRBG provides for a much simpler solution: it lets the consumer contribute to the entropy of the internal state by providing additional input. If the user input contains entropy, that's fine, if it's "snake oil", no harm. The additional input is mixed into the internal state in just the same way as the entropy buffer using the 'ctr_df()' derivation function. One might think of the 'entropy' input as trusted randomness and 'adin' as untrusted randomness.
Matthias.St.Pierre>
Matthias.St.Pierre> For this reason, I would like to see the 'RAND_add()/RAND_bytes()' pattern deprecated and the 'RAND_DRBG_generate() with additional input' pattern advertised instead.
Matthias.St.Pierre>
Matthias.St.Pierre>
Matthias.St.Pierre> The DRBG API supports different implementations
Matthias.St.Pierre> ===============================================
Matthias.St.Pierre>
Matthias.St.Pierre> Well, it _supported_ them, until recently. But that's not irreversible.
Matthias.St.Pierre>
Matthias.St.Pierre> The DRBG concept, as layed out in the NIST standard, provides a generic framework for deterministic RNGs (the acronym DRBG stands for Deterministic Random Bit Generator). It deals with general questions like how to instantiate and reseed the RNG, where does it get the entropy from, etc.
Matthias.St.Pierre>
Matthias.St.Pierre> The standard proposes three concrete implementations, Hash_DRBG, HMAC_DRBG, and CTR_DRBG. In the FIPS code, all three were implemented, and the genericity was achieved using a data union and a set of five function pointers, reminiscent of a vtable in C++:
Matthias.St.Pierre>
Matthias.St.Pierre>     struct drbg_ctx_st
Matthias.St.Pierre>     {
Matthias.St.Pierre>         ...
Matthias.St.Pierre>        
Matthias.St.Pierre>         /* Implementation specific structures */
Matthias.St.Pierre>         union
Matthias.St.Pierre>             {
Matthias.St.Pierre>             DRBG_HASH_CTX hash;
Matthias.St.Pierre>             DRBG_HMAC_CTX hmac;
Matthias.St.Pierre>             DRBG_CTR_CTX  ctr;
Matthias.St.Pierre>             } d;
Matthias.St.Pierre>         /* Initialiase PRNG and setup callbacks below */
Matthias.St.Pierre>         int (*init)(DRBG_CTX *ctx, int nid, int security, unsigned int flags);
Matthias.St.Pierre>         /* Intantiate PRNG */
Matthias.St.Pierre>         int (*instantiate)(DRBG_CTX *ctx,
Matthias.St.Pierre>                     const unsigned char *ent, size_t entlen,
Matthias.St.Pierre>                     const unsigned char *nonce, size_t noncelen,
Matthias.St.Pierre>                     const unsigned char *pers, size_t perslen);
Matthias.St.Pierre>         /* reseed */
Matthias.St.Pierre>         int (*reseed)(DRBG_CTX *ctx,
Matthias.St.Pierre>                     const unsigned char *ent, size_t entlen,
Matthias.St.Pierre>                     const unsigned char *adin, size_t adinlen);
Matthias.St.Pierre>         /* generat output */
Matthias.St.Pierre>         int (*generate)(DRBG_CTX *ctx,
Matthias.St.Pierre>                     unsigned char *out, size_t outlen,
Matthias.St.Pierre>                     const unsigned char *adin, size_t adinlen);
Matthias.St.Pierre>         /* uninstantiate */
Matthias.St.Pierre>         int (*uninstantiate)(DRBG_CTX *ctx);
Matthias.St.Pierre>
Matthias.St.Pierre>         ...
Matthias.St.Pierre>     };
Matthias.St.Pierre>    
Matthias.St.Pierre>
Matthias.St.Pierre> This part of the code was removed during the DRBG port, because currently CTR_DRBG is the only impementation. I would like to suggest to restore this 'polymorphic' implementation, to ease adding new implementations CHACHA20_DRBG in the future
Matthias.St.Pierre>
Matthias.St.Pierre>
Matthias.St.Pierre> The DRBG API is well documented and tested
Matthias.St.Pierre> ==========================================
Matthias.St.Pierre>
Matthias.St.Pierre>
Matthias.St.Pierre> The entire DRBG API is part of the OpenSSL FIPS 2.0 Module and as such is well tested and well documented. Most of the documentation for the manual pages still-to-be-written can be taken from the FIPS User Guide and converted into manual pages, starting with textual modifications like
Matthias.St.Pierre>
Matthias.St.Pierre>
Matthias.St.Pierre> FIPS_drbg_xxx(...)  ->  RAND_DRBG_xxx(...)
Matthias.St.Pierre> DRBG_CTX *dctx      ->  RAND_DRBG *dctx
Matthias.St.Pierre>
Matthias.St.Pierre> and taking the new typedefs into consideration. Here is for example a comparison of the API function to install the callbacks:
Matthias.St.Pierre>
Matthias.St.Pierre>     FIPS DRBG:
Matthias.St.Pierre>
Matthias.St.Pierre>         int FIPS_drbg_set_callbacks(DRBG_CTX *dctx,
Matthias.St.Pierre>             size_t (*get_entropy)(DRBG_CTX *ctx,  <args> ),
Matthias.St.Pierre>             void (*cleanup_entropy)(DRBG_CTX *ctx, <args> ),
Matthias.St.Pierre>             size_t entropy_blocklen,
Matthias.St.Pierre>             size_t (*get_nonce)(DRBG_CTX *ctx, <args> ),
Matthias.St.Pierre>             void (*cleanup_nonce)(DRBG_CTX *ctx, <args>)
Matthias.St.Pierre>         );
Matthias.St.Pierre>    
Matthias.St.Pierre>    
Matthias.St.Pierre>     RAND_DRBG:
Matthias.St.Pierre>    
Matthias.St.Pierre>         typedef size_t (*RAND_DRBG_get_entropy_fn)(RAND_DRBG *ctx, <args> );
Matthias.St.Pierre>         typedef void (*RAND_DRBG_cleanup_entropy_fn)(RAND_DRBG *ctx, <args> );
Matthias.St.Pierre>         typedef size_t (*RAND_DRBG_get_nonce_fn)(RAND_DRBG *ctx,  <args> );
Matthias.St.Pierre>         typedef void (*RAND_DRBG_cleanup_nonce_fn)(RAND_DRBG *ctx,  <args> );
Matthias.St.Pierre>        
Matthias.St.Pierre>         int RAND_DRBG_set_callbacks(RAND_DRBG *dctx,
Matthias.St.Pierre>                                     RAND_DRBG_get_entropy_fn get_entropy,
Matthias.St.Pierre>                                     RAND_DRBG_cleanup_entropy_fn cleanup_entropy,
Matthias.St.Pierre>                                     RAND_DRBG_get_nonce_fn get_nonce,
Matthias.St.Pierre>                                     RAND_DRBG_cleanup_nonce_fn cleanup_nonce);
Matthias.St.Pierre>        
Matthias.St.Pierre>
Matthias.St.Pierre>
Matthias.St.Pierre>
Matthias.St.Pierre>
Matthias.St.Pierre> Conclusion
Matthias.St.Pierre> ==========
Matthias.St.Pierre>
Matthias.St.Pierre> I see no reason why the RAND_DRBG shouldn't be made public as soon as possible, keeping the API as close as possible to the original FIPS DRBG API (FIPS 3.0 is upcoming!). In a second step, the current compatibility binding from RAND_METHOD to RAND_DRBG could be deprecated and faded out smoothly.
Matthias.St.Pierre>
Matthias.St.Pierre>
Matthias.St.Pierre> Looking forward to receiving your comments. (But please be patient with me, I'm currently on physical rehab after a surgery.)
Matthias.St.Pierre>
Matthias.St.Pierre>
Matthias.St.Pierre> Matthias St. Pierre
Matthias.St.Pierre>
Matthias.St.Pierre>
--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Dr. Matthias St. Pierre

> If I understand correctly, the RAND_DRBG API is really a completely
> separate API that has nothing to do with the RAND_METHOD API pers se,
> i.e. any association between the two is more or less "accidental"?

Yes, you're right. The original DRBG API was implemented for the FIPS object module (only) by Stephen Henson. A FIPS capable libcrypto 1.0.2y implements a binding from RAND_METHOD to the FIPS DRBG, FIPS_drbg_method(), which becomes the default RAND_METHOD when FIPS mode is enabled. So this was how the FIPS DRBG was 'plugged in' at runtime. *Note*: In those days, the FIPS DRBG pulled its entropy using the (non-fips) RAND_SSLeay() method, which was seeded using RAND_poll()/RAND_add(). Alltogether a really weird construction.

> Frankly, I cannot see anything wrong with making that a public API.  I
> wonder, though, if this is going to be an implementation that's kinda
> sorta built-in only, or if other implementations of the basic stuff
> will be possible...  or in other words, will we have a "method"
> structure like we have with so many other APIs?  In saying this, I'm
> counting in the possibility that some implementations could come in
> the form of engines, is this something that's been thought about?  I
> do notive the DRGB_CTX structure, but that doesn't quite follow the
> usual "method" pattern we have...

Currently it's only possible to customize the callbacks but not the deterministic algorithm. IMHO this is sufficient for the needs of a standard OpenSSL user who only wants control over the entropy source. A true new algorithm (like e.g. CHACHA2_DRBG) should be implemented by experts and added mainstream. So I don't see any advantage of having an engine over using the 'vtable' approach from the FIPS DRBG, which has been removed.


Regards,
Matthias


--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Richard Levitte - VMS Whacker-2
In message <[hidden email]> on Tue, 29 Aug 2017 13:05:21 +0000, "Dr. Matthias St. Pierre" <[hidden email]> said:

Matthias.St.Pierre> > Frankly, I cannot see anything wrong with making that a public API.  I
Matthias.St.Pierre> > wonder, though, if this is going to be an implementation that's kinda
Matthias.St.Pierre> > sorta built-in only, or if other implementations of the basic stuff
Matthias.St.Pierre> > will be possible...  or in other words, will we have a "method"
Matthias.St.Pierre> > structure like we have with so many other APIs?  In saying this, I'm
Matthias.St.Pierre> > counting in the possibility that some implementations could come in
Matthias.St.Pierre> > the form of engines, is this something that's been thought about?  I
Matthias.St.Pierre> > do notive the DRGB_CTX structure, but that doesn't quite follow the
Matthias.St.Pierre> > usual "method" pattern we have...
Matthias.St.Pierre>
Matthias.St.Pierre> Currently it's only possible to customize the callbacks but not the deterministic algorithm. IMHO this is sufficient for the needs of a standard OpenSSL user who only wants control over the entropy source. A true new algorithm (like e.g. CHACHA2_DRBG) should be implemented by experts and added mainstream. So I don't see any advantage of having an engine over using the 'vtable' approach from the FIPS DRBG, which has been removed.

Essentially, the argument for your last remark is in-structure vtable
vs refered to vtable.  I tend to prefer the latter (and that's the
usual OpenSSL pattern too, even though there are exceptions).

--
Richard Levitte         [hidden email]
OpenSSL Project         http://www.openssl.org/~levitte/
--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Dr. Matthias St. Pierre
> Essentially, the argument for your last remark is in-structure vtable
> vs refered to vtable.  I tend to prefer the latter (and that's the
> usual OpenSSL pattern too, even though there are exceptions).

You are the experts and much more familiar with the code then I am. My role was only to give the starting shot, the rest is up to you.

Regards,
Matthias



--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Richard Levitte - VMS Whacker-2
In message <[hidden email]> on Tue, 29 Aug 2017 13:27:20 +0000, "Dr. Matthias St. Pierre" <[hidden email]> said:

Matthias.St.Pierre> > Essentially, the argument for your last remark is in-structure vtable
Matthias.St.Pierre> > vs refered to vtable.  I tend to prefer the latter (and that's the
Matthias.St.Pierre> > usual OpenSSL pattern too, even though there are exceptions).
Matthias.St.Pierre>
Matthias.St.Pierre> You are the experts and much more familiar with
Matthias.St.Pierre> the code then I am. My role was only to give the
Matthias.St.Pierre> starting shot, the rest is up to you.

Fair enough!  :-)

--
Richard Levitte         [hidden email]
OpenSSL Project         http://www.openssl.org/~levitte/
--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Dr. Stephen Henson
In reply to this post by Richard Levitte - VMS Whacker-2
On Tue, Aug 29, 2017, Richard Levitte wrote:

> I'm late in the game, having only followed the development very
> superficially...
>
> If I understand correctly, the RAND_DRBG API is really a completely
> separate API that has nothing to do with the RAND_METHOD API pers se,
> i.e. any association between the two is more or less "accidental"?
>
> Frankly, I cannot see anything wrong with making that a public API.  I
> wonder, though, if this is going to be an implementation that's kinda
> sorta built-in only, or if other implementations of the basic stuff
> will be possible...  or in other words, will we have a "method"
> structure like we have with so many other APIs?  In saying this, I'm
> counting in the possibility that some implementations could come in
> the form of engines, is this something that's been thought about?  I
> do notive the DRGB_CTX structure, but that doesn't quite follow the
> usual "method" pattern we have...
>

In terms of future FIPS 140-2 compliance having a way to redirect all PRNG
operations to a FIPS compliant version is essential. This would require some
kind of method. We can redirect via RAND_METHOD but that currently means that
any ENGINE supplied PRNG is severely restricted (one global PRNG only)
compared to the built in DRBG. We could resolve that via a DRBG_METHOD or some
extended RAND_METHOD.

That's for the future though: there wont be a new FIPS module for some time
that will need this functionality.

In terms of the API itself. We could make RAND_DRBG public and I can see
reasons both for and against that. The main objection I can see is that it
commits us to DRBG like PRNGs: that is if we (or an ENGINE) in future wants to
use a RNG that isn't a good fit for the DRBG API it's a problem. We're
partially hitting that now because the DRBG API is not a good fit for the
older RAND API.

One alternative is to expand RAND_METHOD so it treats a DRBG as a special case
of the expanded API and it is flexible enough to handle any future needs. That
would (for example) have extended RAND APIs which can be passed an "instance"
of a PRNG (something like RAND_CTX which is analagous to the DRBG instance
structure we currently have).

That is tricky because RAND_METHOD is alas not an opaque structure and can't
be made so before OpenSSL 1.2.0. Tricky but not impossible.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Dr. Matthias St. Pierre
In reply to this post by Richard Levitte - VMS Whacker-2

> -----Ursprüngliche Nachricht-----
> Von: openssl-dev [mailto:[hidden email]] Im Auftrag von Richard Levitte
> Gesendet: Dienstag, 29. August 2017 15:29
> An: [hidden email]
> Betreff: Re: [openssl-dev] Plea for a new public OpenSSL RNG API
>
> In message <[hidden email]> on Tue, 29 Aug 2017 13:27:20 +0000, "Dr.
> Matthias St. Pierre" <[hidden email]> said:
>
> Matthias.St.Pierre> > Essentially, the argument for your last remark is in-structure vtable
> Matthias.St.Pierre> > vs refered to vtable.  I tend to prefer the latter (and that's the
> Matthias.St.Pierre> > usual OpenSSL pattern too, even though there are exceptions).
> Matthias.St.Pierre>
> Matthias.St.Pierre> You are the experts and much more familiar with
> Matthias.St.Pierre> the code then I am. My role was only to give the
> Matthias.St.Pierre> starting shot, the rest is up to you.
>
> Fair enough!  :-)

Just a sudden inspiration: If the RAND_DRBG becomes a truly independent API it might be better to strip the RAND_ prefix and redesign the API such that one has

- a DRBG_CTX     structure for the data members
- a DRBG_METHOD  structure for its methods

Would this look more OpenSSL-like to you?

Matthias


--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Richard Levitte - VMS Whacker-2
In message <[hidden email]> on Tue, 29 Aug 2017 13:39:02 +0000, "Dr. Matthias St. Pierre" <[hidden email]> said:

Matthias.St.Pierre> Just a sudden inspiration: If the RAND_DRBG becomes a truly independent API it might be better to strip the RAND_ prefix and redesign the API such that one has
Matthias.St.Pierre>
Matthias.St.Pierre> - a DRBG_CTX     structure for the data members
Matthias.St.Pierre> - a DRBG_METHOD  structure for its methods
Matthias.St.Pierre>
Matthias.St.Pierre> Would this look more OpenSSL-like to you?

Yes.  And per fairly recent recommendations to avoid cluttering the
name space, that would be OSSL_DRGB_CTX and OSSL_DRGB_METHOD, btw.

--
Richard Levitte         [hidden email]
OpenSSL Project         http://www.openssl.org/~levitte/
--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

OpenSSL - Dev mailing list
In reply to this post by Matt Caswell-2

    dev asap. If there are problems with it we can always revert it before
    it makes it into a release.
 
Someone on the OMC called that “flip-flopping” and seemed to be against this kind of thing.

If we are going to have an additional API, then we should at least see a draft of the header file first.

Keep in mind that the current DRBG organization might change, and we don’t want to lose that freedom.



--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

OpenSSL - Dev mailing list
In reply to this post by Richard Levitte - VMS Whacker-2
➢ Yes.  And per fairly recent recommendations to avoid cluttering the
➢     name space, that would be OSSL_DRGB_CTX and OSSL_DRGB_METHOD, btw.
   
DRBG is used for RAND stuff, so I don’t see a good reason to not using RAND_DRBG as the prefix


--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Matt Caswell-2
In reply to this post by OpenSSL - Dev mailing list


On 29/08/17 15:02, Salz, Rich via openssl-dev wrote:
>
>     dev asap. If there are problems with it we can always revert it before
>     it makes it into a release.
>  
> Someone on the OMC called that “flip-flopping” and seemed to be against this kind of thing.
>
> If we are going to have an additional API, then we should at least see a draft of the header file first.

Matthias, would you be willing to create such a draft as a github PR
(marked as WIP for now)?

Matt
--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Blumenthal, Uri - 0553 - MITLL
In reply to this post by OpenSSL - Dev mailing list
Could you please be more specific wrt. DRBG organization that in your opinion could impact the UI?

NIST 90Ar1 seems specific enough on what functionality DRBG can provide to users, and it doesn't seem like a very elaborate or rich interface. Why would it change, regardless of what you may need to do with it internally?

Regards,
Uri

Sent from my iPhone

> On Aug 29, 2017, at 10:03, Salz, Rich via openssl-dev <[hidden email]> wrote:
>
>
>    dev asap. If there are problems with it we can always revert it before
>    it makes it into a release.
>
> Someone on the OMC called that “flip-flopping” and seemed to be against this kind of thing.
>
> If we are going to have an additional API, then we should at least see a draft of the header file first.
>
> Keep in mind that the current DRBG organization might change, and we don’t want to lose that freedom.
>
>
>
> --
> openssl-dev mailing list
> To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

OpenSSL - Dev mailing list
In reply to this post by Matt Caswell-2
On 08/29/2017 05:17 AM, Matt Caswell wrote:
On 29/08/17 10:45, Dr. Matthias St. Pierre wrote
Currently, the OpenSSL core members seem to be reluctant to make the
API public, at least at this early stage. I understand Rich Salz's
viewpoint that this requires a thorough discussion, because a public
interface can't be easily changed and wrong decisions in the early
phase can become a heavy burdon.
FWIW, I am not against making the API public although I too understand
Rich's reluctance. However it seems inevitable to me that sooner or
later this API will have to be made accessible and I see little benefit
in delaying that and plenty of reasons for doing it now (it only
frustrates those that know the functionality is there but can't get at
it :-)). There is always a risk with every API added that you get it
wrong, and then you are stuck with it for ever more. The only real
answers to this are:

1) Make sure you "design" your API to start with and get it reviewed

Definitely.  It might be nice to see a high-level overview of what parts of the design changed while integrating the FIPS DRBG into the current form in master -- the global "rand_bytes" buffer and the removal of non-CTR implementation support are just a couple things that have been mentioned already in this thread.

2) Encourage people to try the API while it is still in dev i.e. so we
can still change it before the final release.

That also seems reasonable to me.
I might go further and also propose that we seek a more stringent review standard than usual for this work, since it is such a critical subsystem, and the current level of review let a few things slip through unnoticed.

I'm not really convinced that keeping the API hidden now will mean that
it will be better in some future release. It just delays the point at
which we have the realisation of "if only we had designed it like *this*
it would have been soooo much better". Better is to make it public in
dev asap. If there are problems with it we can always revert it before
it makes it into a release.

The 'RAND_add()/RAND_bytes()' pattern is broken 
===============================================

In OpenSSL, the classical way for the RNG consumer to add his own
randomness is to call 'RAND_add()' before calling 'RAND_bytes()'. If
the new 'RAND_OpenSSL()' method (the "compatibility layer" hiding the
public RAND_DRBG instance)  is the default, then this does not work
as expected anymore:

The reason is that a call to 'RAND_add()' adds the provided
randomness only to a global buffer ('rand_bytes'), from which it will
be pulled during the next reseed. But no reseed is triggered. So the
next RAND_bytes() call will be unaffected from the RAND_add(), which
is not what the consumer expected. (The same holds for 'RAND_seed()',
since 'drbg_seed()' only calls into 'drbg_add()')

Reseeding of DRBGs occurs only at the following occasions:

* immediately after a 'fork()' (new) * if the 'reseed_counter'
exceeds the 'reseed_interval' * if 'RAND_DRBG_generate()' is called
requesting 'prediction_resistance' * 'RAND_DRBG_reseed()' is called
explicitely

*Note:* Currently it looks like the situation is even worse: if
'RAND_add()' is called multiple times before a reseed occurs, then
the result of the previous call is overwritten.

Reseeding the 'DRBG' whenever the user calls 'RAND_add()' does not
seem a good solution. It would be too expensive, in particular if
system entropy is pulled for reseeding. Of course it is possible to
fix this issue, but the DRBG provides for a much simpler solution: it
lets the consumer contribute to the entropy of the internal state by
providing additional input. If the user input contains entropy,
that's fine, if it's "snake oil", no harm. The additional input is
mixed into the internal state in just the same way as the entropy
buffer using the 'ctr_df()' derivation function. One might think of
the 'entropy' input as trusted randomness and 'adin' as untrusted
randomness.

For this reason, I would like to see the 'RAND_add()/RAND_bytes()'
pattern deprecated and the 'RAND_DRBG_generate() with additional
input' pattern advertised instead.
Deprecation itself would probably have to wait for a 1.2.0 release. We
have a binary/source compatibility guarantee for 1.1.0 which would not
be met for builds using "--api" or "no-deprecated".

Is there a potential security vulnerability here? Applications using the
"old" APIs expect RAND_add() to behave in a particular way. If we have
silently changed this behaviour in 1.1.1 are they exposed?


Is there potential for vulnerability here?  Yes.  Whether there is actual vulnerability would require some careful analysis, and potentially input from people like Uri who have requirements for including TRBG input.

-Ben

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

OpenSSL - Dev mailing list
In reply to this post by Blumenthal, Uri - 0553 - MITLL

    Could you please be more specific wrt. DRBG organization that in your opinion could impact the UI?

From your use-case:  you want to add entropy into a specific DRBG.  You want to push it, as opposed to the DRBG “pull when needed” model.  That’s an additional API.  Also from your use-case: you want to specify which DRBG instance gets that entropy.  If we move to a pair per thread, as opposed to one per SSL and two in the global space, how do we make sure that API still works and does the right thing.

Does that makes sense, and does it answer your question?


--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Kurt Roeckx
In reply to this post by Dr. Matthias St. Pierre
I actually plan to work on most of not all the things you mention.
This will probably result in the API changing before it's made
public. I'm probably slow, so please be patient.


Kurt

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Kurt Roeckx
In reply to this post by Dr. Matthias St. Pierre
On Tue, Aug 29, 2017 at 11:31:03AM +0000, Dr. Matthias St. Pierre wrote:

> > -----Ursprüngliche Nachricht-----
> > Von: openssl-dev [mailto:[hidden email]] Im Auftrag von Matt Caswell
> > Gesendet: Dienstag, 29. August 2017 12:17
> > An: [hidden email]
> > Betreff: Re: [openssl-dev] Plea for a new public OpenSSL RNG API
> >
> >
> > On 29/08/17 10:45, Dr. Matthias St. Pierre wrote
> > > ...
> > > The 'RAND_add()/RAND_bytes()' pattern is broken
> > > ===============================================
> > >
> > > In OpenSSL, the classical way for the RNG consumer to add his own
> > > randomness is to call 'RAND_add()' before calling 'RAND_bytes()'. If
> > > the new 'RAND_OpenSSL()' method (the "compatibility layer" hiding the
> > > public RAND_DRBG instance)  is the default, then this does not work
> > > as expected anymore:
> > > ...
> >
> > Is there a potential security vulnerability here? Applications using the
> > "old" APIs expect RAND_add() to behave in a particular way. If we have
> > silently changed this behaviour in 1.1.1 are they exposed?
>
> Don't worry, this issue is new, the global 'rand_bytes' buffer has only been introduced by the DRBG port to master in August. I don't think it's a big deal to fix it. The reason I mentioned it here was to emphasize, that it is really hard to get the different philosophies (push vs. pull) of the two APIs working together correctly. The code was reviewed by several people and nobody noticed it. By the way: the approach using the fixed size global 'rand_bytes' buffer has another issue, which I will try to write down on GitHub within the next days.

I've actually noticed how this works and I have already partially
rewritten it, but I'm still not very happy about it. I think by
RAND_add() should not be called internally. But the question then
is what to do when an application calls RAND_add(), we should be
doing something with the buffer that's given. I think the best way
to deal with it is with the DRBG API is used, RAND_add() is used for
additional data.

We now have 2 global DRBGs, and I think I want to have 1 of them
chain to the other. RAND_add() could then be used for the master.

An other problem with the current implemenation is that the
randomness parameter that's now given to RAND_add() is just
ignored, it assumes it's the same as the length.


Kurt

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Kurt Roeckx
In reply to this post by Dr. Matthias St. Pierre
On Tue, Aug 29, 2017 at 01:05:21PM +0000, Dr. Matthias St. Pierre wrote:
>
> Currently it's only possible to customize the callbacks but not the deterministic algorithm. IMHO this is sufficient for the needs of a standard OpenSSL user who only wants control over the entropy source. A true new algorithm (like e.g. CHACHA2_DRBG) should be implemented by experts and added mainstream. So I don't see any advantage of having an engine over using the 'vtable' approach from the FIPS DRBG, which has been removed.

I've been looking at implementing a chacha20 based DRBG, which
isn't that hard. But there are various choices you can make, and
every chacha20 RNG that I've seen seems to take a random set of
those choices. I really wish there was some standardized version.


Kurt

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
123