Plea for a new public OpenSSL RNG API

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
51 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

OpenSSL - Dev mailing list
➢ An other problem with the current implemenation is that the
➢     randomness parameter that's now given to RAND_add() is just
➢     ignored, it assumes it's the same as the length.
   
For what it’s worth, this was done deliberately, make RAND_add and RAND_seed equivalent.

I am skeptical of the ability to get that estimate correct.

Someone on GH there is a conversation thread about turning that into a percentage, which seems like the best thing to do for any new API.


--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Blumenthal, Uri - 0553 - MITLL
On 8/29/17, 12:45, "openssl-dev on behalf of Salz, Rich via openssl-dev" <[hidden email] on behalf of [hidden email]> wrote:

    ➢ An other problem with the current implemenation is that the
    ➢ randomness parameter that's now given to RAND_add() is just
    ➢ ignored, it assumes it's the same as the length.
       
    For what it’s worth, this was done deliberately, make RAND_add and RAND_seed equivalent.
   
    I am skeptical of the ability to get that estimate correct.
   
    Someone on GH there is a conversation thread about turning that into a percentage, which seems like the best thing to do for any new API.


 What’s the point of having this potentially harmful parameter? If it weren’t ignored – how would OpenSSL use it?

If, based on its value, OpenSSL may decide that it now got “enough” entropy and doesn’t need to pull more from other sources before serving randomness to requestors – then it is harmful. “Over-confidence” in this value by the caller can negatively impact the quality of the produced random numbers.

If this value is not used to guide OpenSSL when to stop pulling entropy sources and start serving randomness – then it causes no harm, but what’s its purpose?

IMHO this interface is a way for the user to improve the quality of the randomness it would get from the given RNG, *not* to replace (or diminish) its other sources. My proposal is to abolish this parameter, especially since now it is simply ignored (and IMHO – for a good reason).



--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

OpenSSL - Dev mailing list
On 08/29/2017 01:50 PM, Blumenthal, Uri - 0553 - MITLL wrote:
IMHO this interface is a way for the user to improve the quality of the randomness it would get from the given RNG, *not* to replace (or diminish) its other sources. My proposal is to abolish this parameter, especially since now it is simply ignored (and IMHO – for a good reason).

That's a fine proposal ... it just can't be implemented until a major release boundary, when our ABI stability policy permits such breaking changes.

-Ben

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Richard Levitte - VMS Whacker-2
In reply to this post by Blumenthal, Uri - 0553 - MITLL
In message <[hidden email]> on Tue, 29 Aug 2017 15:06:12 +0000, "Blumenthal, Uri - 0553 - MITLL" <[hidden email]> said:

uri> Could you please be more specific wrt. DRBG organization that in your opinion could impact the UI?

Are you talking about the UI API or something else?

--
Richard Levitte         [hidden email]
OpenSSL Project         http://www.openssl.org/~levitte/
--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

OpenSSL - Dev mailing list
In reply to this post by Blumenthal, Uri - 0553 - MITLL
➢ IMHO this interface is a way for the user to improve the quality of the randomness it would get from the given RNG, *not* to replace (or diminish) its other sources. My proposal is to abolish this parameter, especially since now it is simply ignored (and IMHO – for a good reason).
   
We cannot do this in a minor release; we have to wait until 1.2

   
   

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Blumenthal, Uri - 0553 - MITLL
In reply to this post by Richard Levitte - VMS Whacker-2
    uri> Could you please be more specific wrt. DRBG organization that in your opinion could impact the UI?
   
    Are you talking about the UI API or something else?
   
No-no-no, just the UI API.

I used the term “UI” to emphasize that this is the “public” part of the API, exposed to the user, expected to be called by the user, and (probably) designed to be stable for some (hopefully considerable) time.


--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Blumenthal, Uri - 0553 - MITLL
In reply to this post by OpenSSL - Dev mailing list
IMHO this interface is a way for the user to improve the quality of the randomness it would get from the given RNG, *not* to replace (or diminish) its other sources. My proposal is to abolish this parameter, especially since now it is simply ignored (and IMHO – for a good reason).


That's a fine proposal ... it just can't be implemented until a major release boundary, when our ABI stability policy permits such breaking changes.

And that is fine. The sooner the better, but ABI stability makes sense too.

 

My main point is:  RAND_add() and whatever similar in purpose interface calls we may define in the future should exhibit the following behavior:

 

  1. Mix the provided randomness into the RNG state *immediately*, and
  2. Keep pulling other sources and mixing them into the state – don’t subtract from the “needed entropy” count the amount you presumably got from the user.

Frankly, the need to provide double entropy argument doesn’t bother me all that much – especially if the value 0 is accepted there. ;-)


--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Blumenthal, Uri - 0553 - MITLL
In reply to this post by OpenSSL - Dev mailing list
On 8/29/17, 11:33, "openssl-dev on behalf of Salz, Rich via openssl-dev" <[hidden email] on behalf of [hidden email]> wrote:
   
        Could you please be more specific wrt. DRBG organization that in your opinion could impact the UI?
   
 >   From your use-case:  you want to add entropy into a specific DRBG.

Yes.

  >  You want to push it, as opposed to the DRBG “pull when needed” model.  

I’d like to suggest that any approach other than “immediately mix the received randomness into the RNG state” is bad. If a user does RAND_add() now, as opposed to 100 source code lines before, there may be a reason for that.

Would you agree? Besides possible but probably negligible performance difference – why would you ever want to delay mixing the received “additional input” in?

  >  That’s an additional API.  

I wouldn’t call it “additional”. It may be a change from the current behavior – but I think everybody would welcome such a change. IMHO it can only help, never hurt.

   >  Also from your use-case: you want to specify which DRBG instance gets that entropy.

Yeah, but that probably isn’t a part of the API that DRBG “object” exposes.

One “API” is how to get a reference/pointer/instance/whatever to the DRBG object responsible for the operation I’m now concerned with, and that I want to influence/improve. This would probably differ between per-SSL DRBG and per-thread DRBG, etc. I haven’t even thought about this part of the API (and I’m sure others on the team have more experience and understanding of the OpenSSL code to do it better than I would anyway).

Another “API” is how to tell this specific DRBG instance what exactly I want from it now. E.g., mix more randomness that I provide into its state, give me some random bits, whatever. This part probably can stay close to what it currently is. I think 90A would be satisfied with 3-4 interface calls here…

   >   If we move to a pair per thread, as opposed to one per SSL and two in the global space, how do we make sure that API still works and does the right thing.

Yeah. That’s the “other” part of the API. I think the two API “groups” are pretty (completely?) orthogonal – independent from each other.
   
   >    Does that makes sense, and does it answer your question?
   
 Yeah… What do you think of my reasoning above?


--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

OpenSSL - Dev mailing list
➢ I’d like to suggest that any approach other than “immediately mix the received randomness into the RNG state” is bad. If a user does RAND_add() now, as opposed to 100 source code lines before, there may be a reason for that.
   
I think the only way to do that in the DRBG model is to treat it as “additional input” and do a generate call.  I think that would achieve the same effect, but it wouldn’t delay any possible seeding of randomness that the DRBG itself needs at some point.


➢ One “API” is how to get a reference/pointer/instance/whatever to the DRBG object responsible for the operation I’m now concerned with, and that I want to influence/improve. This would probably differ between per-SSL DRBG and per-thread DRBG, etc. I haven’t even thought about this part of the API (and I’m sure others on the team have more experience and understanding of the OpenSSL code to do it better than I would anyway).
   
  Yes, it is separate.  But I want to make sure that if we are doing a comprehensive RAND overhaul that this is included in the requirements.

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Richard Levitte - VMS Whacker-2
In reply to this post by Blumenthal, Uri - 0553 - MITLL
In message <[hidden email]> on Tue, 29 Aug 2017 19:04:48 +0000, "Blumenthal, Uri - 0553 - MITLL" <[hidden email]> said:

uri>     uri> Could you please be more specific wrt. DRBG organization that in your opinion could impact the UI?
uri>    
uri>     Are you talking about the UI API or something else?
uri>    
uri> No-no-no, just the UI API.
uri>
uri> I used the term “UI” to emphasize that this is the “public” part of the API, exposed to the user, expected to be called by the user, and (probably) designed to be stable for some (hopefully considerable) time.

Ah, ok.  In an OpenSSL context, this gets a bit confusing since there
is an API called UI ( <openssl/ui.h> ).

--
Richard Levitte         [hidden email]
OpenSSL Project         http://www.openssl.org/~levitte/
--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Blumenthal, Uri - 0553 - MITLL
In reply to this post by OpenSSL - Dev mailing list
On 8/29/17, 15:22, "openssl-dev on behalf of Salz, Rich via openssl-dev" <[hidden email] on behalf of [hidden email]> wrote:
    ➢ I’d like to suggest that any approach other than “immediately mix the received randomness into the RNG state” is bad. If a user does RAND_add() now, as opposed to 100 source code lines before, there may be a reason for that.
       
>    I think the only way to do that in the DRBG model is to treat it as “additional input” and
>    do a generate call.  I think that would achieve the same effect, but it wouldn’t delay any
>    possible seeding of randomness that the DRBG itself needs at some point.
   
That is OK if the application manually invokes generate(). If it’s the RNG responsible for feeding key generators (who would call generate() transparently to user), then the only way I see is DRBG reseed(). Even though it implies a complete state replacement.

AFAIK, there are three cases when randomness source(s) is pulled:

a. At the instantiation time (page 29 of NIST SP 800-90A Rev1 draft).
b. Upon reseed call (pages 30-31).
c. During generation call(pages 32-36).

“Additional input” (i.e., what I’m after :) is present in two of the above: reseeding, and generation (pages 19 and 23).

The draft is very clear that this “additional input” shall not be counted towards the “official” entropy (aka calculations when to pull the “official” entropy sources). Which suits my purposes just fine. ;-)

I realize that reseed() not only mixes my “additional input” but also replaces the entire state. NIST does not specify interface to “just” mix the “additional input” into the state without replacing the whole state with some fresh entropy by calling Get_entropy_input(). Maybe we can provide such a function call (that’s what I think RAND_add() is supposed to do), but I’m not certain here…

Wonder what others think? Do you believe that if you want to mix some more randomness into the RNG state to improve upon less-than-perfect entropy sources, you must also replace the entire state with the fresh pull from those sources? Or is it OK to just “spice” the current state with “additional input”?

Note that regardless of the answer to the above, IMHO reseed() should be able to take “additional input”. (NIST considers it optional, but sensitive/critical applications don’t. ;)
   
    ➢    One “API” is how to get a reference/pointer/instance/whatever to the DRBG object responsible for the operation I’m now concerned with,
➢  and that I want to influence/improve. This would probably differ between per-SSL DRBG and per-thread DRBG, etc. I haven’t even
➢  thought about this part of the API (and I’m sure others on the team have more experience and understanding of the OpenSSL code
➢  to do it better than I would anyway).
       
 >     Yes, it is separate.  But I want to make sure that if we are doing a comprehensive RAND overhaul that this is included in the requirements.
   
Yes, it should be included. But don’t let it stop us from defining/reviewing the “clearer” part of the API.  ;-)

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Kurt Roeckx
In reply to this post by Blumenthal, Uri - 0553 - MITLL
On Tue, Aug 29, 2017 at 06:50:37PM +0000, Blumenthal, Uri - 0553 - MITLL wrote:

> On 8/29/17, 12:45, "openssl-dev on behalf of Salz, Rich via openssl-dev" <[hidden email] on behalf of [hidden email]> wrote:
>
>     ➢ An other problem with the current implemenation is that the
>     ➢ randomness parameter that's now given to RAND_add() is just
>     ➢ ignored, it assumes it's the same as the length.
>        
>     For what it’s worth, this was done deliberately, make RAND_add and RAND_seed equivalent.
>    
>     I am skeptical of the ability to get that estimate correct.
>    
>     Someone on GH there is a conversation thread about turning that into a percentage, which seems like the best thing to do for any new API.
>
>
>  What’s the point of having this potentially harmful parameter? If it weren’t ignored – how would OpenSSL use it?
>
> If, based on its value, OpenSSL may decide that it now got “enough” entropy and doesn’t need to pull more from other sources before serving randomness to requestors – then it is harmful. “Over-confidence” in this value by the caller can negatively impact the quality of the produced random numbers.

As long as you have sources that don't provide 1 bit of randomness
per bit that you provide you need to have an estimate of how much
randomness it really contains. And you should probably seriously
underestimate it so that you're sure that you collect enough.

The problem with ignoring an existing parameter is that people
could be calling it with for instance the value of 0, knowing it
contains as good as none entropy. Or they could feed the unwithened
output of an TRNG in that with an estimate of randomness it provides.
And OpenSSL used to do the right thing with that.

But now we just ignore it and assume every bit with get contains 1
bit of randomness and we're sundenly seriously overestimating the
amount of randomness we're getting. This is a documented public API,
you can't just go and ignore this parameter.


Kurt

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Blumenthal, Uri - 0553 - MITLL
> If, based on its value, OpenSSL may decide that it now got “enough” entropy and doesn’t need to
> pull more from other sources before serving randomness to requestors – then it is harmful.
> “Over-confidence” in this value by the caller can negatively impact the quality of the produced
> random numbers.
   
    As long as you have sources that don't provide 1 bit of randomness
    per bit that you provide you need to have an estimate of how much
    randomness it really contains. And you should probably seriously
    underestimate it so that you're sure that you collect enough.

So let me underestimate it to 0. ;-)  
 
    The problem with ignoring an existing parameter is that people
    could be calling it with for instance the value of 0, knowing it
    contains as good as none entropy. Or they could feed the unwithened
    output of an TRNG in that with an estimate of randomness it provides.
    And OpenSSL used to do the right thing with that.

I *don’t want* OpenSSL to make *any* estimation of the amount of provided entropy. All I want it to do is to mix these bits into the RNG state. It’s *my* business how much entropy I’m providing – but I don’t want OpenSSL to make any decision regarding pull from other entropy sources based on my input.

Does it sound reasonable? (He, it does to me ;)
   
    But now we just ignore it and assume every bit with get contains 1
    bit of randomness and we're sundenly seriously overestimating the
    amount of randomness we're getting.

If I had my way, you’d assume that every bit contains 0 bits of entropy, but mix it in regardless because that’s what the user is asking you to do.


    This is a documented public API, you can't just go and ignore this parameter.
   
:-)


--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

OpenSSL - Dev mailing list
In reply to this post by Kurt Roeckx
➢ But now we just ignore it and assume every bit with get contains 1
➢     bit of randomness and we're sundenly seriously overestimating the
➢     amount of randomness we're getting. This is a documented public API,
➢     you can't just go and ignore this parameter.
   
Sure I can.  Because the DRBG seeds from the system anyway

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Kurt Roeckx
In reply to this post by Blumenthal, Uri - 0553 - MITLL
On Tue, Aug 29, 2017 at 08:38:09PM +0000, Blumenthal, Uri - 0553 - MITLL wrote:

> > If, based on its value, OpenSSL may decide that it now got “enough” entropy and doesn’t need to
> > pull more from other sources before serving randomness to requestors – then it is harmful.
> > “Over-confidence” in this value by the caller can negatively impact the quality of the produced
> > random numbers.
>    
>     As long as you have sources that don't provide 1 bit of randomness
>     per bit that you provide you need to have an estimate of how much
>     randomness it really contains. And you should probably seriously
>     underestimate it so that you're sure that you collect enough.
>
> So let me underestimate it to 0. ;-)  
>  
>     The problem with ignoring an existing parameter is that people
>     could be calling it with for instance the value of 0, knowing it
>     contains as good as none entropy. Or they could feed the unwithened
>     output of an TRNG in that with an estimate of randomness it provides.
>     And OpenSSL used to do the right thing with that.
>
> I *don’t want* OpenSSL to make *any* estimation of the amount of provided entropy. All I want it to do is to mix these bits into the RNG state. It’s *my* business how much entropy I’m providing – but I don’t want OpenSSL to make any decision regarding pull from other entropy sources based on my input.
>
> Does it sound reasonable? (He, it does to me ;)

But that is not the API that RAND_add() provides. It's a push not
a pull API. With the DRBG you can do this, assuming using it's an
extraction / derivative function.

One of the suggestions I did before is to have RAND_poll_ex() take
a parameter about how much randomness is needed, but I think it's
also a wrong API and I'm thinking about removing it.

>     But now we just ignore it and assume every bit with get contains 1
>     bit of randomness and we're sundenly seriously overestimating the
>     amount of randomness we're getting.
>
> If I had my way, you’d assume that every bit contains 0 bits of entropy, but mix it in regardless because that’s what the user is asking you to do.

Which is why I suggested we use this for the additional data. But
I think that as long as we have both APIs we might actually need
it for the entropy input. If there is no other way to add
randomness, RAND_add() is our current documented way to add it,
and it will need to keep working.


Kurt

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Kurt Roeckx
In reply to this post by OpenSSL - Dev mailing list
On Tue, Aug 29, 2017 at 08:44:11PM +0000, Salz, Rich via openssl-dev wrote:
> Sure I can.  Because the DRBG seeds from the system anyway

You can't assume that will work for all users.


Kurt

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

OpenSSL - Dev mailing list
➢ > Sure I can.  Because the DRBG seeds from the system anyway
   
➢     You can't assume that will work for all users.
   
And for places where the systesm doen’t have enough randomness, there is nothing we can do.

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Blumenthal, Uri - 0553 - MITLL
In reply to this post by Kurt Roeckx
I *don’t want* OpenSSL to make *any* estimation of the amount of provided entropy. All I want it to do is to mix these bits into the RNG state. It’s *my* business how much entropy I’m providing – but I don’t want OpenSSL to make any decision regarding pull from other entropy sources based on my input.

Does it sound reasonable? (He, it does to me ;)

But that is not the API that RAND_add() provides.

Yes, I realize that, sort of. But nonetheless, that’s the interface/behavior that I think an RNG should offer.

It's a push not
a pull API. With the DRBG you can do this, assuming using it's an
extraction / derivative function.

Not sure I understand. But “…you can do this…” sounds encouraging. ;-)

One of the suggestions I did before is to have RAND_poll_ex() take
a parameter about how much randomness is needed, but I think it's
also a wrong API and I'm thinking about removing it.

There are two aspects:

  1. The RNG itself keeping track of how much entropy it has in its pool/state, and thus when to pull its (standard) entropy sources for more (and specifically for how much more). I don’t want to interfere with this part.
  2. The user-visible/accessible interface to RNG that allows:
    • Forcing the “reseed”, making the RNG to to its sources for “fresh” entropy regardless of what’s up with its internal state;
    • Telling the RNG to immediately mix the given bits into its state, without adjusting its count of available/present entropy.



If I had my way, you’d assume that every bit contains 0 bits of entropy, but mix it in regardless because that’s what the user is asking you to do.

Which is why I suggested we use this for the additional data.

Yes, precisely. That’s exactly what I plan to use this interface for - mixing in additional data. NIST 800-90A Rev1 has “additional input” only as part of Reseed() or Generate(). I can live with that (i.e., with the fact that right before my randomness is mixed in, the entire state s replaced). Though ideally we’d have “RAND_add_ex()” or such that does not require replacing the entire state with fresh data from entropy source (because if that capability is needed, it is easy to emulate by issuing two calls: “Reseed(); Add_ex();"


But
I think that as long as we have both APIs we might actually need
it for the entropy input. If there is no other way to add
randomness, RAND_add() is our current documented way to add it,
and it will need to keep working.

It’s perfectly OK for me if RAND_add() keeps working. What I’d like to change about it is exactly when the mixing occurs. I’d like the state updated immediately upon RAND_add(), not when the RNG in question decides that it needs to replenish its entropy supply. I think this change would only improve security of applications that use it.

TNX!

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Paul Dale
In reply to this post by Dr. Matthias St. Pierre
My thoughts are that the new RNG API should be made public once it has been properly designed.  We've a chance to get this right, let's take the time and make an effort to do so.  There is no rush.

I also feel that having an engine interface to the new RNG API is worthwhile.  It allows hardware sources to be used via the same API.  Again, this will require design work to get right.  Likewise, supporting multiple DRBGs seems reasonable from a defence in depth perspective.

I would like to see an entropy argument to the equivalent to RAND_add.  Anyone is welcome to always pass zero in but there should be the option to not.  Consider an hardware source with _provable_ output quality, why shouldn't it be allowed to meaningfully contribute?  Each RNG should decide if it uses the argument or not.  Thus, a DRBG that uses RAND_add to provide additional data needn't count it.

I like the idea of two independent global RNGs.  Keeping the generation of long lived key material segregated from other uses of randomness seems sensible -- there is no possibility of cross compromise.  This does increase seeding requirements however.


Pauli
--
Oracle
Dr Paul Dale | Cryptographer | Network Security & Encryption
Phone +61 7 3031 7217
Oracle Australia


-----Original Message-----
From: Dr. Matthias St. Pierre [mailto:[hidden email]]
Sent: Tuesday, 29 August 2017 7:45 PM
To: [hidden email]
Subject: [openssl-dev] Plea for a new public OpenSSL RNG API

Hi everybody,

on the [openssl-dev] mailing list, there has been a long ongoing discussion about the new RAND_DRBG API and comparing it with the old RAND_METHOD API (see "[openssl-dev] Work on a new RNG for OpenSSL"). Two of the most controversal questions were:

 - Do we really need a new RNG API? Should the RAND_DRBG API be made public or kept private? (Currently, it's exported from libcrypto but only used internally by libssl.)
 - How much control should the user (programmer) be given over the reseeding process and/or should he be allowed to add his own additional randomness?

Many developers seem to be realizing the interesting possibilities of the DRBG API and are asking for public access to this new and promising API. One of the driving forces behind it is the question about how to do seeding and reseeding right. Among others, Uri Blumenthal asked for making the DRBG API public.

Currently, the OpenSSL core members seem to be reluctant to make the API public, at least at this early stage. I understand Rich Salz's viewpoint that this requires a thorough discussion, because a public interface can't be easily changed and wrong decisions in the early phase can become a heavy burdon.

Nevertheless, I agree with Uri Blumenthal that the DRBG API should be made public. So here comes my

======================================
Plea for a new public OpenSSL RNG API:
======================================

    The new RAND_DRBG is the superior API. It shouldn't be kept private and hidden behind the ancient RAND_METHOD API.
    The philosophy of the two APIs is not very well compatible, in particular when it comes to reseeding and adding
    additional unpredictable input. Hiding the RAND_DRBG behind the RAND_METHOD API only causes problems.
        Also, it will force people to patch their OpenSSL copy if they want to use the superior API.

    The RAND_DRBG API should become the new public OpenSSL RNG API and the old RAND_METHOD API should be deprecated
    in the long run. This transition does not need to be rushed, but it would be good if there would be early consent
    on the road map. I am thinking of a smooth transition with a phase of coexistence and a compatibility layer
    mapping the default RAND_METHOD to the default public RAND_DRBG instance. (This compatibility layer already exists,
    it's the 'RAND_OpenSSL()' method.)



Historical Background
=====================

As Rich already mentioned in his blog post, the RAND_DRBG isn't new. It's been a part of OpenSSL for a long time, hidden in the FIPS 2.0 Object Module.

I have been working with the FIPS DRBG for quite a while now, using a FIPS-capable OpenSSL 1.0.2x crypto library. The reason why our company switched to the FIPS DRBG is that one of our products runs on a small hardware device which does not have a reliable entropy source, but the product has to meet high security standards, in particular w.r.t. its RNG. So we decided to use the SmartCard RNG as primary entropy source for a deterministic AES-CTR based RNG and use /dev/urandom as additional input. Reseeding should occur on every generate request. Using the FIPS DRBG, these requirements were easily met, because the API gives such a fine grained control over reseeding and adding additional entropy.

The DRBG was well documented, its design in NIST SP800-90A (now: NIST SP800-90Ar1)  and its API in the OpenSSL FIPS 2.0 User Guide. The implementation was thoroughly tested and assessed during the FIPS certification process. So the only minor obstacle was that we had to patch the crypto library (not the FIPS object module) in order to get public access to the FIPS_drbg_*() methods.

I always considered the DRBG API more mature than the good old RAND_METHOD API and I wondered, why the DRBG code lay forgotten for so many years in the FIPS 2.0 object module sources and was never ported to master.

When in June of this year the thread "[openssl-dev] Work on a new RNG for OpenSSL" popped up (https://mta.openssl.org/pipermail/openssl-dev/2017-June), I closely watched the discussion, and when John Denker suggested having a look at NIST SP800-90A, I was electrified:

    > Constructive suggestion:  If you want to see what a RNG looks
    > like when designed by cryptographers, take a look at:
    >   Elaine Barker and John Kelsey,    >   "Recommendation for Random Number Generation Using Deterministic Random Bit Generators"
    >   http://csrc.nist.gov/publications/nistpubs/800-90A/SP800-90A.pdf
    >
    > That design may look complicated, but if you think you can
    > leave out some of the blocks in their diagram, proceed with
    > caution.  Every one of those blocks is there for a reason.

    <https://mta.openssl.org/pipermail/openssl-dev/2017-June/009423.html>

From his mail and the reaction to it, I had the impression that nobody seemed to remember the fact that the DRBG code was already present in the FIPS object module. The hidden treasure seemed forgotten! When I seized the opportunity and proposed to port the FIPS DRBG code into master, Rich Salz liked my idea and immediately started working on PR #3789:

       <https://mta.openssl.org/pipermail/openssl-dev/2017-June/009439.html>
       <https://mta.openssl.org/pipermail/openssl-dev/2017-June/009440.html>
       <https://github.com/openssl/openssl/pull/3789>

I am very grateful to Rich for picking up the idea so fast and and giving it so much speed and momentum. He has done a lot of work to convince others and defend the idea against initial scepticism by voices objecting to the seemingly new and unknown API. In the middle of August, when the first bulk of work was finished, Rich wrote a detailed blog post to advertise the new DRBG and explain his work <https://www.openssl.org/blog/blog/2017/08/12/random> and the discussion restarted:

     "[openssl-dev] Work on a new RNG for OpenSSL>  (see https://mta.openssl.org/pipermail/openssl-dev/2017-August)

It became quickly evident that users that the DRBG API had promising features, so they started asking for public access to this new API. The driving force was the question about how to do seeding and reseeding right. Among others, Uri Blumenthal was a dedicated advocate to make the DRBG API public (https://mta.openssl.org/pipermail/openssl-dev/2017-August/009594.html).

But the OpenSSL members currently seem to be reluctant to make the API public right away. I understand Rich's viewpoint that this decision requires a thorough discussion, because a public interface can't be easily changed and wrong decisions in the early phase can become a heavy burdon.


Nevertheless, I agree with Uri Blumenthal that the DRBG API should be made public. Here are some of my arguments for it.


The DRBG API supports multiple instances and chaining =====================================================

The NIST DRBG standard had chaining of multiple DRBG instances in the mind from the very beginnning, see for example footnote [4] on page 25 of NIST SP800-90Ar1:

    > Entropy input may be obtained from an entropy source or an NRBG, both of which provide fresh entropy.
    > Entropy input could also be obtained from a DRBG that has access to an entropy source or NRBG.

The original OpenSSL FIPS DRBG implementation did not support chaining, but this support has been added by Rich during the DRBG port.



The DRBG API has a highly flexible concept for seeding and reseeding ====================================================================

As mentioned previously, the DRBG has a callback mechanism with callbacks like get_entropy() and get_adin() callbacks, which make it easy to fine tune the default instantiation process by either adding additional randomness input, or changing the entropy source entirely. The callbacks are primarily intended for obtaining randomness instantiation and reseeing. There is a clear concept for reseeding, which can be adjusted by changing the reseed_interval: Normally, the DRBG reseeds itself automatically, whenever the reseed_interval has been reached.

In addition, there is a way for the DRBG consumer to add his own unpredictability when requesting random bytes: by adding additional input 'adin' to the RAND_DRBG_generate() call:

    int RAND_DRBG_generate(RAND_DRBG *drbg, unsigned char *out, size_t outlen,
                           int prediction_resistance,
                           const unsigned char *adin, size_t adinlen)

So why are there so many ways to add randomness and additional input? And what is the difference?

* The get_entropy() and get_adin() callbacks are used by the DRBG itself to _pull_ unpredictable data from some backend entropy source (which can also be a chained DRBG which is connected to some entropy source) during instantiation or reseeding.
* The 'adin' argument can be used by the DRBG consumer to _push_ unpredictable input into the DRBG when generating random output.


Note that the DRBG consumer has no possibility to push out-of-band randomness into the DRBG. Adding randomness is always coupled with a generate or (re-)seed operation. This is a very import difference between the RAND_DRBG and the RAND_METHOD and one of the reasons why it's so hard to do reseeding right in both the RAND and RAND_DRBG API simultaneously. In fact, currently


The 'RAND_add()/RAND_bytes()' pattern is broken ===============================================

In OpenSSL, the classical way for the RNG consumer to add his own randomness is to call 'RAND_add()' before calling 'RAND_bytes()'. If the new 'RAND_OpenSSL()' method (the "compatibility layer" hiding the public RAND_DRBG instance)  is the default, then this does not work as expected anymore:

The reason is that a call to 'RAND_add()' adds the provided randomness only to a global buffer ('rand_bytes'), from which it will be pulled during the next reseed. But no reseed is triggered. So the next RAND_bytes() call will be unaffected from the RAND_add(), which is not what the consumer expected. (The same holds for 'RAND_seed()', since 'drbg_seed()' only calls into 'drbg_add()')

Reseeding of DRBGs occurs only at the following occasions:

* immediately after a 'fork()' (new)
* if the 'reseed_counter' exceeds the 'reseed_interval'
* if 'RAND_DRBG_generate()' is called requesting 'prediction_resistance'
* 'RAND_DRBG_reseed()' is called explicitely

*Note:* Currently it looks like the situation is even worse: if 'RAND_add()' is called multiple times before a reseed occurs, then the result of the previous call is overwritten.

Reseeding the 'DRBG' whenever the user calls 'RAND_add()' does not seem a good solution. It would be too expensive, in particular if system entropy is pulled for reseeding. Of course it is possible to fix this issue, but the DRBG provides for a much simpler solution: it lets the consumer contribute to the entropy of the internal state by providing additional input. If the user input contains entropy, that's fine, if it's "snake oil", no harm. The additional input is mixed into the internal state in just the same way as the entropy buffer using the 'ctr_df()' derivation function. One might think of the 'entropy' input as trusted randomness and 'adin' as untrusted randomness.

For this reason, I would like to see the 'RAND_add()/RAND_bytes()' pattern deprecated and the 'RAND_DRBG_generate() with additional input' pattern advertised instead.


The DRBG API supports different implementations ===============================================

Well, it _supported_ them, until recently. But that's not irreversible.

The DRBG concept, as layed out in the NIST standard, provides a generic framework for deterministic RNGs (the acronym DRBG stands for Deterministic Random Bit Generator). It deals with general questions like how to instantiate and reseed the RNG, where does it get the entropy from, etc.

The standard proposes three concrete implementations, Hash_DRBG, HMAC_DRBG, and CTR_DRBG. In the FIPS code, all three were implemented, and the genericity was achieved using a data union and a set of five function pointers, reminiscent of a vtable in C++:

    struct drbg_ctx_st
    {
        ...
       
        /* Implementation specific structures */
        union
            {
            DRBG_HASH_CTX hash;
            DRBG_HMAC_CTX hmac;
            DRBG_CTR_CTX  ctr;
            } d;
        /* Initialiase PRNG and setup callbacks below */
        int (*init)(DRBG_CTX *ctx, int nid, int security, unsigned int flags);
        /* Intantiate PRNG */
        int (*instantiate)(DRBG_CTX *ctx,
                    const unsigned char *ent, size_t entlen,
                    const unsigned char *nonce, size_t noncelen,
                    const unsigned char *pers, size_t perslen);
        /* reseed */
        int (*reseed)(DRBG_CTX *ctx,
                    const unsigned char *ent, size_t entlen,
                    const unsigned char *adin, size_t adinlen);
        /* generat output */
        int (*generate)(DRBG_CTX *ctx,
                    unsigned char *out, size_t outlen,
                    const unsigned char *adin, size_t adinlen);
        /* uninstantiate */
        int (*uninstantiate)(DRBG_CTX *ctx);

        ...
    };
   

This part of the code was removed during the DRBG port, because currently CTR_DRBG is the only impementation. I would like to suggest to restore this 'polymorphic' implementation, to ease adding new implementations CHACHA20_DRBG in the future


The DRBG API is well documented and tested ==========================================


The entire DRBG API is part of the OpenSSL FIPS 2.0 Module and as such is well tested and well documented. Most of the documentation for the manual pages still-to-be-written can be taken from the FIPS User Guide and converted into manual pages, starting with textual modifications like


        FIPS_drbg_xxx(...)  ->  RAND_DRBG_xxx(...)
        DRBG_CTX *dctx      ->  RAND_DRBG *dctx

and taking the new typedefs into consideration. Here is for example a comparison of the API function to install the callbacks:

    FIPS DRBG:
       
        int FIPS_drbg_set_callbacks(DRBG_CTX *dctx,
            size_t (*get_entropy)(DRBG_CTX *ctx,  <args> ),
            void (*cleanup_entropy)(DRBG_CTX *ctx, <args> ),
            size_t entropy_blocklen,
            size_t (*get_nonce)(DRBG_CTX *ctx, <args> ),
            void (*cleanup_nonce)(DRBG_CTX *ctx, <args>)
        );
   
   
    RAND_DRBG:
   
        typedef size_t (*RAND_DRBG_get_entropy_fn)(RAND_DRBG *ctx, <args> );
        typedef void (*RAND_DRBG_cleanup_entropy_fn)(RAND_DRBG *ctx, <args> );
        typedef size_t (*RAND_DRBG_get_nonce_fn)(RAND_DRBG *ctx,  <args> );
        typedef void (*RAND_DRBG_cleanup_nonce_fn)(RAND_DRBG *ctx,  <args> );
       
        int RAND_DRBG_set_callbacks(RAND_DRBG *dctx,
                                    RAND_DRBG_get_entropy_fn get_entropy,
                                    RAND_DRBG_cleanup_entropy_fn cleanup_entropy,
                                    RAND_DRBG_get_nonce_fn get_nonce,
                                    RAND_DRBG_cleanup_nonce_fn cleanup_nonce);
       




Conclusion
==========

I see no reason why the RAND_DRBG shouldn't be made public as soon as possible, keeping the API as close as possible to the original FIPS DRBG API (FIPS 3.0 is upcoming!). In a second step, the current compatibility binding from RAND_METHOD to RAND_DRBG could be deprecated and faded out smoothly.


Looking forward to receiving your comments. (But please be patient with me, I'm currently on physical rehab after a surgery.)


Matthias St. Pierre


--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
Reply | Threaded
Open this post in threaded view
|

Re: Plea for a new public OpenSSL RNG API

Dr. Matthias St. Pierre
In reply to this post by OpenSSL - Dev mailing list
Hi Rich,

> I think the only way to do that in the DRBG model is to treat it as “additional input” and do a generate
> call. I think that would achieve the same effect, but it wouldn’t delay any possible seeding of randomness
> that the DRBG itself needs at some point.

You're right, well, almost. The implementation of drbg_add() (which implements RAND_add()) could be actually as simple as this little function:

    static int drbg_add(const void *buf, int num, double randomness)
    {
        int ret = 0;
       
        CRYPTO_THREAD_write_lock(rand_drbg.lock);
        ret = RAND_DRBG_reseed(&rand_drbg, buf, num);

    err:
        CRYPTO_THREAD_unlock(rand_drbg.lock);
        return ret;
    }

The problem is that currently this would create a circular dependency, because now RAND_add() is involved both in pushing and in pulling entropy.

    RAND_add() -> drbg_add() -> RAND_DRBG_reseed() -> get_entropy() -> get_entropy_from_system() -> RAND_poll_ex() -> RAND_add()


Using RAND_DRBG_generate() instead of  RAND_DRBG_reseed() does not solve the problem, because the former can occasionally trigger a reseed. I already have the solution in my mind how to break up this vicious circle, but I need some time to write it down. Here is a sketch:


1) Use RAND_add() only for pushing to break the circular dependency
===================================================================

Add a new function rand_drbg_add() which is identical to RAND_DRBG_reseed(), except that it does not pull fresh entropy (and consequently does not reset the reseed_counter). Then the left hand side of the becomes

    RAND_add() -> drbg_add() -> rand_drbg_add()
       
The code of drbg_add() and rand_drbg_add() is at the end of this mail.  (This is the easy part :-) )


2) Reorganize the backend code for pulling entropy
==================================================

Reorganize get_entropy_from_system() in a way that it collects the entropy in the way it was originally intended, without calling RAND_add() and without using this evil fixed size 'rand_bytes' buffer, from which I tried to dissuade you ever since, Rich (no offence ;-) )

This part is more complicated, because one has to be careful not to break RAND_poll() for applications which don't use the default RAND_OpenSSL() method.

So please give me some time, I will try to turn my ideas into a PR.

Matthias


___


drbg_lib.c:
       
        /*
         * Mix in the specified data without reseeding |drbg|.
         *
         * Used by drbg_add() only.
         *
         * Similar to RAND_DRBG_reseed(), but there is no call to
         * get_entropy() and the reseed_counter is not reset
         */
        int rand_drbg_add(RAND_DRBG *drbg,
                                                 const unsigned char *adin, size_t adinlen)
        {
                if (drbg->state == DRBG_ERROR) {
                        RANDerr(RAND_F_RAND_DRBG_RESEED, RAND_R_IN_ERROR_STATE);
                        return 0;
                }
                if (drbg->state == DRBG_UNINITIALISED) {
                        RANDerr(RAND_F_RAND_DRBG_RESEED, RAND_R_NOT_INSTANTIATED);
                        return 0;
                }

                if (adin == NULL)
                        adinlen = 0;
                else if (adinlen > drbg->max_adinlen) {
                        RANDerr(RAND_F_RAND_DRBG_RESEED, RAND_R_ADDITIONAL_INPUT_TOO_LONG);
                        return 0;
                }

                drbg->state = DRBG_ERROR;
                if (!ctr_reseed(drbg, NULL, 0, adin, adinlen))
                        goto end;
                drbg->state = DRBG_READY;
               
        end:
                if (drbg->state == DRBG_READY)
                        return 1;
                return 0;
        }

    ...

    static int drbg_add(const void *buf, int num, double randomness)
    {
        int ret = 0;
       
        CRYPTO_THREAD_write_lock(rand_drbg.lock);
        ret = rand_drbg_add(&rand_drbg, buf, num);

    err:
        CRYPTO_THREAD_unlock(rand_drbg.lock);
        return ret;
    }




--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

smime.p7s (5K) Download Attachment
123