why does RAND_add() take "randomness" as a "double"?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

why does RAND_add() take "randomness" as a "double"?

Laszlo Ersek
(resending, with my subscription to <[hidden email]> completed)

Hi OpenSSL Developers,

(cross-posting <[hidden email]> and <[hidden email]>,)

OpenSSL commit [1] changed the representation of the "entropy amount" --
later renamed to "randomess" in [2] -- from "int" to "double". I've read
the commit message:

commit 853f757ecea74a271a7c5cdee3f3b5fe0d3ae863
Author: Bodo Möller <[hidden email]>
Date:   Sat Feb 19 15:22:53 2000 +0000

    Allow for higher granularity of entropy estimates by using 'double'
    instead of 'unsigned' counters.
    Seed PRNG in MacOS/GetHTTPS.src/GetHTTPS.cpp.

    Partially submitted by Yoram Meroz <[hidden email]>.

and also checked "MacOS/GetHTTPS.src/GetHTTPS.cpp" at the same commit.
But, I'm none the wiser.

Can someone please explain what is gained by using a floating point type
here?

Is it really a relevant use case that entropy is fed from an external
source to OpenSSL such that truncating the amount to a whole number of
bits would cause significant lossage? (Admittedly, it could be relevant
if the individual randomness bit counts were in the (0, 1) interval,
both boundaries exclusive.)

Using floating point for randomness representation is a problem for
environments that prefer to avoid floating point altogether, such as
edk2 ("UEFI") firmware

Thanks,
Laszlo

[1] https://github.com/openssl/openssl/commit/853f757ecea7
[2] https://github.com/openssl/openssl/commit/f367ac2b2664
Reply | Threaded
Open this post in threaded view
|

Re: why does RAND_add() take "randomness" as a "double"?

Ken Goldman-2
On 5/21/2019 10:15 AM, Laszlo Ersek wrote:

>[snip]
>
> Can someone please explain what is gained by using a floating point type
> here?
>
> Is it really a relevant use case that entropy is fed from an external
> source to OpenSSL such that truncating the amount to a whole number of
> bits would cause significant lossage? (Admittedly, it could be relevant
> if the individual randomness bit counts were in the (0, 1) interval,
> both boundaries exclusive.)
>
> Using floating point for randomness representation is a problem for
> environments that prefer to avoid floating point altogether, such as
> edk2 ("UEFI") firmware

I agree, and I reported this back in 2016.  We also have an environment
that does not have floating point.


Reply | Threaded
Open this post in threaded view
|

Re: why does RAND_add() take "randomness" as a "double"?

OpenSSL - User mailing list
When I overhauled the RAND mechanism, I tried to deprecate this use of floating point, in favor of just a number from 0 to 100 but was voted down.

It *is* stupid.  Luckily, on a modern system with system-provided randomness to seed the RNG, you never need this call.


Reply | Threaded
Open this post in threaded view
|

Re: why does RAND_add() take "randomness" as a "double"?

Blumenthal, Uri - 0553 - MITLL
On 5/21/19, 10:45 AM, "openssl-users on behalf of Salz, Rich via openssl-users" <[hidden email] on behalf of [hidden email]> wrote:

    When I overhauled the RAND mechanism, I tried to deprecate this use of floating point,
    in favor of just a number from 0 to 100 but was voted down.

If it's a sarcasm, I'm missing the point.
   
    It *is* stupid.

In general, yes, it is.

    Luckily, on a modern system with system-provided randomness to seed the RNG, you never need this call.

Respectfully disagree.

There are use cases when one wants to mix/add extra randomness from, e.g., an external source (that, for whatever reasons, is trusted more than what's provided by the system).
   
   
   

smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: why does RAND_add() take "randomness" as a "double"?

OpenSSL - User mailing list
>    If it's a sarcasm, I'm missing the point.
 
I was't being sarcastic, I was trying to show that the team, recently, still liked the use of floating point.

>    There are use cases when one wants to mix/add extra randomness from, e.g., an external source (that, for whatever reasons, is trusted more than what's provided by the system).
 
Then just set it to 1.0 and be done with it.
       
       
   

Reply | Threaded
Open this post in threaded view
|

Re: why does RAND_add() take "randomness" as a "double"?

Dennis Clarke-2
On 5/21/19 3:27 PM, Salz, Rich via openssl-users wrote:
>>     If it's a sarcasm, I'm missing the point.
>    
> I was't being sarcastic, I was trying to show that the team, recently, still liked the use of floating point.
>
>>     There are use cases when one wants to mix/add extra randomness from, e.g., an external source (that, for whatever reasons, is trusted more than what's provided by the system).
>  
> Then just set it to 1.0 and be done with it.
>          

External 300 baud serial attached coin flipper also works well.


--
Dennis Clarke
RISC-V/SPARC/PPC/ARM/CISC
UNIX and Linux spoken
GreyBeard and suspenders optional
Reply | Threaded
Open this post in threaded view
|

RE: why does RAND_add() take "randomness" as a "double"?

Dr Paul Dale
In reply to this post by Laszlo Ersek
Double makes sense.  Entropy is often estimated as a real value.

E.g. we have the aforementioned coin flipper feeding data serially.
Adding each bit sequentially means 0.125 bytes of entropy per call.

Not the best example....

Pauli
--
Oracle
Dr Paul Dale | Cryptographer | Network Security & Encryption
Phone +61 7 3031 7217
Oracle Australia


-----Original Message-----
From: Laszlo Ersek [mailto:[hidden email]]
Sent: Wednesday, 22 May 2019 12:15 AM
To: [hidden email]
Cc: Jian J Wang <[hidden email]>; edk2-devel-groups-io <[hidden email]>; Lu, XiaoyuX <[hidden email]>; Ard Biesheuvel <[hidden email]>
Subject: why does RAND_add() take "randomness" as a "double"?

(resending, with my subscription to <[hidden email]> completed)

Hi OpenSSL Developers,

(cross-posting <[hidden email]> and <[hidden email]>,)

OpenSSL commit [1] changed the representation of the "entropy amount" -- later renamed to "randomess" in [2] -- from "int" to "double". I've read the commit message:

commit 853f757ecea74a271a7c5cdee3f3b5fe0d3ae863
Author: Bodo Möller <[hidden email]>
Date:   Sat Feb 19 15:22:53 2000 +0000

    Allow for higher granularity of entropy estimates by using 'double'
    instead of 'unsigned' counters.
    Seed PRNG in MacOS/GetHTTPS.src/GetHTTPS.cpp.

    Partially submitted by Yoram Meroz <[hidden email]>.

and also checked "MacOS/GetHTTPS.src/GetHTTPS.cpp" at the same commit.
But, I'm none the wiser.

Can someone please explain what is gained by using a floating point type here?

Is it really a relevant use case that entropy is fed from an external source to OpenSSL such that truncating the amount to a whole number of bits would cause significant lossage? (Admittedly, it could be relevant if the individual randomness bit counts were in the (0, 1) interval, both boundaries exclusive.)

Using floating point for randomness representation is a problem for environments that prefer to avoid floating point altogether, such as
edk2 ("UEFI") firmware

Thanks,
Laszlo

[1] https://github.com/openssl/openssl/commit/853f757ecea7
[2] https://github.com/openssl/openssl/commit/f367ac2b2664
Reply | Threaded
Open this post in threaded view
|

Re: why does RAND_add() take "randomness" as a "double"?

Richard Levitte - VMS Whacker-2
In reply to this post by OpenSSL - User mailing list


"Salz, Rich via openssl-users" <[hidden email]> skrev: (21 maj 2019 17:27:44 CEST)

>>    If it's a sarcasm, I'm missing the point.
>  
>I was't being sarcastic, I was trying to show that the team, recently,
>still liked the use of floating point.
>
>>    There are use cases when one wants to mix/add extra randomness
>from, e.g., an external source (that, for whatever reasons, is trusted
>more than what's provided by the system).
>
>Then just set it to 1.0 and be done with it.

That hardly helps on systems that don't have floating point at all.

Cheers
Richard
>        
>        
>    

--
Skickat från min Android-enhet med K-9 Mail. Ursäkta min fåordighet.
Reply | Threaded
Open this post in threaded view
|

Re: why does RAND_add() take "randomness" as a "double"?

OpenSSL - User mailing list
    >Then just set it to 1.0 and be done with it.
   
>    That hardly helps on systems that don't have floating point at all.

No it doesn't.  Such systems aren't supported by OpenSSL.  There are many places were floating point is used/supported.
Removing the second arg to RAND_add is the least of the problems (look at various asm files)

Reply | Threaded
Open this post in threaded view
|

Re: why does RAND_add() take "randomness" as a "double"?

Ken Goldman-2
On 5/21/2019 10:07 PM, Salz, Rich via openssl-users wrote:
>      >Then just set it to 1.0 and be done with it.
>      
>>     That hardly helps on systems that don't have floating point at all.
>
> No it doesn't.  Such systems aren't supported by OpenSSL.  There are many places were floating point is used/supported.
> Removing the second arg to RAND_add is the least of the problems (look at various asm files)

The assembler code can be bypassed on those systems.

I see a few places where it's used to force an alignment, but perhaps
there's another way.

It's used in test programs to report performance.

For us, the random number generator was the problem.



Reply | Threaded
Open this post in threaded view
|

Re: why does RAND_add() take "randomness" as a "double"?

Ken Goldman-2
In reply to this post by Dr Paul Dale
On 5/21/2019 9:48 PM, Paul Dale wrote:
> Double makes sense.  Entropy is often estimated as a real value.
>

Having a human readable calculation using floating point doesn't (to me)
mean that an API argument has to be a double.

 From what I see in the code, the parameter 'double entropy' is used
to increment a value that eventually reaches # define ENTROPY_NEEDED 32.

Couldn't the number have been an unsigned long?  If more precision was
needed, make the units 1/64k and make ENTROPY_NEEDED 32 * 64k.  It's a
bit more work for the caller, but removes the (perhaps only) place
floating point is needed.



Reply | Threaded
Open this post in threaded view
|

Re: why does RAND_add() take "randomness" as a "double"?

OpenSSL - User mailing list
In reply to this post by OpenSSL - User mailing list
On 21/05/2019 16:44, Salz, Rich via openssl-users wrote:
> When I overhauled the RAND mechanism, I tried to deprecate this use of floating point, in favor of just a number from 0 to 100 but was voted down.
>
> It *is* stupid.  Luckily, on a modern system with system-provided randomness to seed the RNG, you never need this call.
>
>

Perhaps it would have been more acceptable to use a binary base,
instead of a decimal percentage, as there is nothing inherently
decimal about this value.

Good options inspired by other cryptographic libraries include:

- Number of bits of entropy passed in call (For example, a
  perfectly balanced coin flipper could provide the 4 byte
  values "head" or "tail" with an entropy of 1 bit).

- 256th of bits ditto (for example a coin flipper with a known
  slight imbalance could report 252/256th of a bit for each flip
  included in the buffer).

- 32th of bits ditto (makes the "100%" case equal to
  (bytecount << 8)).

In each of those 3 cases, the internal measure of "entropy
left" would be in that same unit, and a compatibility mapping
for the old API would do the conversion of the double as a
pure inline macro that doesn't trigger "float used" compiler
magics in programs that don't use it.

Clarifying notes:

- The current limit of considering only 32 bytes of entropy
  is an artifact of the current set of RNG algorithms, and
  should not be exposed in the API design.  For example
  future addition of post-quantum algorithms may necessitate
  having an RNG with an internal state entropy larger than
  256 bits.

- Future RNG implementations may include logic to safely
  accumulate obtained entropy into "batches" before updating
  the RNG state, as this may have cryptographic benefits.

- The use of a dummy double to force the alignment of
  structures and unions to the "highest known" value can
  be trivially replaced by another type where it is not
  already treated as "not actually floating point
  operations" by the compiler.  For example by passing
  "-Ddouble=long long" as a compiler option.

- The use of floating point registers in CPU-specific
  vector unit optimizations can be readily avoided by
  a no-asm compile.

- Floating point calculations in test programs such as
  "openssl speed" is not relevant to actual library use.

- On Linux x86, test programs that avoid all floating
  point can be checked via the PF_USED_MATH flag or its
  upcoming Linux 5.x replacement.  This may be useful
  in the test suite.


Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

Reply | Threaded
Open this post in threaded view
|

AW: why does RAND_add() take "randomness" as a "double"?

Dr. Matthias St. Pierre
I think nobody of us needs to be convinced anymore that making it a 'double' was a bad idea.
But the RAND api is very ancient and changing the argument type would be a breaking change.
That's why we didn't dare to touch it when we overhauled the RNG implementation for 1.1.1,
because we tried very hard not to add unnecessary breaking changes to the ones made in 1.1.0.

Matthias

Reply | Threaded
Open this post in threaded view
|

Re: why does RAND_add() take "randomness" as a "double"?

Blumenthal, Uri - 0553 - MITLL
I can understand why you may not want to break existing API. Why not add another similar interface done the right way?

On 5/22/19, 1:11 PM, "openssl-users on behalf of Dr. Matthias St. Pierre" <[hidden email] on behalf of [hidden email]> wrote:

    I think nobody of us needs to be convinced anymore that making it a 'double' was a bad idea.
    But the RAND api is very ancient and changing the argument type would be a breaking change.
    That's why we didn't dare to touch it when we overhauled the RNG implementation for 1.1.1,
    because we tried very hard not to add unnecessary breaking changes to the ones made in 1.1.0.
   
    Matthias
   
   

smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: why does RAND_add() take "randomness" as a "double"?

Dennis Clarke-2
In reply to this post by OpenSSL - User mailing list

> Good options inspired by other cryptographic libraries include:
>
> - Number of bits of entropy passed in call (For example, a
>   perfectly balanced coin flipper could provide the 4 byte
>   values "head" or "tail" with an entropy of 1 bit).

Let's drop the coin flipper. It was an off hand remark and by now we
all know there ain't no such thing as a good coin flip for rng.

     See Professor Persi Diaconis at Stanford for that :
         https://www.youtube.com/watch?v=AYnJv68T3MM

Bell's theorem and kolmogorov aside get a radiation decay source as
that is really the *only* real rng that we know of.
Or that I know of.   http://www.fourmilab.ch/hotbits/hardware.html


--
Dennis Clarke
RISC-V/SPARC/PPC/ARM/CISC
UNIX and Linux spoken
GreyBeard and suspenders optional

ps: see "futility of foresight"
Reply | Threaded
Open this post in threaded view
|

Re: why does RAND_add() take "randomness" as a "double"?

OpenSSL - User mailing list
On 22/05/2019 19:32, Dennis Clarke wrote:

>
>> Good options inspired by other cryptographic libraries include:
>>
>> - Number of bits of entropy passed in call (For example, a
>>   perfectly balanced coin flipper could provide the 4 byte
>>   values "head" or "tail" with an entropy of 1 bit).
>
> Let's drop the coin flipper. It was an off hand remark and by now we
> all know there ain't no such thing as a good coin flip for rng.
>
>     See Professor Persi Diaconis at Stanford for that :
>         https://www.youtube.com/watch?v=AYnJv68T3MM
>
> Bell's theorem and kolmogorov aside get a radiation decay source as
> that is really the *only* real rng that we know of.
> Or that I know of.   http://www.fourmilab.ch/hotbits/hardware.html
The coin flipper, even if theoretically problematic, is the standard
statistical example used to describe a 1-bit-at-a-time hardware RNG.

It includes a nice conceptual model to discuss hardware bias (using
Shannon's entropy formula etc.).  Actual 1-bit sources include the
classic semiconductor shot noise fed to a comparator and some primitive
implementations of radioactive RNGs.

Also, radioactive sources are an unacceptable danger in many of the
embedded and portable applications most likely to lack floating point
support.

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded