RNG behavior by default

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

RNG behavior by default

Mike Blaguszewski
I am using the EVP API (version 1.1.1) for performing public key and symmetric key operations across a variety of platforms (macOS, Windows, Linux, iOS and Android). I am currently not doing anything to explicitly seed OpenSSL’s random number generator. My understanding is that the default behavior should be cryptographically secure.

So my concerns are:
1. Whether I really can count on getting a high-entropy PRNG across these various platforms, without any explicit initialization.
2. If something goes wrong with PRNG initialization, that it will fail hard rather than fall back to something less secure. And if so how I detect such a failure.

Our current implementation uses libsodium, which relies on the usual system calls to generate entropy, so if I can count on OpenSSL always doing this then I’m happy. 

Thanks,
Mike

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: RNG behavior by default

Kurt Roeckx
On Thu, Jan 03, 2019 at 11:03:01AM -0500, Mike Blaguszewski wrote:
> I am using the EVP API (version 1.1.1) for performing public key and symmetric key operations across a variety of platforms (macOS, Windows, Linux, iOS and Android). I am currently not doing anything to explicitly seed OpenSSL’s random number generator. My understanding is that the default behavior <https://www.openssl.org/blog/blog/2017/08/12/random/> should be cryptographically secure.
>
> So my concerns are:
> 1. Whether I really can count on getting a high-entropy PRNG across these various platforms, without any explicit initialization.
> 2. If something goes wrong with PRNG initialization, that it will fail hard rather than fall back to something less secure. And if so how I detect such a failure.
>
> Our current implementation uses libsodium, which relies on the usual system calls to generate entropy, so if I can count on OpenSSL always doing this then I’m happy.

It will make use of system calls when available. Those are known
to provide system calls:
- Linux since 3.17
- Darwin since 16 (OSX 10.12, IOS 10.0).
- Solaris since 11.3
- OpenBSD since 5.6
- FreeBSD since 12.0 (1200061)

By default it will fall back to use something like /dev/urandom if
the system call is not available or returns an error.

On Windows we are also using the system provided entropy by using
function calls.

You do not need to do anything to initialize RNG. It will
automatically initiailze on first use.

It will hard fail when it's not able to get entropy.

Since it now reseeds from time to time, it can actually start to
fail after having run succesfully for some time. But it's very
unlikely that you would run into that, by default we should make
sure that we can always get entropy.


Kurt

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: RNG behavior by default

Matthias St. Pierre
In reply to this post by Mike Blaguszewski
> So my concerns are:
> 1. Whether I really can count on getting a high-entropy PRNG across these various platforms, without any explicit initialization.

Yes, for the mentioned platforms, the default configuration is `--with-rand-seed=os`, which means the DRBG automatically seeds
and reseeds using os entropy sources.

2. If something goes wrong with PRNG initialization, that it will fail hard rather than fall back to something less secure. And if so how I detect such a failure.

If the (re-)seeding fails, the DRBG enters an error state. When you try to generate random bytes it will detect the error state and try
automatically to heal the error state by reinstantiating. But if reseeding fails, it will return and error code and not generate any pseudo random bytes.

Citing from the manual pages:

        OpenSSL comes with a default implementation of the RAND API which is based on the
        deterministic random bit generator (DRBG) model as described in [NIST SP 800-90A Rev. 1].
        The default random generator will initialize automatically on first use and will be fully functional
        without having to be initialized ('seeded') explicitly. It seeds and reseeds itself automatically using
        trusted random sources provided by the operating system.

        As a normal application developer, you do not have to worry about any details, just use RAND_bytes(3)
        to obtain random data. Having said that, there is one important rule to obey: Always check the error
        return value of RAND_bytes(3) and do not take randomness for granted.

        https://www.openssl.org/docs/man1.1.1/man7/RAND.html

(See also https://www.openssl.org/docs/man1.1.1/man7/RAND_DRBG.html)

Matthias

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: RNG behavior by default

Steffen Nurpmeso-2
Dr. Matthias St. Pierre wrote in <450169f8ca7c43d1841c4c8052e78c72@Ex13.\
ncp.local>:
 |> So my concerns are:
 |> 1. Whether I really can count on getting a high-entropy PRNG across \
 |> these various platforms, without any explicit initialization.
 |
 |Yes, for the mentioned platforms, the default configuration is `--with-r\
 |and-seed=os`, which means the DRBG automatically seeds
 |and reseeds using os entropy sources.
 |
 |2. If something goes wrong with PRNG initialization, that it will fail \
 |hard rather than fall back to something less secure. And if so how \
 |I detect such a failure.
 |
 |If the (re-)seeding fails, the DRBG enters an error state. When you \
 |try to generate random bytes it will detect the error state and try
 |automatically to heal the error state by reinstantiating. But if reseeding \
 |fails, it will return and error code and not generate any pseudo random \
 |bytes.
 |
 |Citing from the manual pages:
 ...
 | As a normal application developer, you do not have to worry about \
 | any details, just use RAND_bytes(3)
 | to obtain random data. Having said that, there is one important rule \
 | to obey: Always check the error
 | return value of RAND_bytes(3) and do not take randomness for granted.
 |
 | https://www.openssl.org/docs/man1.1.1/man7/RAND.html

That is new however, _imho_.  The wording of RAND_bytes(3) (still)
says that "an error occurs [.if.] not [been] seeded with enough
[data]", and RAND_status(3) returns 1 if the PRNG "has been seeded
with enough data".  So if it is seeded it is seeded, in my
understanding anything further on up the road only mixes in noise
(which likely will undergo further maths and be stirred into the
pool, i have not looked, actually).  I do not test RAND_bytes(3)
return (yet), because i have ensured the PRNG is sufficiently
seeded, and RAND_status(3) returns success, before RAND_bytes(3)
is used the first time.

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: RNG behavior by default

Kurt Roeckx
On Fri, Jan 04, 2019 at 02:48:48PM +0100, Steffen Nurpmeso wrote:

> Dr. Matthias St. Pierre wrote in <450169f8ca7c43d1841c4c8052e78c72@Ex13.\
> ncp.local>:
>  |> So my concerns are:
>  |> 1. Whether I really can count on getting a high-entropy PRNG across \
>  |> these various platforms, without any explicit initialization.
>  |
>  |Yes, for the mentioned platforms, the default configuration is `--with-r\
>  |and-seed=os`, which means the DRBG automatically seeds
>  |and reseeds using os entropy sources.
>  |
>  |2. If something goes wrong with PRNG initialization, that it will fail \
>  |hard rather than fall back to something less secure. And if so how \
>  |I detect such a failure.
>  |
>  |If the (re-)seeding fails, the DRBG enters an error state. When you \
>  |try to generate random bytes it will detect the error state and try
>  |automatically to heal the error state by reinstantiating. But if reseeding \
>  |fails, it will return and error code and not generate any pseudo random \
>  |bytes.
>  |
>  |Citing from the manual pages:
>  ...
>  | As a normal application developer, you do not have to worry about \
>  | any details, just use RAND_bytes(3)
>  | to obtain random data. Having said that, there is one important rule \
>  | to obey: Always check the error
>  | return value of RAND_bytes(3) and do not take randomness for granted.
>  |
>  | https://www.openssl.org/docs/man1.1.1/man7/RAND.html
>
> That is new however, _imho_.  The wording of RAND_bytes(3) (still)
> says that "an error occurs [.if.] not [been] seeded with enough
> [data]", and RAND_status(3) returns 1 if the PRNG "has been seeded
> with enough data".  So if it is seeded it is seeded, in my
> understanding anything further on up the road only mixes in noise
> (which likely will undergo further maths and be stirred into the
> pool, i have not looked, actually).  I do not test RAND_bytes(3)
> return (yet), because i have ensured the PRNG is sufficiently
> seeded, and RAND_status(3) returns success, before RAND_bytes(3)
> is used the first time.

For 1.1.0 and older that works, because they do not reseed. Since
1.1.1 it does reseed, and if the reseed fails, it will go to an
error state. So yes, this is new behavior.

The RAND_bytes and RAND_status manpages can clearly be improved.
Since you always have to check RAND_bytes's return value now,
RAND_status is mostly useless.


Kurt

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: RNG behavior by default

Dr Paul Dale
In reply to this post by Matthias St. Pierre
I know that iOS (which was listed) has a good randomness source (SecRandomCopyBytes) but I don’t think OpenSSL uses it yet.
I’m not sure about the quality of Android’s sources, but would expect them to be decent.


Pauli


On 4 Jan 2019, at 10:46 pm, Dr. Matthias St. Pierre <[hidden email]> wrote:

So my concerns are:
1. Whether I really can count on getting a high-entropy PRNG across these various platforms, without any explicit initialization.

Yes, for the mentioned platforms, the default configuration is `--with-rand-seed=os`, which means the DRBG automatically seeds
and reseeds using os entropy sources.

2. If something goes wrong with PRNG initialization, that it will fail hard rather than fall back to something less secure. And if so how I detect such a failure.

If the (re-)seeding fails, the DRBG enters an error state. When you try to generate random bytes it will detect the error state and try
automatically to heal the error state by reinstantiating. But if reseeding fails, it will return and error code and not generate any pseudo random bytes.

Citing from the manual pages:

OpenSSL comes with a default implementation of the RAND API which is based on the
deterministic random bit generator (DRBG) model as described in [NIST SP 800-90A Rev. 1].
The default random generator will initialize automatically on first use and will be fully functional
without having to be initialized ('seeded') explicitly. It seeds and reseeds itself automatically using
trusted random sources provided by the operating system.

As a normal application developer, you do not have to worry about any details, just use RAND_bytes(3)
to obtain random data. Having said that, there is one important rule to obey: Always check the error
return value of RAND_bytes(3) and do not take randomness for granted.

https://www.openssl.org/docs/man1.1.1/man7/RAND.html

(See also https://www.openssl.org/docs/man1.1.1/man7/RAND_DRBG.html)

Matthias

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users


--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: RNG behavior by default

Matthias St. Pierre
In reply to this post by Kurt Roeckx

I agree with Kurt, except for one point:

> The RAND_bytes and RAND_status manpages can clearly be improved.

Both manpages got an update during the DRBG rewrite (by me) and I don't
see any contradiction. You bring it to the point yourself:

> So _IF_ it is seeded it is seeded...

It is true that the DRBG will automatically seed, but it is equally true
that it can still end up in an unseeded (error) state, if no suitable entropy
source is available. And since this can also happen during reseeding (which
in particular is enforced after a fork), it is always necessary to check the return
value of the RAND_bytes() function. Because in the error state, the buffer is not
filled at all.

Matthias


--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: RNG behavior by default

Kurt Roeckx
In reply to this post by Dr Paul Dale
On Sat, Jan 05, 2019 at 08:45:37AM +1000, Dr Paul Dale wrote:
> I’m not sure about the quality of Android’s sources, but would expect them to be decent.

Android is just a Linux kernel. It always had /dev/urandom. Oreo
(8.0) requires at least Linux kernel 4.4. There were no
requirements for the kernel version before this. Some devices with
Marshmallow (6.0) will run a Linux kernel with 3.18, but not all
of them.


Kurt

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: RNG behavior by default

Steffen Nurpmeso-2
In reply to this post by Kurt Roeckx
Good evening.

Please excuse the late reply.

Kurt Roeckx wrote in <[hidden email]>:
 |On Fri, Jan 04, 2019 at 02:48:48PM +0100, Steffen Nurpmeso wrote:
 |> Dr. Matthias St. Pierre wrote in <450169f8ca7c43d1841c4c8052e78c72@Ex13.\
 |> ncp.local>:
 ...
 |>|2. If something goes wrong with PRNG initialization, that it will fail \
 |>|hard rather than fall back to something less secure. And if so how \
 |>|I detect such a failure.
 |>|
 |>|If the (re-)seeding fails, the DRBG enters an error state. When you \
 |>|try to generate random bytes it will detect the error state and try
 |>|automatically to heal the error state by reinstantiating. But if \
 |>|reseeding \
 |>|fails, it will return and error code and not generate any pseudo random \
 |>|bytes.
 |>|
 |>|Citing from the manual pages:
 |>  ...
 |>| As a normal application developer, you do not have to worry about \
 |>| any details, just use RAND_bytes(3)
 |>| to obtain random data. Having said that, there is one important rule \
 |>| to obey: Always check the error
 |>| return value of RAND_bytes(3) and do not take randomness for granted.
 |>|
 |>| https://www.openssl.org/docs/man1.1.1/man7/RAND.html
 |>
 |> That is new however, _imho_.  The wording of RAND_bytes(3) (still)
 |> says that "an error occurs [.if.] not [been] seeded with enough
 |> [data]", and RAND_status(3) returns 1 if the PRNG "has been seeded
 |> with enough data".  So if it is seeded it is seeded, in my
 |> understanding anything further on up the road only mixes in noise
 |> (which likely will undergo further maths and be stirred into the
 |> pool, i have not looked, actually).  I do not test RAND_bytes(3)
 |> return (yet), because i have ensured the PRNG is sufficiently
 |> seeded, and RAND_status(3) returns success, before RAND_bytes(3)
 |> is used the first time.
 |
 |For 1.1.0 and older that works, because they do not reseed. Since
 |1.1.1 it does reseed, and if the reseed fails, it will go to an
 |error state. So yes, this is new behavior.

Thanks.  Well this is good to know, i have to think about how
i will deal with this.

(I am also really interested and will look into OpenSSL to see if
the abort() that seems to happen if the initial seed fails is in
a linker-resolved constructor, and if not, why later failures do
not also abort.  Yes, while i am going, the full truth is that
i do not like it, i do not like that possibly a constructor call
is made for something that is possibly not needed, if it is done
like that, that someone simply aborts because of a some internal
PRNG, especially so if it is not in a constructor call and if
errors can and are expected to be handled by PRNG users anyway,
i do not like that the stuff is instantiated in all threads, which
in addition requires forks handlers to be installed all over the
place.  Now, on NetBSD for example, and because of libraries i am
linking in, i do have two complete sets of fork handlers and PRNG
buffers all over the place, one for NetBSD LibC, one for OpenSSL.
In my unfortunately not so humble opinion this looks like a modern
white first world child.  Including the postural defect.

I am sorry for this rant.  Yeah, i think OpenSSL even offers PRNG
objects which i can use and control directly, and through this
i regain a bit control of resource usage etc.  Without rewriting
software in order to pregenerate randoms in the main( manager)
thread to pass them through to jobs in thread pools.  Just look
for those and do not use the TSD/fork-aware RAND_bytes().
But before i become a total troll.. the Linux kernel that drives
the world from smallest to hugest has one internal entropy pool
that feeds two public pools, whereas i the lucent little hobby
server from user space get an armada of these.  Wow.

It is not your fault of course.  I am sorry for this rant.
Thanks for your answer.)

How do i deal with this?  Maybe i simply always use my builtin
ARC4, it is so small, or upgrade to ChaCha20 or so, and add
something like a Blake2 filter to protect that pool, and only
serve the output of it.  Or something.

 |The RAND_bytes and RAND_status manpages can clearly be improved.
 |Since you always have to check RAND_bytes's return value now,
 |RAND_status is mostly useless.

Yes, in my opinion good documentation is hard, almost impossible.
I am not using it, but i always enjoy reading the concise
(concise!  Says Mr. McIlroy.) yet helpful manuals from the really
great guys, for example the Plan9 manuals regarding programming.
Maybe a bit too concise for end-users, except academics.

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: RNG behavior by default

Steffen Nurpmeso-2
In reply to this post by Matthias St. Pierre
Dr. Matthias St. Pierre wrote in <b89b48fce6b54eadaba632402d789ee9@Ex13.\
ncp.local>:
 |I agree with Kurt, except for one point:
 |
 |> The RAND_bytes and RAND_status manpages can clearly be improved.
 |
 |Both manpages got an update during the DRBG rewrite (by me) and I don't
 |see any contradiction. You bring it to the point yourself:

I had a superficial look yesterday, but i think i have to reread
them in total, anyway.

 |> So _IF_ it is seeded it is seeded...
 |
 |It is true that the DRBG will automatically seed, but it is equally true
 |that it can still end up in an unseeded (error) state, if no suitable \
 |entropy
 |source is available. And since this can also happen during reseeding (which
 |in particular is enforced after a fork), it is always necessary to \
 |check the return
 |value of the RAND_bytes() function. Because in the error state, the \
 |buffer is not
 |filled at all.

That is really bad.  Of course you had to do it like this, and you
surely have looked around to see what servers and other software
which use OpenSSL do with the PRNG after forking (i.e., whether
they reiterate the [RAND_file_name(),] RAND_load_file(),
[:[RAND_add(),] RAND_status()], [RAND_write_file()] dance as
documented, or not).

I think i will move away from RAND_ then, nonetheless, and at
least for the things i have control of.
But i will definitely reread the manual now.

Thanks for your answer.
Ciao and a nice weekend from Germany,

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: RNG behavior by default

Kurt Roeckx
In reply to this post by Steffen Nurpmeso-2
On Sat, Jan 05, 2019 at 08:33:18PM +0100, Steffen Nurpmeso wrote:
>
> (I am also really interested and will look into OpenSSL to see if
> the abort() that seems to happen if the initial seed fails is in
> a linker-resolved constructor, and if not, why later failures do
> not also abort.

We do not call abort(). A function like RAND_bytes() or
RAND_status() will just return an error.

> Yes, while i am going, the full truth is that
> i do not like it, i do not like that possibly a constructor call
> is made for something that is possibly not needed, if it is done
> like that, that someone simply aborts because of a some internal
> PRNG, especially so if it is not in a constructor call and if
> errors can and are expected to be handled by PRNG users anyway,

RAND_bytes() has always documented that it can fail. Most function
calls can return errors for various reasons. Not checking for such
errors is in my opinion a bad style.

> i do not like that the stuff is instantiated in all threads

It's only created in each thread that tries to the RNG. I'm not
sure what exactly your problem with it is. Either we need a global
lock, or we need an RNG per thread. We went with the per thread
RNG.

We have a master DRBG that you can get with
RAND_DRBG_get0_master(). I recommend that you do not use it. It
requires that you take an external lock to use it. Internally we
lock it, but there is no way for you to use the same lock.

> which
> in addition requires forks handlers to be installed all over the
> place.

OpenSSL adds it's own fork handler now. You should not need to do
anything on fork related to OpenSSL.

> the Linux kernel that drives
> the world from smallest to hugest has one internal entropy pool
> that feeds two public pools, whereas i the lucent little hobby
> server from user space get an armada of these.  Wow.

Linux has a master pool, and a pool per core for the very same
reason as we also have a master DRBG and per thread DRBG.


Kurt

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: RNG behavior by default

Matthias St. Pierre
In reply to this post by Steffen Nurpmeso-2
>  |Both manpages got an update during the DRBG rewrite (by me) and I don't
>  |see any contradiction. You bring it to the point yourself:
>
> I had a superficial look yesterday, but i think i have to reread
> them in total, anyway.

Yes, please start with RAND(7) and RAND_DRBG(7).

> That is really bad.  Of course you had to do it like this, and you
> surely have looked around to see what servers and other software
> which use OpenSSL do with the PRNG after forking (i.e., whether
> they reiterate the [RAND_file_name(),] RAND_load_file(),
> [:[RAND_add(),] RAND_status()], [RAND_write_file()] dance as
> documented, or not).

I really don't understand your frustration: the new PRNG was designed
to relieve the application from the burden of seeding. It is easier to use,
not more complicated: No need to call RAND_add(), RAND_seed(),
RAND_load_file() and all this stuff. Just make sure the os entropy sources
are available and then simply use RAND_bytes(). But don't expect it to
succeed always.

> I think i will move away from RAND_ then, nonetheless, and at
> least for the things i have control of.

I don't think this is a good idea. In the case when there is no suitable entropy
source, the OpenSSL RNG will fail to generate random bytes, indeed.
But how do you expect your application to seed the RNG properly in this
case? What is your application's entropy source? If you have one, which
OpenSSL does not handle yet, you could theoretically register your
own get_entropy callback for the master DRBG at application startup time.

But if you don't have a better entropy source than OpenSSL, you are bound to
fail, too. And isn't it better for your application to fail gracefully in this case
than to provide snake oil security?

HTH,
Matthias


P.S: As for automatic reseeding: There are potential issues when upgrading
applications which do a fork and chroot from 1.1.0 to 1.1.1. Because in
version 1.1.1, the application will reseed after fork, and the reseeding can fail
if the application developer (or administrator) did not pay attention to provide
a random device in the chroot jail. This was no problem with 1.1.0 and below,
because the RNG never reseeded automatically.

OpenSSL does everything to avoid such a chroot reseeding failure (#6432 and #7437).
Also, on modern linux systems OpenSSL will prefer the getrandom system call
which does not have this limitation of the random devices.


https://github.com/openssl/openssl/pull/6432
https://github.com/openssl/openssl/pull/7437



--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: RNG behavior by default

Steffen Nurpmeso-2
In reply to this post by Kurt Roeckx
A wonderful Monday in the beautiful Winter time i wish.

I am sorry for the late reply again, i got a bug report for the
mailer i maintain, and from a long time user.
I hope it is ok that i compress the answers in one message, i am
talking much too much...

Kurt Roeckx wrote in <[hidden email]>:
 |On Sat, Jan 05, 2019 at 08:33:18PM +0100, Steffen Nurpmeso wrote:
 |>
 |> (I am also really interested and will look into OpenSSL to see if
 |> the abort() that seems to happen if the initial seed fails is in
 |> a linker-resolved constructor, and if not, why later failures do
 |> not also abort.
 |
 |We do not call abort(). A function like RAND_bytes() or
 |RAND_status() will just return an error.

Aah, i see.  I obviously had a completely false impression of the
new design.  I do not know where this came from, if i now search
in doc/ of the [master] there no mention of such wording, neither
abort nor exit nor whatever.  Hmm.?

  ...
 |RAND_bytes() has always documented that it can fail. Most function
 |calls can return errors for various reasons. Not checking for such
 |errors is in my opinion a bad style.

Well i, and until just now, interpreted the PRNG as something like
a "cipher stream", which simply applies (magic, to me) math to
a buffer of undefined content and an algorithm-chosen size.  Just
like MD5, which' RFC (1321) does not have error conditions.  (A
digest, but i had the RFC number at hand.)

So, to me.., i do not see any possible error condition, since the
initial seeding has been testified with RAND_status().

This is different now, and i will change the implementation as
soon as possible.  (This week.)

 |> i do not like that the stuff is instantiated in all threads
 |
 |It's only created in each thread that tries to the RNG. I'm not
 |sure what exactly your problem with it is. Either we need a global
 |lock, or we need an RNG per thread. We went with the per thread
 |RNG.

All my fault.  As a side note, i remember a commit message of
DragonFlyBSD and M. Dillon fly by (some years ago) where he
utilized the effect of SMP races as a random source.  If i recall
correctly.  Of course you had to protect the reseed count or
whatever (but there atomic cas or a spinlock would suffice
i guess).

 |We have a master DRBG that you can get with
 |RAND_DRBG_get0_master(). I recommend that you do not use it. It
 |requires that you take an external lock to use it. Internally we
 |lock it, but there is no way for you to use the same lock.
 |
 |> which
 |> in addition requires forks handlers to be installed all over the
 |> place.
 |
 |OpenSSL adds it's own fork handler now. You should not need to do
 |anything on fork related to OpenSSL.

It is just, to me, lack of standard.  But i do not dlopen() you,
and i do not dlclose() you.  In this scenario libraries may do use
anything, install atexit, onfork, whatever.  Other SSL libraries
even removed the possibility to hook the memory allocation
methods!

 |> the Linux kernel that drives
 |> the world from smallest to hugest has one internal entropy pool
 |> that feeds two public pools, whereas i the lucent little hobby
 |> server from user space get an armada of these.  Wow.
 |
 |Linux has a master pool, and a pool per core for the very same
 |reason as we also have a master DRBG and per thread DRBG.

I have read [1], but not that thorougly.  I was referring to the
user space interface, anyway.  (The thing is, i would have
preferred reading a one or two pages abstract instead of flying
over 138 pages.  I seem to know he knows.  I have read some
dissertations in my life, and i like the dedication if people
translate a complete system manual to German there, or show in
detail that they know know know their topic.  Having said that,
a very nice such work i have read from the Uni Kaiserslautern, it
was about 30 pages and it had been clear that "he can", shaking
off his hand.  Why need more?  Very good.  Imho.)

  [1] https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/Publications/Studies/LinuxRNG/LinuxRNG_EN.pdf?__blob=publicationFile&v=7

 --End of <[hidden email]>

Dr. Matthias St. Pierre wrote in <bb6e16e3fee246e887f185d3fbbfd3a2@Ex13.\
ncp.local>:
 |>|Both manpages got an update during the DRBG rewrite (by me) and I don't
 |>|see any contradiction. You bring it to the point yourself:
 |>
 |> I had a superficial look yesterday, but i think i have to reread
 |> them in total, anyway.
 |
 |Yes, please start with RAND(7) and RAND_DRBG(7).

Thanks.  I did just now.

 |> That is really bad.  Of course you had to do it like this, and you
 |> surely have looked around to see what servers and other software
 |> which use OpenSSL do with the PRNG after forking (i.e., whether
 |> they reiterate the [RAND_file_name(),] RAND_load_file(),
 |> [:[RAND_add(),] RAND_status()], [RAND_write_file()] dance as
 |> documented, or not).
 |
 |I really don't understand your frustration: the new PRNG was designed
 |to relieve the application from the burden of seeding. It is easier to use,
 |not more complicated: No need to call RAND_add(), RAND_seed(),
 |RAND_load_file() and all this stuff. Just make sure the os entropy sources
 |are available and then simply use RAND_bytes(). But don't expect it to
 |succeed always.

And may i ask, what to do if it fails?  Try it in a loop a finite
number of times, assuming it over-and-over detects it is in an
error state and tries to reseed unless that succeeds, and bail if
still without success thereafter?

Talking about the only program in the public that i represent:
i will not abort that one just because a PRNG fails to produce
some bytes that this program uses for the purpose of, well, having
some (pseudo but) random bytes; because: this is _completely_
disproportionate to the use cases of those random numbers.
I guess the same could be said about UUIDS generated via PRNGs
instead of how the standard defines them, etc. etc.
Please do not say "use random(3) or rand(3)" now, please.
So what to do?

 |> I think i will move away from RAND_ then, nonetheless, and at
 |> least for the things i have control of.
 |
 |I don't think this is a good idea. In the case when there is no suitable \
 |entropy
 |source, the OpenSSL RNG will fail to generate random bytes, indeed.
 |But how do you expect your application to seed the RNG properly in this
 |case? What is your application's entropy source? If you have one, which

Yes, that is the problem.  We do have several possibilities for
VAL_RANDOM.  Some of them only seed our builtin ARC4
implementation, which until now exposes the internal buffer as
such (sufficient for our purpose yet).  Examples of this are
/dev/urandom, Linux getrandom as system call or library call,
aehh...  If that seeding fails, we create a completely
unscientific homebrew seed via CLOCK_REALTIME or gettimeofday(2),
whatever is available, wildly XORing through the buffer, further
randomizing those weakly through the well-known algorithm from
"Random number generators: good ones are hard to find", Park and
Miller, Communications of the ACM, vol. 31, no. 10, October 1988,
p. 1195 (a_aux_rand_weak()).  This is the code:

   if(n_poption & n_PO_D_V)
      n_err(_("P(seudo)R(andomNumber)G(enerator): creating homebrew seed\n"));
   for(seed = (uintptr_t)a_aux_rand & UI32_MAX, rnd = 21; rnd != 0; --rnd){
      for(u.i = n_NELEM(a_aux_rand->b32); u.i-- != 0;){
         ui32_t t, k;

# ifdef mx_HAVE_CLOCK_GETTIME
         clock_gettime(CLOCK_REALTIME, &ts);
         t = (ui32_t)ts.tv_nsec;
# else
         gettimeofday(&ts, NULL);
         t = (ui32_t)ts.tv_usec;
# endif
         if(rnd & 1)
            t = (t >> 16) | (t << 16);
         a_aux_rand->b32[u.i] ^= a_aux_rand_weak(seed ^ t);
         a_aux_rand->b32[t % n_NELEM(a_aux_rand->b32)] ^= seed;
         if(rnd == 7 || rnd == 17)
            a_aux_rand->b32[u.i] ^= a_aux_rand_weak(seed ^ (ui32_t)ts.tv_sec);
         k = a_aux_rand->b32[u.i] % n_NELEM(a_aux_rand->b32);
         a_aux_rand->b32[k] ^= a_aux_rand->b32[u.i];
         seed ^= a_aux_rand_weak(a_aux_rand->b32[k]);
         if((rnd & 3) == 3)
            seed ^= su_prime_lookup_next(seed);
      }

So this is totally unscientific nonsense and overkill in one, but
at least we mix the low bits into the upper ones of the smallest
moving time fractions, and as such perform better than many
C libraries seem to have done in the past, as far as i know.
I think this is the best i can do.  Wait.  We _could_ check
whether we have subsecond precision sleep, and do nanosleep(0) do
gain some kind of sched_yield().

Others completely replace all this code: if we find a native
arc4random(3) (posix_random(3) will not become true,
unfortunately) we use that.  If we are linked against OpenSSL, we
do use yours exclusively -- but this possibly has to be changed.

 |OpenSSL does not handle yet, you could theoretically register your
 |own get_entropy callback for the master DRBG at application startup time.

So by hooking in such after the initial seeding succeeded (as
testified via RAND_status()) i can ensure that further reseeding
goes over my table and will (thus) always succeed?  I will look
into this possibility, thanks for the suggestion!

 |But if you don't have a better entropy source than OpenSSL, you are \
 |bound to
 |fail, too. And isn't it better for your application to fail gracefully \
 |in this case
 |than to provide snake oil security?

Well, as above, it is disproportionate to the use case of randoms
here.  (But this ARC4 stuff is a port of an old C++ library class,
and will move to an external C library in the future.  That is to
say, for now we could get by simply with a_aux_rand_weak(), but
since this code is old and what i have used so long.)

And i for one think that "snake oil security" is a wording much
too strong in the case of reseeding of a very secure stream
cipher, the entropy pool of which is protected by a very secure
distorting digest.  I do not know about forward security, however.
And i have read in the documented quoted above however, that the
in-between-boots random entropy file is not even counted against
security no more, which is why sshd blocks booting a minute.
That i find total overkill.

 |P.S: As for automatic reseeding: There are potential issues when upgrading
 |applications which do a fork and chroot from 1.1.0 to 1.1.1. Because in
 |version 1.1.1, the application will reseed after fork, and the reseeding \
 |can fail
 |if the application developer (or administrator) did not pay attention \
 |to provide
 |a random device in the chroot jail. This was no problem with 1.1.0 \
 |and below,
 |because the RNG never reseeded automatically.
 |
 |OpenSSL does everything to avoid such a chroot reseeding failure (#6432 \
 |and #7437).
 |Also, on modern linux systems OpenSSL will prefer the getrandom system call
 |which does not have this limitation of the random devices.
 |
 |https://github.com/openssl/openssl/pull/6432
 |https://github.com/openssl/openssl/pull/7437

Thanks for the information.  (It does not apply here, but thank
you nonetheless!)

 --End of <[hidden email]>

Ciao from Germany,

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: RNG behavior by default

OpenSSL - User mailing list
Small corrections below:

On 07/01/2019 19:31, Steffen Nurpmeso wrote:
...


 |> That is really bad.  Of course you had to do it like this, and you
 |> surely have looked around to see what servers and other software
 |> which use OpenSSL do with the PRNG after forking (i.e., whether
 |> they reiterate the [RAND_file_name(),] RAND_load_file(),
 |> [:[RAND_add(),] RAND_status()], [RAND_write_file()] dance as
 |> documented, or not).
 |
 |I really don't understand your frustration: the new PRNG was designed
 |to relieve the application from the burden of seeding. It is easier to use,
 |not more complicated: No need to call RAND_add(), RAND_seed(),
 |RAND_load_file() and all this stuff. Just make sure the os entropy sources
 |are available and then simply use RAND_bytes(). But don't expect it to
 |succeed always.

And may i ask, what to do if it fails?  Try it in a loop a finite
number of times, assuming it over-and-over detects it is in an
error state and tries to reseed unless that succeeds, and bail if
still without success thereafter?

Talking about the only program in the public that i represent:
i will not abort that one just because a PRNG fails to produce
some bytes that this program uses for the purpose of, well, having
some (pseudo but) random bytes; because: this is _completely_
disproportionate to the use cases of those random numbers.
I guess the same could be said about UUIDS generated via PRNGs
instead of how the standard defines them, etc. etc.
Please do not say "use random(3) or rand(3)" now, please.
So what to do?

"Just failing" is precisely to allow you (and others) to do something
sensible without the random bytes, such as using something less secure,
or doing a polite error reporting to the user.



 |> I think i will move away from RAND_ then, nonetheless, and at
 |> least for the things i have control of.
 |
 |I don't think this is a good idea. In the case when there is no suitable \
 |entropy
 |source, the OpenSSL RNG will fail to generate random bytes, indeed.
 |But how do you expect your application to seed the RNG properly in this
 |case? What is your application's entropy source? If you have one, which

Yes, that is the problem.  We do have several possibilities for
VAL_RANDOM.  Some of them only seed our builtin ARC4
implementation, which until now exposes the internal buffer as
such (sufficient for our purpose yet).  Examples of this are
/dev/urandom, Linux getrandom as system call or library call,
aehh...  If that seeding fails, we create a completely
unscientific homebrew seed via CLOCK_REALTIME or gettimeofday(2),
whatever is available, wildly XORing through the buffer, further
randomizing those weakly through the well-known algorithm from
"Random number generators: good ones are hard to find", Park and
Miller, Communications of the ACM, vol. 31, no. 10, October 1988,
p. 1195 (a_aux_rand_weak()).  This is the code:

Note that since that ancient article, ARC4 was not only invented,
but also found too insecure for modern use.



Enjoy

Jakob
-- 
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded 

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: RNG behavior by default

Steffen Nurpmeso-2
Good evening.

Jakob Bohm via openssl-users wrote in <95bceb59-b299-015a-f9c2-e2487a699\
[hidden email]>:
 |Small corrections below:
 |
 |On 07/01/2019 19:31, Steffen Nurpmeso wrote:
 |
 ||...
 |||RAND_load_file() and all this stuff. Just make sure the os entropy \
 |||sources
 |||are available and then simply use RAND_bytes(). But don't expect it to
 |||succeed always.
 |
 ||And may i ask, what to do if it fails?  Try it in a loop a finite
 ||number of times, assuming it over-and-over detects it is in an
 ||error state and tries to reseed unless that succeeds, and bail if
 ||still without success thereafter?
 |
 ||Talking about the only program in the public that i represent:
 ||i will not abort that one just because a PRNG fails to produce
 ||some bytes that this program uses for the purpose of, well, having
 ||some (pseudo but) random bytes; because: this is _completely_
 ||disproportionate to the use cases of those random numbers.
 ||I guess the same could be said about UUIDS generated via PRNGs
 ||instead of how the standard defines them, etc. etc.
 ||Please do not say "use random(3) or rand(3)" now, please.
 ||So what to do?
 |
 |"Just failing" is precisely to allow you (and others) to do something
 |sensible without the random bytes, such as using something less secure,
 |or doing a polite error reporting to the user.

I have to say i struggle with the OpenSSL project release
announcements.  Really.  I see a short mail which simply points
users to some (granted super duper) web site (with custom fonts
and such -- it must look fantastic on such a tablet thing that
i do not own!), and on these pages, which i, unfortunately and
until now, over and over again look at, ... to see nothing.

I mean, OpenSSL is such a vivid part of the infrastructure, top
that with security, of the world, it is used a billion of times
each and every day, and how many people really now about the
details inside this library with the terrible interface names (a
few minutes ago i discovered the also new ADMISSIONS.pod
documentation, and this is a statement for a language with
namespaces and method-on-object not instance-on-function; i always
think by myself that some university should perform neurological
science on what looking at

  const STACK_OF(ADMISSIONS) *ADMISSION_SYNTAX_get0_contentsOfAdmissions(const ADMISSION_SYNTAX *as);

from 9 to 5 (or 14 to 04 or whatever)/365-52 actually causes, how
do you say that in english?, what damages to the brain it causes,
anyway.)

The thing is, i have just seen the release candidate announcement
for bash 5.0 a while ago, and know what my own release message
will (have to) look like.

I did know nothing of all that.

 |||> I think i will move away from RAND_ then, nonetheless, and at
 |||> least for the things i have control of.
 |||
 |||I don't think this is a good idea. In the case when there is no suitable \
 |||\
 |||entropy
 |||source, the OpenSSL RNG will fail to generate random bytes, indeed.
 |||But how do you expect your application to seed the RNG properly in this
 |||case? What is your application's entropy source? If you have one, which
 |
 ||Yes, that is the problem.  We do have several possibilities for
 ||VAL_RANDOM.  Some of them only seed our builtin ARC4
 ||implementation, which until now exposes the internal buffer as
 ...
 ||randomizing those weakly through the well-known algorithm from
 ||"Random number generators: good ones are hard to find", Park and
 ||Miller, Communications of the ACM, vol. 31, no. 10, October 1988,
 ||p. 1195 (a_aux_rand_weak()).  This is the code:
 ...
 |Note that since that ancient article, ARC4 was not only invented,
 |but also found too insecure for modern use.

I seem to know that for things like save transport.
I have for example suppressed adding the

   for(u.i = 5 * sizeof(a_aux_rand->b8); u.i != 0; --u.i)
      a_aux_rand_get8();

which follows the unscientific seeding.  (And which likely is also
unscientific given that we have been seeded with random data
rather than with key/nonce ... whatever.)

 |Enjoy

Not really!  For example i read

  The reseeding default values are applied only during creation of
  a DRBG instance.  To ensure that they are applied to the global
  and thread-local DRBG instances (<master>, resp. <public> and
  <private>), it is necessary to call
  RAND_DRBG_set_reseed_defaults() before creating any thread and
  before calling any cryptographic routines that obtain random
  data directly or indirectly.

Does this give me at least the promise that OPENSSL_init_ssl() can
be called without rendering RAND_DRBG_set_reseed_defaults(0,0,0,0)
effectively a useless call?
I see that said function may return error, but it seems that the
MAX_RESEED_INTERVAL it fails to accept is not a public variable!
I tend to do the much more expensive dance with
RAND_DRBG_get0_public() and RAND_DRBG_get0_master() and
individually applying RAND_DRBG_set_reseed_time_interval() as well
as RAND_DRBG_set_reseed_interval() on each of them.
Is this even allowed without calling OPENSSL_init_ssl() first?

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: RNG behavior by default

OpenSSL - User mailing list
On 07/01/2019 22:31, Steffen Nurpmeso wrote:
> Good evening.
>
> Jakob Bohm via openssl-users wrote in <95bceb59-b299-015a-f9c2-e2487a699\
> [hidden email]>:
>   |Small corrections below:
>   | ...

Note that I do not represent the project at all, I am just another user
trying to help you.

As such, I cannot really respond to your criticism of other project
aspects, some of which I myself don't like either, while others are
obvious to me as someone who has more or less followed along since this
was a two-man Australian project.

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded
--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: RNG behavior by default

Steffen Nurpmeso-2
Jakob Bohm via openssl-users wrote in <07f4dea3-1a62-0c8c-76a4-cbe56abc8\
[hidden email]>:
 |On 07/01/2019 22:31, Steffen Nurpmeso wrote:
 |> Good evening.
 |>
 |> Jakob Bohm via openssl-users wrote in <95bceb59-b299-015a-f9c2-e2487a699\
 |> [hidden email]>:
 |>|Small corrections below:
 |>| ...
 |
 |Note that I do not represent the project at all, I am just another user
 |trying to help you.

Surprise!  Indeed i do see your name only once in the commits!
So now, what to do?  What is due?  Congratulations, or even an
excuse?  I had a very false impression indeed.  Excuse this miss
of mine, please.

 |As such, I cannot really respond to your criticism of other project
 |aspects, some of which I myself don't like either, while others are
 |obvious to me as someone who has more or less followed along since this
 |was a two-man Australian project.

Then you lead me by more than a decade in respect to "following"
the project.  (I only saw, otherwise offline, the headers and
libraries fly by.)  (And in my granted lengthy text i thought
about changing my ARC4 thing to ChaCha, and not feed the bytes of
the pool as such, but run a Blake2 digest on top of it, and feed
the output of that.  For now, however, this ARC4 PRNG, and then
also well seeded, is really good enough for my purpose(s).)

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: RNG behavior by default

Steffen Nurpmeso-2
In reply to this post by Steffen Nurpmeso-2
Steffen Nurpmeso wrote in <20190107183136.-eW61%[hidden email]>:
  ...
 |  ...
 ||RAND_bytes() has always documented that it can fail. Most function
 ...
 |So, to me.., i do not see any possible error condition, since the
 |initial seeding has been testified with RAND_status().
 |
 |This is different now, and i will change the implementation as
 |soon as possible.  (This week.)

So i did, we disable the OpenSSL reseeding by directly calling
RAND_DRBG_set_reseed_defaults() after calling OPENSSL_init_ssl(),
which i hope will always be possible.
Be warned that i gave credit to both of you.

I have seen DRBG offers a lot of possibilities to control what
OpenSSL does, also regarding the fork handlers and all that.
Thanks for these possibilities, it is a terribly huge interface,
but it allows users to have control on what happens, instead of
sitting on an intransparent black box!  Getting something going on
such a thing causes grief, as it is hacky and otherwise
troublesome!

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users