ideas on replacing where ERR_STATE is stored?

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

ideas on replacing where ERR_STATE is stored?

Rich Salz
An ERR_STATE block (crypto/err.c) is stored in a global hashtable keyed
by thread-id.  If threads are re-used (like the windows worker thread
concept), this scheme doesn't work.  It looks like replacing the internal
int_thread_get_item/int_thread_set_item pair, or the higher-level
ERR_get_state/ERR_remove_state pair is the way to go.

The complications are (1) the code tries very hard to not expose any
functions or function-dispatch structures; and (2) I need to do this at
startup-time, and the code tries VERY hard to make err_fns_check calls all
over the place -- that latter part means replacing the
ERR_get/remove_state seems like the cleaner solution.

Attached is a proposed diff.  Any comments?

        /r$

--
Rich Salz                  Chief Security Architect
DataPower Technology       http://www.datapower.com
XS40 XML Security Gateway  http://www.datapower.com/products/xs40.html

errdiff.txt (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: ideas on replacing where ERR_STATE is stored?

Steven Reddie
Rich,

How do you mean that the scheme doesn't work with worker threads?  Doesn't
judicious use of ERR_remove_state overcome any problems of a new job on a
given thread "remembering" the error state of the previous job(s)?

-----Original Message-----
From: [hidden email] [mailto:[hidden email]]
On Behalf Of Rich Salz
Sent: Wednesday, 28 September 2005 1:30 PM
To: [hidden email]
Subject: ideas on replacing where ERR_STATE is stored?

An ERR_STATE block (crypto/err.c) is stored in a global hashtable keyed by
thread-id.  If threads are re-used (like the windows worker thread concept),
this scheme doesn't work.  It looks like replacing the internal
int_thread_get_item/int_thread_set_item pair, or the higher-level
ERR_get_state/ERR_remove_state pair is the way to go.

The complications are (1) the code tries very hard to not expose any
functions or function-dispatch structures; and (2) I need to do this at
startup-time, and the code tries VERY hard to make err_fns_check calls all
over the place -- that latter part means replacing the ERR_get/remove_state
seems like the cleaner solution.

Attached is a proposed diff.  Any comments?

        /r$

--
Rich Salz                  Chief Security Architect
DataPower Technology       http://www.datapower.com
XS40 XML Security Gateway  http://www.datapower.com/products/xs40.html

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: ideas on replacing where ERR_STATE is stored?

Rich Salz
> How do you mean that the scheme doesn't work with worker threads?  Doesn't
> judicious use of ERR_remove_state overcome any problems of a new job on a
> given thread "remembering" the error state of the previous job(s)?

That's a re-usable worker-thread model.

Another model is to have "n" threads, where n is the number of CPU's in
the system and use non-blocking I/O to pick up and "put down" multiple SSL
"sessions" within a single thread.

        /r$

--
Rich Salz                  Chief Security Architect
DataPower Technology       http://www.datapower.com
XS40 XML Security Gateway  http://www.datapower.com/products/xs40.html

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: ideas on replacing where ERR_STATE is stored?

Steven Reddie
Do you mean using select() to handle multiple simultaneous connections?

-----Original Message-----
From: [hidden email] [mailto:[hidden email]]
On Behalf Of Rich Salz
Sent: Wednesday, 28 September 2005 2:28 PM
To: Steven Reddie
Cc: [hidden email]
Subject: RE: ideas on replacing where ERR_STATE is stored?

> How do you mean that the scheme doesn't work with worker threads?  
> Doesn't judicious use of ERR_remove_state overcome any problems of a
> new job on a given thread "remembering" the error state of the previous
job(s)?

That's a re-usable worker-thread model.

Another model is to have "n" threads, where n is the number of CPU's in the
system and use non-blocking I/O to pick up and "put down" multiple SSL
"sessions" within a single thread.

        /r$

--
Rich Salz                  Chief Security Architect
DataPower Technology       http://www.datapower.com
XS40 XML Security Gateway  http://www.datapower.com/products/xs40.html

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: ideas on replacing where ERR_STATE is stored?

Steven Reddie
In reply to this post by Rich Salz
Ok, the reference to worker threads threw me.  Your fix seems not so much
related to whether the application is single- or multi-threaded, but for, in
effect, cooperatively multitasking jobs within a single thread; ie. you want
finer grained control to track error state for "jobs" (however you choose to
define them) rather than threads?

-----Original Message-----
From: Rich Salz [mailto:[hidden email]]
Sent: Thursday, 29 September 2005 12:15 AM
To: Steven Reddie
Subject: Re: ideas on replacing where ERR_STATE is stored?

> Do you mean using select() to handle multiple simultaneous connections?

basically, yes.

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: ideas on replacing where ERR_STATE is stored?

Michael Sierchio
In reply to this post by Steven Reddie
Steven Reddie wrote:
> Do you mean using select() to handle multiple simultaneous connections?

I'm late in catching this thread, but I'll wager that Rich would
use poll rather than select, or /dev/poll, or some such.  The
model he describes is the most efficient, but makes application
programming harder - essentially trading off the ease of single-task
worker threads for higher performance, but having to write a state
machine.

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: ideas on replacing where ERR_STATE is stored?

Steven Reddie
Hi Michael,

I'm familiar with that approach, having used it many times myself.  The
choice of poll over select isn't important since they're basically the same;
in fact, poll is sometimes implemented with select.  I was originally thrown
by "threads are re-used" and the reference to "worker threads" since the
select/poll approach is independent of threadedness and in fact is typically
used in single-threaded code.  Re-using worker threads sounded more like
what happens when a worker thread is finished a job and picks up the next
one from a queue, and in that case ERR_remove_state() does the job perfectly
just the way it is.  I see what Rich is getting at though since he means
select/poll, it just wasn't clear at the beginning.

Regards,

Steven

-----Original Message-----
From: [hidden email] [mailto:[hidden email]]
On Behalf Of Michael Sierchio
Sent: Friday, 14 October 2005 3:07 PM
To: [hidden email]
Subject: Re: ideas on replacing where ERR_STATE is stored?

Steven Reddie wrote:
> Do you mean using select() to handle multiple simultaneous connections?

I'm late in catching this thread, but I'll wager that Rich would use poll
rather than select, or /dev/poll, or some such.  The model he describes is
the most efficient, but makes application programming harder - essentially
trading off the ease of single-task worker threads for higher performance,
but having to write a state machine.

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: ideas on replacing where ERR_STATE is stored?

Michael Sierchio
Steven Reddie wrote:
> Hi Michael,
>
> I'm familiar with that approach, having used it many times myself.  The
> choice of poll over select isn't important since they're basically the same;
> in fact, poll is sometimes implemented with select.  

Who implements poll with select should suffer a fate worse than
death -- waking up a thousand sleeping threads to see if one
has some i/o ready is what poll was designed to avoid.

"But that's just my opinion..."
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: ideas on replacing where ERR_STATE is stored?

Lev Walkin
Michael Sierchio wrote:

> Steven Reddie wrote:
>
>> Hi Michael,
>>
>> I'm familiar with that approach, having used it many times myself.  The
>> choice of poll over select isn't important since they're basically the
>> same;
>> in fact, poll is sometimes implemented with select.  
>
>
> Who implements poll with select should suffer a fate worse than
> death -- waking up a thousand sleeping threads to see if one
> has some i/o ready is what poll was designed to avoid.

I think you're confusing a file I/O notification with mutual exclusion
and condition variables paradigm, another kind of IPC. The following
page provides necessary background information:

        http://www.kegel.com/c10k.html

Besides, some select() implementations are actually poll()-based.

--
Lev Walkin
[hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: ideas on replacing where ERR_STATE is stored?

Jack Lloyd
On Mon, Oct 17, 2005 at 02:43:19PM -0700, Lev Walkin wrote:

> Michael Sierchio wrote:
>
> >Steven Reddie wrote:
> >
> >>Hi Michael,
> >>
> >>I'm familiar with that approach, having used it many times myself.  The
> >>choice of poll over select isn't important since they're basically the
> >>same;
> >>in fact, poll is sometimes implemented with select.  
> >
> >
> >Who implements poll with select should suffer a fate worse than
> >death -- waking up a thousand sleeping threads to see if one
> >has some i/o ready is what poll was designed to avoid.
>
> I think you're confusing a file I/O notification with mutual exclusion
> and condition variables paradigm, another kind of IPC. The following
> page provides necessary background information:

I believe Michael is actually talking about the "thundering herd" problem, when
many processes are all waiting on a single event, which only one of them will
end up responding to. That is a classic problem affecting some uses of select
(and also accept, and IIRC a few other socket calls as well).

Jack
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: ideas on replacing where ERR_STATE is stored?

Michael Sierchio
Jack Lloyd wrote:

> I believe Michael is actually talking about the "thundering herd" problem, when
> many processes are all waiting on a single event, which only one of them will
> end up responding to. That is a classic problem affecting some uses of select
> (and also accept, and IIRC a few other socket calls as well).

Precisely what I was thinking, though I suppose I should have
actually said it. ;-)  Thanks.

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: ideas on replacing where ERR_STATE is stored?

Lev Walkin
Michael Sierchio wrote:

> Jack Lloyd wrote:
>
>> I believe Michael is actually talking about the "thundering herd"
>> problem, when
>> many processes are all waiting on a single event, which only one of
>> them will
>> end up responding to. That is a classic problem affecting some uses of
>> select
>> (and also accept, and IIRC a few other socket calls as well).
>
>
> Precisely what I was thinking, though I suppose I should have
> actually said it. ;-)  Thanks.

Yes, but your words

"Who implements poll with select should suffer a fate worse than
death -- waking up a thousand sleeping threads to see if one
has some i/o ready is what poll was designed to avoid."

are not substantiated. Poll() provides no advantage over select()
for the thundering herd problem.


--
Lev Walkin
[hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: ideas on replacing where ERR_STATE is stored?

Michael Sierchio
Lev Walkin wrote:

 > ... Poll() provides no advantage over select()
> for the thundering herd problem.

Sorry, I'm not here to chew your food for you.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: ideas on replacing where ERR_STATE is stored?

Lev Walkin
In reply to this post by Jack Lloyd
Jack Lloyd wrote:

> On Mon, Oct 17, 2005 at 02:43:19PM -0700, Lev Walkin wrote:
>
>>Michael Sierchio wrote:
[skip]

>>>Who implements poll with select should suffer a fate worse than
>>>death -- waking up a thousand sleeping threads to see if one
>>>has some i/o ready is what poll was designed to avoid.
>>
>>I think you're confusing a file I/O notification with mutual exclusion
>>and condition variables paradigm, another kind of IPC. The following
>>page provides necessary background information:
>
>
> I believe Michael is actually talking about the "thundering herd" problem, when
> many processes are all waiting on a single event, which only one of them will
> end up responding to. That is a classic problem affecting some uses of select
> (and also accept, and IIRC a few other socket calls as well).

Including poll().

--
Lev Walkin
[hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: ideas on replacing where ERR_STATE is stored?

Michael Sierchio
Lev Walkin wrote:

> Including poll().

A polling model may be built on /dev/poll or kernel queues, etc.
I made mention of /dev/poll in my first contribution to this
thread.  Go back to class.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: ideas on replacing where ERR_STATE is stored?

Steven Reddie
In reply to this post by Michael Sierchio
Michael,

What does the thundering herd problem have to do with select vs poll?  The
only typically useful advantages of poll over select that I am aware of are
that poll can monitor more descriptors than select and that it can monitor a
large number of descriptors more efficiently -- but this efficiency issue
has nothing to do with thundering herd.

I'm having trouble with the context here.  Rich is talking about simply
using select/poll to monitor multiple sockets.  Thundering herd is a problem
when a large number of threads are monitoring the same events.  While
select/poll can be used in such a way that thundering herd can be a problem,
for the simple single-thread select/poll loop it just isn't in the picture
-- how can a single thread suffer from the thundering herd problem?

Regards,

Steven

-----Original Message-----
From: [hidden email] [mailto:[hidden email]]
On Behalf Of Michael Sierchio
Sent: Tuesday, 18 October 2005 2:49 AM
To: [hidden email]
Subject: Re: ideas on replacing where ERR_STATE is stored?

Steven Reddie wrote:
> Hi Michael,
>
> I'm familiar with that approach, having used it many times myself.  
> The choice of poll over select isn't important since they're basically
> the same; in fact, poll is sometimes implemented with select.

Who implements poll with select should suffer a fate worse than death --
waking up a thousand sleeping threads to see if one has some i/o ready is
what poll was designed to avoid.

"But that's just my opinion..."
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]