Clients glomming onto a listener

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Clients glomming onto a listener

John Hollingum
Hi,

I have a service written in Perl,  running on Linux that presents a very
simple SSL listener.  When this service is hit,  it identifies the
connecting node from its certificate/peer address and just sends some
xml to them containing data from some files in the queue directory that
contains their data.

All the client does is to open a socket and start reading.

This works,  but it is susceptible to problems which I believe are
caused by clients with bad internet connections (the pathology suggests
this).  It seems that something unpleasant occurs in the SSL handshake
process which causes the socket to hang indefinitely.  Nobody else can
connect when this has happened.

I currently have a very crude monitor program that checks the activity
log of the main service and,  if things go suspiciously quiet it
restarts the listener service.  It also logs the result of a filtered
run of 'netstat' which shows who was involved with the problem.  Some
hosts feature a lot more regularly,  which leads me to believe is it
some net environment problem.

But that's all a distraction.  The fact is that the service shouldn't be
susceptible to the vagaries of what stupid clients or bad networks do.

I *do* get some action on my SSL_error_trap routine.  These errors log
nicely and don't cause a hang and don't seem to coincide with any hangs.

Pretty much immediately after the accept the program forks a handler,  
but the rogue clients must be glomming onto the main process before the
SSL negotiation is complete.

I can't help thinking that I should be able to tell SSL to have some
sensible (fairly aggressive) timeouts on connections that fail to
complete an SSL handshake.  Is this possible?  Does it sound like I'm
even identifying the problem correctly?

I'm wondering if I could get more control by using accept in
non-blocking mode.  Is this worth looking into?

I appreciate that if you suggest the use of any particular OpenSSL
features,  I will have to dig around to see if those facilities are
available via the IO::Socket::SSL library (or maybe more directly via
the Net::SSLeay library). Any thoughts appreciated.

Perl version v5.8.8
program uses

use IO::Socket::SSL;
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Clients glomming onto a listener

JoelKatz
On 5/10/2011 2:10 AM, John Hollingum wrote:

> I have a service written in Perl, running on Linux that presents a very
> simple SSL listener. When this service is hit, it identifies the
> connecting node from its certificate/peer address and just sends some
> xml to them containing data from some files in the queue directory that
> contains their data.
>
> All the client does is to open a socket and start reading.
>
> This works, but it is susceptible to problems which I believe are caused
> by clients with bad internet connections (the pathology suggests this).
> It seems that something unpleasant occurs in the SSL handshake process
> which causes the socket to hang indefinitely. Nobody else can connect
> when this has happened.

Well, that's what happens if you take a single-threaded listener and
tell it complete the SSL handshake.

> But that's all a distraction. The fact is that the service shouldn't be
> susceptible to the vagaries of what stupid clients or bad networks do.

Right, that's why a single-threaded listener that *ever* waits for a
client is a terrible design.

> Pretty much immediately after the accept the program forks a handler,
> but the rogue clients must be glomming onto the main process before the
> SSL negotiation is complete.

Calling 'fork' with an accepted SSL connection has all kinds of known
issues. The fundamental problem is that there are many operations that
must occur both before and after the 'fork', for different reasons, and
obviously can't do both.

> I can't help thinking that I should be able to tell SSL to have some
> sensible (fairly aggressive) timeouts on connections that fail to
> complete an SSL handshake. Is this possible? Does it sound like I'm even
> identifying the problem correctly?

It wouldn't help, since it's not SSL that needs to time out but the
socket itself. If you want to timeout the socket operations, you have to
do it. OpenSSL just reads and writes to and from the BIOs and thus the
socket.

> I'm wondering if I could get more control by using accept in
> non-blocking mode. Is this worth looking into?

That would help if your issue was blocking in 'accept'. Since you are a
single-threaded listener that has to handle multiple clients, why are
you doing *anything* with blocking sockets calls? That's an obvious red
flag.

Blocking socket operations only exist to allow sockets to behave like
terminals or files for applications that don't know anything specific
about sockets. Your application not only is sockets-specific
fundamentally but has to do something very unusual that requires
non-blocking operations.

Timeouts are a possible fix, so long as you only accept connections from
trusted clients in the first place. Otherwise, you create a trivial DoS
attack.

DS

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Clients glomming onto a listener

Wim Lewis-3

On 10 May 2011, at 4:13 PM, David Schwartz wrote:
> On 5/10/2011 2:10 AM, John Hollingum wrote:
>> Pretty much immediately after the accept the program forks a handler,
>> but the rogue clients must be glomming onto the main process before the
>> SSL negotiation is complete.
>
> Calling 'fork' with an accepted SSL connection has all kinds of known issues. The fundamental problem is that there are many operations that must occur both before and after the 'fork', for different reasons, and obviously can't do both.

You could accept just the TCP connection in the main process and do all of the SSL handshake in the forked process (I think IO::Socket::SSL->start_SSL() is what you want for that) --- this would not be a high-performance approach (no SSL session cache, fork overhead) but if it's fast enough it's fast enough.

It's possible to use openssl in a non-blocking, event-driven manner but I don't think Perl's SSL modules expose enough of the openssl API to do that.


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Clients glomming onto a listener

Eric S. Eberhard-2
In reply to this post by JoelKatz
I have found that fork() on modern machines as a negligible affect on
performance and in fact I almost always use inetd instead of writing
my own servers, mainly because it is dead reliable, easier to code,
and again seems to have negligible affect on performance.  One would
have to do millions upon millions of connects to notice or
care.  Having said that, I use AIX mostly, and that performs better
under load than Linux on Intel, and even Linux on the IBM p series
platform.  I would do it cheap and easy and worry about performance
after-the-fact. Eric

At 04:46 PM 5/10/2011, you wrote:

>On 10 May 2011, at 4:13 PM, David Schwartz wrote:
> > On 5/10/2011 2:10 AM, John Hollingum wrote:
> >> Pretty much immediately after the accept the program forks a handler,
> >> but the rogue clients must be glomming onto the main process before the
> >> SSL negotiation is complete.
> >
> > Calling 'fork' with an accepted SSL connection has all kinds of
> known issues. The fundamental problem is that there are many
> operations that must occur both before and after the 'fork', for
> different reasons, and obviously can't do both.
>
>You could accept just the TCP connection in the main process and do
>all of the SSL handshake in the forked process (I think
>IO::Socket::SSL->start_SSL() is what you want for that) --- this
>would not be a high-performance approach (no SSL session cache, fork
>overhead) but if it's fast enough it's fast enough.
>
>It's possible to use openssl in a non-blocking, event-driven manner
>but I don't think Perl's SSL modules expose enough of the openssl
>API to do that.
>
>
>______________________________________________________________________
>OpenSSL Project                                 http://www.openssl.org
>User Support Mailing List                    [hidden email]
>Automated List Manager                           [hidden email]


Eric S. Eberhard
(928) 567-3727          Voice
(928) 567-6122          Fax
(928) 301-7537                           Cell

Vertical Integrated Computer Systems, LLC
Metropolis Support, LLC

For Metropolis support and VICS MBA Support!!!!    http://www.vicsmba.com

Pictures of Snake in Spring

http://www.facebook.com/album.php?aid=115547&id=1409661701&l=1c375e1f49

Pictures of Camp Verde

http://www.facebook.com/album.php?aid=12771&id=1409661701&l=fc0e0a2bcf

Pictures of Land Cruiser in Sedona

http://www.facebook.com/album.php?aid=50953&id=1409661701

Pictures of Flagstaff area near our cabin

http://www.facebook.com/album.php?aid=12750&id=1409661701

Pictures of Cheryl in a Horse Show

http://www.facebook.com/album.php?aid=32484&id=1409661701


Pictures of the AZ Desert

http://www.facebook.com/album.php?aid=58827&id=1409661701

(You can see why we love this state :-) )








______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Clients glomming onto a listener

Victor Duchovni
On Wed, May 11, 2011 at 08:39:49AM -0700, Eric S. Eberhard wrote:

> I have found that fork() on modern machines as a negligible affect on
> performance and in fact I almost always use inetd instead of writing my own
> servers, mainly because it is dead reliable, easier to code, and again
> seems to have negligible affect on performance.  One would have to do
> millions upon millions of connects to notice or care.  Having said that, I
> use AIX mostly, and that performs better under load than Linux on Intel,
> and even Linux on the IBM p series platform.  I would do it cheap and easy
> and worry about performance after-the-fact. Eric

Let's not start an OS A is better than OS B discussion here. You can
safely fork single-threaded OpenSSL servers right after accept(3),
and handle the SSL connection in a child. This makes the memory-resident
session cache ineffective, but you can use callbacks to implement an
external (Berkeley DB similar or shared memory, ...) session cache.

Forking after SSL_accept() is tricky, since your parent process will
have partial SSL connections in progress for other clients when a given
handshake completes (event-based connection management) or will serialize
all handshakes, but as you've observed that's not a good option.

So, my suggestion is that a forking server is fine, just use an external
session cache. The Postfix SMTP server is an example of this model. There
before the TLS handshake, we also have an SMTP STARTTLS handshake, but
that does not alter the analysis in any substantive way, just a few more
packets to exchange before the TLS connection is ready.

Note, Postfix is pre-forking, rather than forking, so there is a pool
of processes, that serially accept connections, but this too does not
impact the design analysis.

    - You can use a single process with event-based I/O.
    - You can use multiple threads in a single process.
    - You can fork after accept(2) and use an external session cache
    - You can pre-fork and handle clients serially one per process,
      with re-use of processes for another client after a client hangs-up.
      This too requires an external session cache.

--
        Viktor.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Clients glomming onto a listener

Eric S. Eberhard-2
In reply to this post by Eric S. Eberhard-2
I was not trying to compare O/S, only point out that my experience is
more out of the AIX world than Linux world.

I also want to point out again what I was saying ... you don't need
to make a server and you don't need to fork() and all kinds of
complicated stuff if you write it for inetd.  You don't even need to
write socket code (stdin/stdout read/write is all you need).  The O/S
will create the processes and clean them up on disconnects and so
forth.  Unless you are super performance limited, this is the best
way to go because it always works and is always reliable (if inetd
fails to function on a Unix O/S then the machine is essentially toast
anyway).  In addition it is more easily portable if you care about
porting to more than one Unix.  Using select is not always supported,
socket flags not always the same, etc.  All a non-issue under inetd.

Eric

At 08:57 AM 5/11/2011, you wrote:

>On Wed, May 11, 2011 at 08:39:49AM -0700, Eric S. Eberhard wrote:
>
> > I have found that fork() on modern machines as a negligible affect on
> > performance and in fact I almost always use inetd instead of
> writing my own
> > servers, mainly because it is dead reliable, easier to code, and again
> > seems to have negligible affect on performance.  One would have to do
> > millions upon millions of connects to notice or care.  Having said that, I
> > use AIX mostly, and that performs better under load than Linux on Intel,
> > and even Linux on the IBM p series platform.  I would do it cheap and easy
> > and worry about performance after-the-fact. Eric
>
>Let's not start an OS A is better than OS B discussion here. You can
>safely fork single-threaded OpenSSL servers right after accept(3),
>and handle the SSL connection in a child. This makes the memory-resident
>session cache ineffective, but you can use callbacks to implement an
>external (Berkeley DB similar or shared memory, ...) session cache.
>
>Forking after SSL_accept() is tricky, since your parent process will
>have partial SSL connections in progress for other clients when a given
>handshake completes (event-based connection management) or will serialize
>all handshakes, but as you've observed that's not a good option.
>
>So, my suggestion is that a forking server is fine, just use an external
>session cache. The Postfix SMTP server is an example of this model. There
>before the TLS handshake, we also have an SMTP STARTTLS handshake, but
>that does not alter the analysis in any substantive way, just a few more
>packets to exchange before the TLS connection is ready.
>
>Note, Postfix is pre-forking, rather than forking, so there is a pool
>of processes, that serially accept connections, but this too does not
>impact the design analysis.
>
>     - You can use a single process with event-based I/O.
>     - You can use multiple threads in a single process.
>     - You can fork after accept(2) and use an external session cache
>     - You can pre-fork and handle clients serially one per process,
>       with re-use of processes for another client after a client hangs-up.
>       This too requires an external session cache.
>
>--
>         Viktor.
>______________________________________________________________________
>OpenSSL Project                                 http://www.openssl.org
>User Support Mailing List                    [hidden email]
>Automated List Manager                           [hidden email]


Eric S. Eberhard
(928) 567-3727          Voice
(928) 567-6122          Fax
(928) 301-7537                           Cell

Vertical Integrated Computer Systems, LLC
Metropolis Support, LLC

For Metropolis support and VICS MBA Support!!!!    http://www.vicsmba.com

Pictures of Snake in Spring

http://www.facebook.com/album.php?aid=115547&id=1409661701&l=1c375e1f49

Pictures of Camp Verde

http://www.facebook.com/album.php?aid=12771&id=1409661701&l=fc0e0a2bcf

Pictures of Land Cruiser in Sedona

http://www.facebook.com/album.php?aid=50953&id=1409661701

Pictures of Flagstaff area near our cabin

http://www.facebook.com/album.php?aid=12750&id=1409661701

Pictures of Cheryl in a Horse Show

http://www.facebook.com/album.php?aid=32484&id=1409661701


Pictures of the AZ Desert

http://www.facebook.com/album.php?aid=58827&id=1409661701

(You can see why we love this state :-) )








______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Clients glomming onto a listener

Gayathri Sundar-3
In reply to this post by Eric S. Eberhard-2
Eric, you must be really kidding this time :), servers with this architecture are susceptible to dos and what not..am sure for embedded systems where memory is a big limiting factor the best would be async design, also code becomes easily portable in future.

On Wed, May 11, 2011 at 10:39 AM, Eric S. Eberhard <[hidden email]> wrote:
I have found that fork() on modern machines as a negligible affect on performance and in fact I almost always use inetd instead of writing my own servers, mainly because it is dead reliable, easier to code, and again seems to have negligible affect on performance.  One would have to do millions upon millions of connects to notice or care.  Having said that, I use AIX mostly, and that performs better under load than Linux on Intel, and even Linux on the IBM p series platform.  I would do it cheap and easy and worry about performance after-the-fact. Eric

At 04:46 PM 5/10/2011, you wrote:

On 10 May 2011, at 4:13 PM, David Schwartz wrote:
> On 5/10/2011 2:10 AM, John Hollingum wrote:
>> Pretty much immediately after the accept the program forks a handler,
>> but the rogue clients must be glomming onto the main process before the
>> SSL negotiation is complete.
>
> Calling 'fork' with an accepted SSL connection has all kinds of known issues. The fundamental problem is that there are many operations that must occur both before and after the 'fork', for different reasons, and obviously can't do both.

You could accept just the TCP connection in the main process and do all of the SSL handshake in the forked process (I think IO::Socket::SSL->start_SSL() is what you want for that) --- this would not be a high-performance approach (no SSL session cache, fork overhead) but if it's fast enough it's fast enough.

It's possible to use openssl in a non-blocking, event-driven manner but I don't think Perl's SSL modules expose enough of the openssl API to do that.


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]


Eric S. Eberhard
<a href="tel:%28928%29%20567-3727" value="+19285673727" target="_blank">(928) 567-3727          Voice
<a href="tel:%28928%29%20567-6122" value="+19285676122" target="_blank">(928) 567-6122          Fax
<a href="tel:%28928%29%20301-7537" value="+19283017537" target="_blank">(928) 301-7537                           Cell

Vertical Integrated Computer Systems, LLC
Metropolis Support, LLC

For Metropolis support and VICS MBA Support!!!!    http://www.vicsmba.com

Pictures of Snake in Spring

http://www.facebook.com/album.php?aid=115547&id=1409661701&l=1c375e1f49

Pictures of Camp Verde

http://www.facebook.com/album.php?aid=12771&id=1409661701&l=fc0e0a2bcf

Pictures of Land Cruiser in Sedona

http://www.facebook.com/album.php?aid=50953&id=1409661701

Pictures of Flagstaff area near our cabin

http://www.facebook.com/album.php?aid=12750&id=1409661701

Pictures of Cheryl in a Horse Show

http://www.facebook.com/album.php?aid=32484&id=1409661701


Pictures of the AZ Desert

http://www.facebook.com/album.php?aid=58827&id=1409661701

(You can see why we love this state :-) )








______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Clients glomming onto a listener

Eric S. Eberhard-2
In reply to this post by Eric S. Eberhard-2
Performance is related to the application.  For example, a system
that accepts 10 SSL connects per year has different requirements than
one that accepts 1000 per second.  Obviously there is a middle
ground.  My point is that theoretical performance differences are
very real in the later case, and of no consequence in the first case.

Cost of software development and upkeep and system management is much
lower using say inetd and not bothering to make a server.  I have
systems with thousands of SSL connections per minute, holding
500-1000 at a time, going through inetd on a modest AIX box and have
zero performance issue.  Don't even notice they are there and they
take low single digits of CPU usage combined.

Depending on the application, usage, hardware, cost of software
development, cost of software upkeep, simplicity in system management
-- the answer to what is the best way is different.  I often find
people ignoring that simple concept and developing very complex
software to be theoretically faster ... only to end up with complex
and buggy code that is hard to manage in an environment where the
extra performance was not needed.  One has to also consider the cost
to develop and manage.

So there is no right or wrong answer, I am trying to get the
programmer to think ... does he really need, in his case, blistering
performance?  Can he do it with a simple inetd module (which later
could be the core for his own server)?  Does he want it up quick and
easy with no real management issues?

I am only spurring thought, not telling anyone what is right or wrong
in their case :-)

E

At 10:10 AM 5/11/2011, you wrote:

>Eric, you must be really kidding this time :), servers with this
>architecture are susceptible to dos and what not..am sure for
>embedded systems where memory is a big limiting factor the best
>would be async design, also code becomes easily portable in future.
>
>On Wed, May 11, 2011 at 10:39 AM, Eric S. Eberhard
><<mailto:[hidden email]>[hidden email]> wrote:
>I have found that fork() on modern machines as a negligible affect
>on performance and in fact I almost always use inetd instead of
>writing my own servers, mainly because it is dead reliable, easier
>to code, and again seems to have negligible affect on
>performance.  One would have to do millions upon millions of
>connects to notice or care.  Having said that, I use AIX mostly, and
>that performs better under load than Linux on Intel, and even Linux
>on the IBM p series platform.  I would do it cheap and easy and
>worry about performance after-the-fact. Eric
>
>At 04:46 PM 5/10/2011, you wrote:
>
>On 10 May 2011, at 4:13 PM, David Schwartz wrote:
> > On 5/10/2011 2:10 AM, John Hollingum wrote:
> >> Pretty much immediately after the accept the program forks a handler,
> >> but the rogue clients must be glomming onto the main process before the
> >> SSL negotiation is complete.
> >
> > Calling 'fork' with an accepted SSL connection has all kinds of
> known issues. The fundamental problem is that there are many
> operations that must occur both before and after the 'fork', for
> different reasons, and obviously can't do both.
>
>You could accept just the TCP connection in the main process and do
>all of the SSL handshake in the forked process (I think
>IO::Socket::SSL->start_SSL() is what you want for that) --- this
>would not be a high-performance approach (no SSL session cache, fork
>overhead) but if it's fast enough it's fast enough.
>
>It's possible to use openssl in a non-blocking, event-driven manner
>but I don't think Perl's SSL modules expose enough of the openssl
>API to do that.
>
>
>______________________________________________________________________
>OpenSSL
>Project
><http://www.openssl.org>http://www.openssl.org
>User Support Mailing
>List
><mailto:[hidden email]>[hidden email]
>Automated List
>Manager
><mailto:[hidden email]>[hidden email]
>
>
>
>Eric S. Eberhard
><tel:%28928%29%20567-3727>(928) 567-3727          Voice
><tel:%28928%29%20567-6122>(928) 567-6122          Fax
><tel:%28928%29%20301-7537>(928) 301-7537                           Cell
>
>Vertical Integrated Computer Systems, LLC
>Metropolis Support, LLC
>
>For Metropolis support and VICS MBA
>Support!!!!    <http://www.vicsmba.com>http://www.vicsmba.com
>
>Pictures of Snake in Spring
>
><http://www.facebook.com/album.php?aid=115547&id=1409661701&l=1c375e1f49>http://www.facebook.com/album.php?aid=115547&id=1409661701&l=1c375e1f49
>
>Pictures of Camp Verde
>
><http://www.facebook.com/album.php?aid=12771&id=1409661701&l=fc0e0a2bcf>http://www.facebook.com/album.php?aid=12771&id=1409661701&l=fc0e0a2bcf
>
>Pictures of Land Cruiser in Sedona
>
><http://www.facebook.com/album.php?aid=50953&id=1409661701>http://www.facebook.com/album.php?aid=50953&id=1409661701
>
>Pictures of Flagstaff area near our cabin
>
><http://www.facebook.com/album.php?aid=12750&id=1409661701>http://www.facebook.com/album.php?aid=12750&id=1409661701
>
>Pictures of Cheryl in a Horse Show
>
><http://www.facebook.com/album.php?aid=32484&id=1409661701>http://www.facebook.com/album.php?aid=32484&id=1409661701
>
>
>Pictures of the AZ Desert
>
><http://www.facebook.com/album.php?aid=58827&id=1409661701>http://www.facebook.com/album.php?aid=58827&id=1409661701
>
>(You can see why we love this state :-) )
>
>
>
>
>
>
>
>
>______________________________________________________________________
>OpenSSL
>Project
><http://www.openssl.org>http://www.openssl.org
>User Support Mailing
>List
><mailto:[hidden email]>[hidden email]
>Automated List
>Manager
><mailto:[hidden email]>[hidden email]
>


Eric S. Eberhard
(928) 567-3727          Voice
(928) 567-6122          Fax
(928) 301-7537                           Cell

Vertical Integrated Computer Systems, LLC
Metropolis Support, LLC

For Metropolis support and VICS MBA Support!!!!    http://www.vicsmba.com

Pictures of Snake in Spring

http://www.facebook.com/album.php?aid=115547&id=1409661701&l=1c375e1f49

Pictures of Camp Verde

http://www.facebook.com/album.php?aid=12771&id=1409661701&l=fc0e0a2bcf

Pictures of Land Cruiser in Sedona

http://www.facebook.com/album.php?aid=50953&id=1409661701

Pictures of Flagstaff area near our cabin

http://www.facebook.com/album.php?aid=12750&id=1409661701

Pictures of Cheryl in a Horse Show

http://www.facebook.com/album.php?aid=32484&id=1409661701


Pictures of the AZ Desert

http://www.facebook.com/album.php?aid=58827&id=1409661701

(You can see why we love this state :-) )








______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]