BIO_get_accept_socket weirdness

[Kurt Roeckx]

Hi,

I've been working on IPv6 support, and one of the strangest things
I find is BIO_get_accept_socket().

If bind_mode == BIO_BIND_REUSEADDR_IF_UNUSED, and bind() fails
with EADDRINUSE it creates a new socket and tries to connect
to the port it tried to bind() to, and if that fails tries to
bind to it again but now with setting SO_REUSEADDR.

Does anybody have an idea why it's trying to do that, and why we
shouldn't just do SO_REUSEADDR the first time?  Was there some
OS that maybe did strange things when trying to use SO_REUSEADDR
and it was already in use?

The documentation also says:
| BIO_set_bind_mode() and BIO_get_bind_mode() set and retrieve
| the current bind mode. If BIO_BIND_NORMAL (the default) is set
| then another socket cannot be bound to the same port. If
| BIO_BIND_REUSEADDR is set then other sockets can bind to the
| same port. If BIO_BIND_REUSEADDR_IF_UNUSED is set then and
| attempt is first made to use BIO_BIN_NORMAL, if this fails
| and the port is not in use then a second attempt is made
| using BIO_BIND_REUSEADDR.

The documentation is at least confusing, since it's about the
address and port combination and not just the port.  The main
reason for using SO_REUSEADDR is that you can bind to an
address / port combination that is in TIME_WAIT state, and it's
recommended for server applications to always set this, and it's
what I ended up doing in my new function.  There are also some
other cases you might want to use SO_REUSEADDR, and I suggest you
read Stevens for that.

In no case should it be possible to bind to the same TCP address /
port from different applications, SO_REUSEADDR does not allow you
to do that.

But then I found some MSDN documentation that says that Windows
allows others to hijack your socket when you've set SO_REUSEADDR
and the results are non-deterministic.  They also created an
SO_EXCLUSIVEADDRUSE and I'm getting confused what it really does,
but they say that server applications should set it.

Anyway, is that whole logic with the connect() needed, or can I
just always use SO_REUSEADDR (on non-windows hosts)?


Kurt

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]

[Eric Covener]

On Sat, Jul 5, 2014 at 7:37 AM, Kurt Roeckx <[hidden email]> wrote:
> Does anybody have an idea why it's trying to do that, and why we
> shouldn't just do SO_REUSEADDR the first time?  Was there some
> OS that maybe did strange things when trying to use SO_REUSEADDR
> and it was already in use?

FWLIW: I've seen this pattern in some other proprietary software,
where they try hard to not set SO_REUSEADDR unless it appears needed
due to a bind failure. But whatever they were working around, it is
detrimental to modern Linux where the outgoing TIME_WAIT socket has to
also have been opened with SO_REUSEADDR for the reuse to be allowed.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]

[Kurt Roeckx]

On Sat, Jul 05, 2014 at 08:13:04AM -0400, Eric Covener wrote:

> On Sat, Jul 5, 2014 at 7:37 AM, Kurt Roeckx <[hidden email]> wrote:
> > Does anybody have an idea why it's trying to do that, and why we
> > shouldn't just do SO_REUSEADDR the first time?  Was there some
> > OS that maybe did strange things when trying to use SO_REUSEADDR
> > and it was already in use?
>
> FWLIW: I've seen this pattern in some other proprietary software,
> where they try hard to not set SO_REUSEADDR unless it appears needed
> due to a bind failure. But whatever they were working around, it is
> detrimental to modern Linux where the outgoing TIME_WAIT socket has to
> also have been opened with SO_REUSEADDR for the reuse to be allowed.

man socket(7) documents that behavior on Linux.  They also say that
you ussually don't notice it because most things always set
SO_REUSEADDR.  So it seems to me like the behavior of that piece
of code will at least don't do what you want it to do on Linux so
other than being weird it looks like an other reason to just drop
it.


Kurt

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]

[Tim Hudson]

On 5/07/2014 9:12 AM, Kurt Roeckx wrote:

> On Sat, Jul 05, 2014 at 08:13:04AM -0400, Eric Covener wrote:
>> On Sat, Jul 5, 2014 at 7:37 AM, Kurt Roeckx <[hidden email]> wrote:
>>> Does anybody have an idea why it's trying to do that, and why we
>>> shouldn't just do SO_REUSEADDR the first time?  Was there some
>>> OS that maybe did strange things when trying to use SO_REUSEADDR
>>> and it was already in use?
>> FWLIW: I've seen this pattern in some other proprietary software,
>> where they try hard to not set SO_REUSEADDR unless it appears needed
>> due to a bind failure. But whatever they were working around, it is
>> detrimental to modern Linux where the outgoing TIME_WAIT socket has to
>> also have been opened with SO_REUSEADDR for the reuse to be allowed.
> man socket(7) documents that behavior on Linux.  They also say that
> you ussually don't notice it because most things always set
> SO_REUSEADDR.  So it seems to me like the behavior of that piece
> of code will at least don't do what you want it to do on Linux so
> other than being weird it looks like an other reason to just drop
> it.

If you have SO_REUSEADDR set and a listener already in place you will
start a new listener and who gets the incoming connections is not well
defined.
You need to determine if this is just a time-wait socket stopping the
bind or there is a listener in place. That is the semantics being
handled in the code.

See crypto/bio/bss_acpt.c where the bind_mode is "documented" in a comment:

        /* If 0, it means normal, if 1, do a connect on bind failure,
         * and if there is no-one listening, bind with SO_REUSEADDR.
         * If 2, always use SO_REUSEADDR. */


Setting SO_REUSEADDR on always will not result in what you are expecting ...

Tim.

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]

[Kurt Roeckx]

On Sat, Jul 05, 2014 at 12:45:37PM -0400, Tim Hudson wrote:

> On 5/07/2014 9:12 AM, Kurt Roeckx wrote:
> > On Sat, Jul 05, 2014 at 08:13:04AM -0400, Eric Covener wrote:
> >> On Sat, Jul 5, 2014 at 7:37 AM, Kurt Roeckx <[hidden email]> wrote:
> >>> Does anybody have an idea why it's trying to do that, and why we
> >>> shouldn't just do SO_REUSEADDR the first time?  Was there some
> >>> OS that maybe did strange things when trying to use SO_REUSEADDR
> >>> and it was already in use?
> >> FWLIW: I've seen this pattern in some other proprietary software,
> >> where they try hard to not set SO_REUSEADDR unless it appears needed
> >> due to a bind failure. But whatever they were working around, it is
> >> detrimental to modern Linux where the outgoing TIME_WAIT socket has to
> >> also have been opened with SO_REUSEADDR for the reuse to be allowed.
> > man socket(7) documents that behavior on Linux.  They also say that
> > you ussually don't notice it because most things always set
> > SO_REUSEADDR.  So it seems to me like the behavior of that piece
> > of code will at least don't do what you want it to do on Linux so
> > other than being weird it looks like an other reason to just drop
> > it.
>
> If you have SO_REUSEADDR set and a listener already in place you will
> start a new listener

No you won't.  You will get a bind() error:
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
bind(3, {sa_family=AF_INET, sin_port=htons(3344), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EADDRINUSE (Address already in use)

Except on windows it seems.


Kurt

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]

[Tim Hudson]

On 5/07/2014 2:14 PM, Kurt Roeckx wrote:
> On Sat, Jul 05, 2014 at 12:45:37PM -0400, Tim Hudson wrote:
>> If you have SO_REUSEADDR set and a listener already in place you will
>> start a new listener
> No you won't.  You will get a bind() error:
> socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
> setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
> bind(3, {sa_family=AF_INET, sin_port=htons(3344), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EADDRINUSE (Address already in use)
>
> Except on windows it seems.

All the world is not (yet) Linux :-) ... and those semantics were
defined log ago - and evolved ... and there is also REUSEPORT (added
later) and a variety of interpretations - but the base REUSEADDR can
indeed behave that way depending on what platform you are on. Linux has
its own slightly different interpretation. Some google engineering
(search) will show the the variety of confusion that this causes in
cross-platform code.

Tim

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]

[Salz, Rich]

Those who forget history are doomed to re-implement it, wrongly.

SO_REUSEADDR was implemented in 4.2BSD so that a server could restart without waiting for the various FIN_WAIT timeouts to happen.

:)

        /r$

--  
Principal Security Engineer
Akamai Technologies, Cambridge, MA
IM: [hidden email]; Twitter: RichSalz

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]

[Kurt Roeckx]

On Sat, Jul 05, 2014 at 02:37:49PM -0400, Tim Hudson wrote:

> On 5/07/2014 2:14 PM, Kurt Roeckx wrote:
> > On Sat, Jul 05, 2014 at 12:45:37PM -0400, Tim Hudson wrote:
> >> If you have SO_REUSEADDR set and a listener already in place you will
> >> start a new listener
> > No you won't.  You will get a bind() error:
> > socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
> > setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
> > bind(3, {sa_family=AF_INET, sin_port=htons(3344), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EADDRINUSE (Address already in use)
> >
> > Except on windows it seems.
>
> All the world is not (yet) Linux :-) ... and those semantics were
> defined log ago - and evolved ... and there is also REUSEPORT (added
> later) and a variety of interpretations - but the base REUSEADDR can
> indeed behave that way depending on what platform you are on. Linux has
> its own slightly different interpretation. Some google engineering
> (search) will show the the variety of confusion that this causes in
> cross-platform code.

Are you saying BSD has different behavior than Linux, other than
Linux requiring that it's also set on the old socket?


Kurt

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]

[Jeff Trawick]

On Sat, Jul 5, 2014 at 2:37 PM, Tim Hudson <[hidden email]> wrote:

On 5/07/2014 2:14 PM, Kurt Roeckx wrote:
> On Sat, Jul 05, 2014 at 12:45:37PM -0400, Tim Hudson wrote:

>> If you have SO_REUSEADDR set and a listener already in place you will
>> start a new listener
> No you won't.  You will get a bind() error:
> socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
> setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
> bind(3, {sa_family=AF_INET, sin_port=htons(3344), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EADDRINUSE (Address already in use)
>
> Except on windows it seems.

All the world is not (yet) Linux :-) ... and those semantics were
defined log ago - and evolved ... and there is also REUSEPORT (added
later) and a variety of interpretations - but the base REUSEADDR can
indeed behave that way depending on what platform you are on. Linux has
its own slightly different interpretation. Some google engineering
(search) will show the the variety of confusion that this causes in
cross-platform code.

FWIW, Windows is the only platform for which Apache httpd does not enable SO_REUSEADDR prior to bind.  That itself doesn't make it right, but there has been plenty of time to complain.  And admins notice things like multiple instances on the same port and confusion about which instance handles which connection.

As for Windows, SO_EXCLUSIVEADDRUSE looks interesting:

http://msdn.microsoft.com/en-us/library/windows/desktop/ms740621(v=vs.85).aspx

Incidentally, I don't readily see a Windows bind-while-TIME_WAIT discussion.  And I see that httpd enables SO_REUSEADDR *after* the bind on Windows.  The commit message for that doesn't lead me to believe that there was definite rhyme or reason, but it was 12 years ago so it probably doesn't hurt.  I *guess* the TIME_WAIT issue doesn't hinder.

 

Tim

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]

--

Born in Roswell... married an alien...
http://emptyhammock.com/

http://edjective.org/

[Tim Hudson]

> Some google engineering (search) will show the the variety of confusion that this causes in cross-platform code.

Start here for some interesting reading - http://stackoverflow.com/questions/14388706/socket-options-so-reuseaddr-and-so-reuseport-how-do-they-differ-do-they-mean-t
You will find many exchanges of details about that.

And yes - originally this was (mostly) for being able to start a server again after stopping it when sockets were remaining in TIME_WAIT state - but it wasn't implemented as just that and the semantics have varied.

Tim.

[Kurt Roeckx]

On Sat, Jul 05, 2014 at 02:55:36PM -0400, Tim Hudson wrote:
> > Some google engineering (search) will show the the variety of
> confusion that this causes in cross-platform code.
>
> Start here for some interesting reading -
> http://stackoverflow.com/questions/14388706/socket-options-so-reuseaddr-and-so-reuseport-how-do-they-differ-do-they-mean-t
> You will find *many* exchanges of details about that.

It doesn't tell me anything not mentioned in Stevens, except that
Windows thing that I already found earlier.

So I still fail to see why we shouldn't just set SO_REUSEADDR
except on Windows.


Kurt

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]

[Ben Laurie-2]

On 5 July 2014 12:37, Kurt Roeckx <[hidden email]> wrote:
> But then I found some MSDN documentation that says that Windows
> allows others to hijack your socket when you've set SO_REUSEADDR
> and the results are non-deterministic.  They also created an
> SO_EXCLUSIVEADDRUSE and I'm getting confused what it really does,
> but they say that server applications should set it.

If you think that opening the socket first is a security measure, then
you've got some pretty serious problems.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]