Receive throttling on SSL sockets

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Receive throttling on SSL sockets

Alex Hultman
How do you properly implement receive throttling on SSL sockets without hindering writing?

As opposed to raw TCP sockets, an SSL socket cannot be receive-throttled simply by stop polling for readable events on the underlying raw TCP socket. SSL_write still could require reading of data so simply stop polling for readable would potentially hinder writing of data which is not okay.

Is there any such receive-throttling functionality in the SSL protocol itself? I don't see how SSL_peek would solve the issue since I would still be buffering (potentially uncontrolled amount of) data in a BIO.

Even if I would _only_ enable readable polling when _absolutely needed_ as per SSL_write error, I still cannot guarantee not reading a chunk of data (which I would then need to buffer up in a BIO since the application is not expecting it).

How are we supposed to solve this issue without potentially building up backpressure?

Thanks

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: Receive throttling on SSL sockets

OpenSSL - User mailing list

TLS is a bidirectional protocol.  You can’t throttle only one side.

 

From: Alex H <[hidden email]>
Reply-To: openssl-users <[hidden email]>
Date: Friday, May 18, 2018 at 7:21 PM
To: openssl-users <[hidden email]>
Subject: [openssl-users] Receive throttling on SSL sockets

 

How do you properly implement receive throttling on SSL sockets without hindering writing?

 

As opposed to raw TCP sockets, an SSL socket cannot be receive-throttled simply by stop polling for readable events on the underlying raw TCP socket. SSL_write still could require reading of data so simply stop polling for readable would potentially hinder writing of data which is not okay.

 

Is there any such receive-throttling functionality in the SSL protocol itself? I don't see how SSL_peek would solve the issue since I would still be buffering (potentially uncontrolled amount of) data in a BIO.

 

Even if I would _only_ enable readable polling when _absolutely needed_ as per SSL_write error, I still cannot guarantee not reading a chunk of data (which I would then need to buffer up in a BIO since the application is not expecting it).

 

How are we supposed to solve this issue without potentially building up backpressure?

 

Thanks


--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: Receive throttling on SSL sockets

Alex Hultman
Okay that's a good theoretical answer but practically not very useful.

I know for instance Node.js to implement their Streams interface with both TCP and SSL sockets. They both have pause / resume functions for receive-throttling and I've tested it with SSL and it seems to work somehow.

One solution (I guess?) would be to stop polling for readable until SSL_write demands data then immediately stop polling for readable again once SSL_write is happy. In the case of getting unwanted data while throttling then SSL_peek can be used instead of SSL_read. That would not guarantee no buildup but would work for the most part, right?

Do you see any flaw with this? Could it still fail due to mass buildup when throttling for long?

Den lör 19 maj 2018 04:57Salz, Rich via openssl-users <[hidden email]> skrev:

TLS is a bidirectional protocol.  You can’t throttle only one side.

 

From: Alex H <[hidden email]>
Reply-To: openssl-users <[hidden email]>
Date: Friday, May 18, 2018 at 7:21 PM
To: openssl-users <[hidden email]>
Subject: [openssl-users] Receive throttling on SSL sockets

 

How do you properly implement receive throttling on SSL sockets without hindering writing?

 

As opposed to raw TCP sockets, an SSL socket cannot be receive-throttled simply by stop polling for readable events on the underlying raw TCP socket. SSL_write still could require reading of data so simply stop polling for readable would potentially hinder writing of data which is not okay.

 

Is there any such receive-throttling functionality in the SSL protocol itself? I don't see how SSL_peek would solve the issue since I would still be buffering (potentially uncontrolled amount of) data in a BIO.

 

Even if I would _only_ enable readable polling when _absolutely needed_ as per SSL_write error, I still cannot guarantee not reading a chunk of data (which I would then need to buffer up in a BIO since the application is not expecting it).

 

How are we supposed to solve this issue without potentially building up backpressure?

 

Thanks

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: Receive throttling on SSL sockets

Alex Hultman
I think the best solution would be to simply state in the documentation of my implementation that "throttling receives on SSL sockets will significantly reduce data receive but will not guarantee a total halt".

Agree?

What do you think of the way Node.js handles this? They _must_ be using some kind of internal backup queue for cases like these, right?

2018-05-19 11:02 GMT+02:00 Alex H <[hidden email]>:
Okay that's a good theoretical answer but practically not very useful.

I know for instance Node.js to implement their Streams interface with both TCP and SSL sockets. They both have pause / resume functions for receive-throttling and I've tested it with SSL and it seems to work somehow.

One solution (I guess?) would be to stop polling for readable until SSL_write demands data then immediately stop polling for readable again once SSL_write is happy. In the case of getting unwanted data while throttling then SSL_peek can be used instead of SSL_read. That would not guarantee no buildup but would work for the most part, right?

Do you see any flaw with this? Could it still fail due to mass buildup when throttling for long?

Den lör 19 maj 2018 04:57Salz, Rich via openssl-users <[hidden email]> skrev:

TLS is a bidirectional protocol.  You can’t throttle only one side.

 

From: Alex H <[hidden email]>
Reply-To: openssl-users <[hidden email]>
Date: Friday, May 18, 2018 at 7:21 PM
To: openssl-users <[hidden email]>
Subject: [openssl-users] Receive throttling on SSL sockets

 

How do you properly implement receive throttling on SSL sockets without hindering writing?

 

As opposed to raw TCP sockets, an SSL socket cannot be receive-throttled simply by stop polling for readable events on the underlying raw TCP socket. SSL_write still could require reading of data so simply stop polling for readable would potentially hinder writing of data which is not okay.

 

Is there any such receive-throttling functionality in the SSL protocol itself? I don't see how SSL_peek would solve the issue since I would still be buffering (potentially uncontrolled amount of) data in a BIO.

 

Even if I would _only_ enable readable polling when _absolutely needed_ as per SSL_write error, I still cannot guarantee not reading a chunk of data (which I would then need to buffer up in a BIO since the application is not expecting it).

 

How are we supposed to solve this issue without potentially building up backpressure?

 

Thanks

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users


--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: Receive throttling on SSL sockets

OpenSSL - User mailing list
In reply to this post by Alex Hultman

There are TLS control messages which could flow in either direction, spontaneously.  Renegotiation (pre TLS 1.3), tickets (TLS 1.3), and so on.

 

I cannot comment on if your proposal would work or not, sorry.

 

From: Alex H <[hidden email]>
Date: Saturday, May 19, 2018 at 5:03 AM
To: Rich Salz <[hidden email]>, openssl-users <[hidden email]>
Subject: Re: [openssl-users] Receive throttling on SSL sockets

 

Okay that's a good theoretical answer but practically not very useful.

 

I know for instance Node.js to implement their Streams interface with both TCP and SSL sockets. They both have pause / resume functions for receive-throttling and I've tested it with SSL and it seems to work somehow.

 

One solution (I guess?) would be to stop polling for readable until SSL_write demands data then immediately stop polling for readable again once SSL_write is happy. In the case of getting unwanted data while throttling then SSL_peek can be used instead of SSL_read. That would not guarantee no buildup but would work for the most part, right?

 

Do you see any flaw with this? Could it still fail due to mass buildup when throttling for long?

 

Den lör 19 maj 2018 04:57Salz, Rich via openssl-users <[hidden email]> skrev:

TLS is a bidirectional protocol.  You can’t throttle only one side.

 

From: Alex H <[hidden email]>
Reply-To: openssl-users <[hidden email]>
Date: Friday, May 18, 2018 at 7:21 PM
To: openssl-users <[hidden email]>
Subject: [openssl-users] Receive throttling on SSL sockets

 

How do you properly implement receive throttling on SSL sockets without hindering writing?

 

As opposed to raw TCP sockets, an SSL socket cannot be receive-throttled simply by stop polling for readable events on the underlying raw TCP socket. SSL_write still could require reading of data so simply stop polling for readable would potentially hinder writing of data which is not okay.

 

Is there any such receive-throttling functionality in the SSL protocol itself? I don't see how SSL_peek would solve the issue since I would still be buffering (potentially uncontrolled amount of) data in a BIO.

 

Even if I would _only_ enable readable polling when _absolutely needed_ as per SSL_write error, I still cannot guarantee not reading a chunk of data (which I would then need to buffer up in a BIO since the application is not expecting it).

 

How are we supposed to solve this issue without potentially building up backpressure?

 

Thanks

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users


--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: Receive throttling on SSL sockets

Michael Wojcik
> From: openssl-users [mailto:[hidden email]] On Behalf Of Salz, Rich via openssl-users
> Sent: Saturday, May 19, 2018 08:48
> To: Alex H; [hidden email]
> Subject: Re: [openssl-users] Receive throttling on SSL sockets

> There are TLS control messages which could flow in either direction, spontaneously.  Renegotiation (pre TLS 1.3),
> tickets (TLS 1.3), and so on.

Right. And TCP is an ordered byte-stream protocol. That means to receive a control message from the peer, the local stack *must* have received everything transmitted prior to it. (Modulo SACK, but SACK'd data preceeded by a gap is invisible to the application, so we should ignore it.)

And if that data fills the receive window so the peer can't send the control message, then the application *must* receive the data from the local stack. Peeking won't help in this case, because the control message is still stuck on the peer, waiting for the window to open. (Of course this is one reason why FTP used a separate control connection, for example - so that control messages didn't sit behind a lot of application data. That led to a number of other difficulties, so it wasn't a widely used design.)

In short:  When you get SSL_WANT_READ, you have to receive and buffer data. You can try peeking, but it's not guaranteed to be able to get far enough to find the control message, and SSL_peek is just buffering data within OpenSSL, so you're not doing any throttling that way.

This will be true of any TCP-based TLS implementation. It's not an OpenSSL implementation, and whatever else Node.js is doing, it must be buffering TLS traffic somewhere when it has to receive a control message.

Post-handshake control message flows don't happen that frequently; relatively short-lived conversations may never see one (until the final close_notify alert). So throttling may often work. But in the general case, sooner or later you'll have to buffer at the application level.

Michael Wojcik
Distinguished Engineer, Micro Focus


--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: Receive throttling on SSL sockets

JordanBrown
On 5/19/2018 6:51 AM, Michael Wojcik wrote:
Right. And TCP is an ordered byte-stream protocol. That means to receive a control message from the peer, the local stack *must* have received everything transmitted prior to it. (Modulo SACK, but SACK'd data preceeded by a gap is invisible to the application, so we should ignore it.)

And yet TCP itself moves ACKs when there's no window available.

TLS could (but as far as I can tell does not) have such a mechanism.  It could have a window, like TCP, where the receiver would say "you can send me 64K of data", and the sender wouldn't be allowed to send data (but could send control messages) when that window is exhausted, until the receiver reopens the window.  It could have control messages like XON and XOFF that say "please stop sending me data (but control is OK)" and "resume sending data".

Each scheme has its problems (mostly around how much data can be in flight at any one time), but they're both clearly possible.

It does seem like some sort of flow control would be desirable, so that the receiver doesn't have to have some way to handle arbitrarily large amounts of data to keep the connection healthy.

Maybe in TLS 1.4.

-- 
Jordan Brown, Oracle Solaris

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: Receive throttling on SSL sockets

Alex Hultman
Yeah TCP is really the same as TLS in terms of being "bidirectional". Even if you stop polling for readable and never call recv, you will still receive ACKS for whatever you write.

A receive window for TLS implemented completely ontop of TCP would solve this issue and allow applications to truly throttle data receives without hindering their own writes or forcing them to buffer up data.

2018-05-19 20:07 GMT+02:00 Jordan Brown <[hidden email]>:
On 5/19/2018 6:51 AM, Michael Wojcik wrote:
Right. And TCP is an ordered byte-stream protocol. That means to receive a control message from the peer, the local stack *must* have received everything transmitted prior to it. (Modulo SACK, but SACK'd data preceeded by a gap is invisible to the application, so we should ignore it.)

And yet TCP itself moves ACKs when there's no window available.

TLS could (but as far as I can tell does not) have such a mechanism.  It could have a window, like TCP, where the receiver would say "you can send me 64K of data", and the sender wouldn't be allowed to send data (but could send control messages) when that window is exhausted, until the receiver reopens the window.  It could have control messages like XON and XOFF that say "please stop sending me data (but control is OK)" and "resume sending data".

Each scheme has its problems (mostly around how much data can be in flight at any one time), but they're both clearly possible.

It does seem like some sort of flow control would be desirable, so that the receiver doesn't have to have some way to handle arbitrarily large amounts of data to keep the connection healthy.

Maybe in TLS 1.4.

-- 
Jordan Brown, Oracle Solaris


--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: Receive throttling on SSL sockets

Michael Wojcik
In reply to this post by JordanBrown
> From: Jordan Brown [mailto:[hidden email]]
> Sent: Saturday, May 19, 2018 14:08
> To: [hidden email]; Michael Wojcik; Alex H
> Subject: Re: [openssl-users] Receive throttling on SSL sockets

> TLS could (but as far as I can tell does not) have such a mechanism.  It could have a window, like TCP, where the receiver
> would say "you can send me 64K of data", and the sender wouldn't be allowed to send data (but could send control
> messages) when that window is exhausted, until the receiver reopens the window.  It could have control messages like
> XON and XOFF that say "please stop sending me data (but control is OK)" and "resume sending data".

Hey, if we're all bored with reinventing TCP on top of UDP, we can reinvent TCP on top of TCP!

> It does seem like some sort of flow control would be desirable, so that the receiver doesn't have to have some way to
> handle arbitrarily large amounts of data to keep the connection healthy.
> Maybe in TLS 1.4.

Good lord, isn't TLS complicated enough already? How many pages is the new edition of /Bulletproof TLS/? (I don't know because I have it in Kindle form. But it's long. Loooooong.)

Flow control really, really, *really* seems like an application-layer task to me in the case of TLS. I think adding it to TLS itself would be a mistake.

Michael Wojcik
Distinguished Engineer, Micro Focus



--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: Receive throttling on SSL sockets

Alex Hultman
Flow control really, really, *really* seems like an application-layer task to me in the case of TLS. I think adding it to TLS itself would be a mistake.

This whole thread of messages kind of already concluded that this is not possible currently. You simply cannot implement proper flow control since doing so would potentially throttle writes, not just reads. You need a TLS data window to do it properly.

2018-05-19 21:42 GMT+02:00 Michael Wojcik <[hidden email]>:
> From: Jordan Brown [mailto:[hidden email]]
> Sent: Saturday, May 19, 2018 14:08
> To: [hidden email]; Michael Wojcik; Alex H
> Subject: Re: [openssl-users] Receive throttling on SSL sockets

> TLS could (but as far as I can tell does not) have such a mechanism.  It could have a window, like TCP, where the receiver
> would say "you can send me 64K of data", and the sender wouldn't be allowed to send data (but could send control
> messages) when that window is exhausted, until the receiver reopens the window.  It could have control messages like
> XON and XOFF that say "please stop sending me data (but control is OK)" and "resume sending data".

Hey, if we're all bored with reinventing TCP on top of UDP, we can reinvent TCP on top of TCP!

> It does seem like some sort of flow control would be desirable, so that the receiver doesn't have to have some way to
> handle arbitrarily large amounts of data to keep the connection healthy.
> Maybe in TLS 1.4.

Good lord, isn't TLS complicated enough already? How many pages is the new edition of /Bulletproof TLS/? (I don't know because I have it in Kindle form. But it's long. Loooooong.)

Flow control really, really, *really* seems like an application-layer task to me in the case of TLS. I think adding it to TLS itself would be a mistake.

Michael Wojcik
Distinguished Engineer, Micro Focus



--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users


--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: Receive throttling on SSL sockets

JordanBrown
In reply to this post by Michael Wojcik
I should make it clear that I don't have a stake here.  Lack of flow control hasn't caused me problems personally, and I'm not responsible for implementing and maintaining a TLS infrastructure.  This is purely an intellectual exercise for me.

There were comments suggesting that, because TLS is an ordered-byte-stream protocol that needs control messages in both directions at all times, TLS couldn't support flow control.  That seems clearly wrong; it clearly could.  (As you say, we could just layer TCP on top of it.)

Should it?  My mild feeling is "yes", since it's already got a record and control message structure and so it wouldn't be necessary to invent another protocol on top of it.  Yes, that makes TLS more complicated, but would it be any more complicated than an additional application-visible layer would be?  It seems like the answer is that any complexity from a TLS-layer implementation would be primarily in the TLS implementation, whereas an additional layer would necessarily impose complexity on the application, over and above the complexity of the flow control implementation itself.
-- 
Jordan Brown, Oracle Solaris

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Reply | Threaded
Open this post in threaded view
|

Re: Receive throttling on SSL sockets

Michael Wojcik
In reply to this post by Alex Hultman
> From: openssl-users [mailto:[hidden email]] On Behalf Of Alex H
> Sent: Saturday, May 19, 2018 15:53
> To: [hidden email]
> Subject: Re: [openssl-users] Receive throttling on SSL sockets

> >  Flow control really, really, *really* seems like an application-layer task to me in the case of TLS. I think adding it to TLS
> > itself would be a mistake.

> This whole thread of messages kind of already concluded that this is not possible currently.

I don't believe it did. It concluded that the server can't impose flow control by itself.

> You simply cannot implement proper flow control since doing so would potentially throttle writes, not just reads. You
> need a TLS data window to do it properly.

If the client and server both participate in flow control - that is, if they implement the window announcements and output throttling at the application level - then there's no need to do it in TLS.

A cooperating client and server can implement any sort of traffic shaping they want.

What's not possible in TLS is throttling implemented solely by one side, without cooperation from the peer application.


In any case, this has drifted far afield from the purpose of openssl-users. I pesonally don't think flow control should be part of TLS, but I don't care strongly enough to, for example, argue against it on the IETF TLS mailing list.

Michael Wojcik
Distinguished Engineer, Micro Focus



--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users