Saving (and restoring) cipher context

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Saving (and restoring) cipher context

Jim Wong
Saving (and restoring) cipher context

Folks,

I apologize if this is a somewhat naïve or misinformed question, as I'm new
to the OpenSSL APIs and not quite sure how things work yet.

I'm developing an application in which we're using AES to encrypt files as
they're transferred from another system and saved to disk.  We'd like to
provide the ability for the application to resume a transfer that was
interrupted mid-stream, but the encryption throws a bit of a wrench into
things because of the state associated with the encryption context.

Is there a safe, supported way to stash the context somewhere on disk so
that encryption can be resumed where it left off when the file transfer
starts up again?  We're currently looking at the EVP functions; would we
have to drop down to the lower-level, algorithm-specific routines to do this
right?

Thanks in advance for any guidance you can provide.

--
Jim Wong ([hidden email])

Reply | Threaded
Open this post in threaded view
|

RE: Saving (and restoring) cipher context

JoelKatz

> I'm developing an application in which we're using AES
> to encrypt files as they're transferred from another system
> and saved to disk.  We'd like to provide the ability for the
> application to resume a transfer that was interrupted mid-stream,
> but the encryption throws a bit of a wrench into things because of
> the state associated with the encryption context.

> Is there a safe, supported way to stash the context somewhere on
> disk so that encryption can be resumed where it left off when the
> file transfer starts up again?  We're currently looking at the EVP
> functions; would we have to drop down to the lower-level,
> algorithm-specific routines to do this right?

Maybe you're locked into an implementation that isn't logical for your
problem set, but if not, change to a rational implementation.

If you are encrypting/decrypting during the transfer, that means the
encryption is to protect the transport. Since the resumption is over a new
transport, there is no rational reason to resume the previous encryption.
Use a new encryption context for the new transport. If the encryption is for
storage, then resume sending the already-encrypted data.

The only case where you even have the problem you are describing is where
the data has to be encrypted differently from its normal state for storage
on the receiver. (Perhaps to be decrypted later.) Unless you're in this very
unusual situation, this shouldn't be an issue.

If you really are in this very unusual situation, I recommend one of the
following solutions:

1) Have a resync protocol that runs on both ends, re-encrypting the data and
sending block checksums. Let the other side verify the checksum until you
have a mismatch, then continue. You don't need to store the context because
this process recovers it.

2) Encrypt the file as a group of separate blocks. Simply discard any
incomplete blocks. Each block can have its own encryption context, so there
is no need to resume.

3) Use an encryption scheme that has negligible context. For example, if you
XOR each block of data with its block number and then encrypt it, all you
need to pick back up where you left off is the block number, which you
should already know.

Give us more details and tell us what you can and can't change. Your
proposed solution may not be the best one to your actual outer problem.

DS


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Saving (and restoring) cipher context

Bear Giles
In reply to this post by Jim Wong
You may have already gotten this answer, but just in case you haven't...

You're looking at two distinct problems.  The first is protecting the data
"in motion", and you basically need to look at the SSL routines and
related authentication information (e.g., X.509 certificates).  The data
you put in the pipe is what you will get out on the other side.  It could
be plaintext or a previously encrypted file.

The other problem is protecting the data "at rest".  This is where your
AES encryption comes in, and all of the key management issues involved
with symmetrical keys.

HTH

Bear

> Folks,
>
> I apologize if this is a somewhat naïve or misinformed question, as I'm
> new
> to the OpenSSL APIs and not quite sure how things work yet.
>
> I'm developing an application in which we're using AES to encrypt files as
> they're transferred from another system and saved to disk.  We'd like to
> provide the ability for the application to resume a transfer that was
> interrupted mid-stream, but the encryption throws a bit of a wrench into
> things because of the state associated with the encryption context.
>
> Is there a safe, supported way to stash the context somewhere on disk so
> that encryption can be resumed where it left off when the file transfer
> starts up again?  We're currently looking at the EVP functions; would we
> have to drop down to the lower-level, algorithm-specific routines to do
> this
> right?
>
> Thanks in advance for any guidance you can provide.
>
> --
> Jim Wong ([hidden email])
>
>


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Saving (and restoring) cipher context

Jim Wong
In reply to this post by JoelKatz
Thanks for the response.  Sorry for the ambiguity in my original
message: I thought keeping things abstract would make what we're doing
easier to understand at a high level, but I probably left things too
vague.

The short version is that we're implementing a kind of remote file
service that clients can use to store and retrieve files.  One of the
features we're trying to provide is the ability to resume uploads: for
example, if a client is storing a large file over a spotty network
connection, we don't want to have to restart the process every time the
connection drops.

For the actual file transfer, we're thinking of using vanilla SSL, for
several reasons: a) we don't want to reinvent the wheel; b) we'd like to
avoid having to deploy our own encryption code on all our client
platforms; and c) clients also transmit metadata that isn't part of the
file itself (and is stored separately from the file) that we don't want
to send over the wire in the clear.

Thus, the encryption I'm talking about is purely for the purpose of
storing the file on the server.  The idea is that even if someone were
able to gain unauthorized remote access to the storage server, he
shouldn't be able to read the contents of any of the uploaded files.  I
think this is somewhat closer to the very unusual case than the other
scenarios you described, but could be mistaken.

Fortunately, we're not locked into any particular implementation
strategy, so if we're barking up the wrong tree, there's opportunity to
change.  For example, we've also considered:

        - Having the client do the encryption, which makes resuming
uploads trivial, but complicates the client implementation (and requires
that we pass the key around).

        - Doing block-by-block encryption, as you suggest, but this adds
management overhead associated with dealing with the blocks after the
upload is complete.

        - When an upload is resumed, read back, decrypt and re-encrypt
the data already stored to get the context back to its previous state.

        - etc...

I'd love to hear what you think.  Are there philosophical issues with
saving the context across failed uploads?  Looking at the low-level AES
routines, it actually looks like it would be pretty straightforward from
an implementation standpoint, but if it's the wrong thing to do...

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of David Schwartz
Sent: Wednesday, May 30, 2007 1:42 PM
To: [hidden email]
Subject: RE: Saving (and restoring) cipher context


> I'm developing an application in which we're using AES to encrypt
> files as they're transferred from another system and saved to disk.  
> We'd like to provide the ability for the application to resume a
> transfer that was interrupted mid-stream, but the encryption throws a
> bit of a wrench into things because of the state associated with the
> encryption context.

> Is there a safe, supported way to stash the context somewhere on disk
> so that encryption can be resumed where it left off when the file
> transfer starts up again?  We're currently looking at the EVP
> functions; would we have to drop down to the lower-level,
> algorithm-specific routines to do this right?

Maybe you're locked into an implementation that isn't logical for your
problem set, but if not, change to a rational implementation.

If you are encrypting/decrypting during the transfer, that means the
encryption is to protect the transport. Since the resumption is over a
new transport, there is no rational reason to resume the previous
encryption.
Use a new encryption context for the new transport. If the encryption is
for storage, then resume sending the already-encrypted data.

The only case where you even have the problem you are describing is
where the data has to be encrypted differently from its normal state for
storage on the receiver. (Perhaps to be decrypted later.) Unless you're
in this very unusual situation, this shouldn't be an issue.

If you really are in this very unusual situation, I recommend one of the
following solutions:

1) Have a resync protocol that runs on both ends, re-encrypting the data
and sending block checksums. Let the other side verify the checksum
until you have a mismatch, then continue. You don't need to store the
context because this process recovers it.

2) Encrypt the file as a group of separate blocks. Simply discard any
incomplete blocks. Each block can have its own encryption context, so
there is no need to resume.

3) Use an encryption scheme that has negligible context. For example, if
you XOR each block of data with its block number and then encrypt it,
all you need to pick back up where you left off is the block number,
which you should already know.

Give us more details and tell us what you can and can't change. Your
proposed solution may not be the best one to your actual outer problem.

DS


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Saving (and restoring) cipher context

Jim Wong
In reply to this post by Bear Giles
Thanks for the response.  As I mentioned in my more detailed followup, we're planning to use SSL to protect the file data (and metadata) while it's in flight, and a separate encryption scheme for the data "at rest".  Because the file upload might require several (or many) attempts, I was hoping to find a way to avoid a) either breaking the file up into separately encrypted chunks on the server or b) doing a lot of processing (decrypting, re-encrypting, etc.) to sync things back up.

Note that we're expecting upload interruptions to be not infrequent, and that a not insignificant amount of time may pass between upload attempts--hours or even days.

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of [hidden email]
Sent: Wednesday, May 30, 2007 3:30 PM
To: [hidden email]
Subject: Re: Saving (and restoring) cipher context

You may have already gotten this answer, but just in case you haven't...

You're looking at two distinct problems.  The first is protecting the data "in motion", and you basically need to look at the SSL routines and related authentication information (e.g., X.509 certificates).  The data you put in the pipe is what you will get out on the other side.  It could be plaintext or a previously encrypted file.

The other problem is protecting the data "at rest".  This is where your AES encryption comes in, and all of the key management issues involved with symmetrical keys.

HTH

Bear

> Folks,
>
> I apologize if this is a somewhat naïve or misinformed question, as
> I'm new to the OpenSSL APIs and not quite sure how things work yet.
>
> I'm developing an application in which we're using AES to encrypt
> files as they're transferred from another system and saved to disk.  
> We'd like to provide the ability for the application to resume a
> transfer that was interrupted mid-stream, but the encryption throws a
> bit of a wrench into things because of the state associated with the encryption context.
>
> Is there a safe, supported way to stash the context somewhere on disk
> so that encryption can be resumed where it left off when the file
> transfer starts up again?  We're currently looking at the EVP
> functions; would we have to drop down to the lower-level,
> algorithm-specific routines to do this right?
>
> Thanks in advance for any guidance you can provide.
>
> --
> Jim Wong ([hidden email])
>
>


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Saving (and restoring) cipher context

JoelKatz
In reply to this post by Jim Wong

> Fortunately, we're not locked into any particular implementation
> strategy, so if we're barking up the wrong tree, there's opportunity to
> change.  For example, we've also considered:
>
> - Having the client do the encryption, which makes resuming
> uploads trivial, but complicates the client implementation (and requires
> that we pass the key around).
>
> - Doing block-by-block encryption, as you suggest, but this adds
> management overhead associated with dealing with the blocks after the
> upload is complete.
>
> - When an upload is resumed, read back, decrypt and re-encrypt
> the data already stored to get the context back to its previous state.
>
> - etc...
>
> I'd love to hear what you think.  Are there philosophical issues with
> saving the context across failed uploads?  Looking at the low-level AES
> routines, it actually looks like it would be pretty straightforward from
> an implementation standpoint, but if it's the wrong thing to do...

One huge problem with many encryption modes is that if the plaintext changes
at all, the ciphertext may become completely incomprehensible across the
resume point because the context will not be the same. So you will probably
need some resynch mechanism.

You actually have an interesting problem. If the client does the encryption,
it's not too complicated. But if you want the added flexibility of allowing
the server to totally control the encryption algorithm, it gets trickier.
You can have the server able to request SHA1 checksums over arbitrary byte
ranges, allowing the resynchronization to be totally controlled by the
server. But there's still the issue of how the server should do it.

DS


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Saving (and restoring) cipher context

Richard Koenning
In reply to this post by Jim Wong
Jim Wong wrote:

>
> I'm developing an application in which we're using AES to encrypt files as
> they're transferred from another system and saved to disk.  We'd like to
> provide the ability for the application to resume a transfer that was
> interrupted mid-stream, but the encryption throws a bit of a wrench into
> things because of the state associated with the encryption context.
>
> Is there a safe, supported way to stash the context somewhere on disk so
> that encryption can be resumed where it left off when the file transfer
> starts up again?  We're currently looking at the EVP functions; would we
> have to drop down to the lower-level, algorithm-specific routines to do this
> right?

If you use the CBC or the CFB cipher mode, you can use the last cipher block
written to the disk as IV (that means, before the restart point) for a call to
EVP_EncryptInit at the restart of data transmission. Another possibility would
be to implement counter mode on top of ECB, here should a restart also be simple.
Ciao,
Richard Könning
--
Dr. Richard W. Könning
Fujitsu Siemens Computers GmbH
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Saving (and restoring) cipher context

Jason Resch
In reply to this post by Jim Wong
If you have concerns about doing encryption client side and worry about
how to securely pass the key around you can try the oft-used strategy of
using an asymmetric (RSA) public key to encrypt a randomly generated
symmetric (AES) key.  This encrypted symmetric key can then be safely
stored with the file without worry, so long as the private RSA key is
not compromised.  Then you only need to worry about how this asymmetric
private key is managed.  The benefit of this model is that clients are
ensured that no one (not even the server) can see what data they are
uploading.

Regards,

Jason

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Jim Wong
Sent: Wednesday, May 30, 2007 6:20 PM
To: [hidden email]
Subject: RE: Saving (and restoring) cipher context

Thanks for the response.  Sorry for the ambiguity in my original
message: I thought keeping things abstract would make what we're doing
easier to understand at a high level, but I probably left things too
vague.

The short version is that we're implementing a kind of remote file
service that clients can use to store and retrieve files.  One of the
features we're trying to provide is the ability to resume uploads: for
example, if a client is storing a large file over a spotty network
connection, we don't want to have to restart the process every time the
connection drops.

For the actual file transfer, we're thinking of using vanilla SSL, for
several reasons: a) we don't want to reinvent the wheel; b) we'd like to
avoid having to deploy our own encryption code on all our client
platforms; and c) clients also transmit metadata that isn't part of the
file itself (and is stored separately from the file) that we don't want
to send over the wire in the clear.

Thus, the encryption I'm talking about is purely for the purpose of
storing the file on the server.  The idea is that even if someone were
able to gain unauthorized remote access to the storage server, he
shouldn't be able to read the contents of any of the uploaded files.  I
think this is somewhat closer to the very unusual case than the other
scenarios you described, but could be mistaken.

Fortunately, we're not locked into any particular implementation
strategy, so if we're barking up the wrong tree, there's opportunity to
change.  For example, we've also considered:

        - Having the client do the encryption, which makes resuming
uploads trivial, but complicates the client implementation (and requires
that we pass the key around).

        - Doing block-by-block encryption, as you suggest, but this adds
management overhead associated with dealing with the blocks after the
upload is complete.

        - When an upload is resumed, read back, decrypt and re-encrypt
the data already stored to get the context back to its previous state.

        - etc...

I'd love to hear what you think.  Are there philosophical issues with
saving the context across failed uploads?  Looking at the low-level AES
routines, it actually looks like it would be pretty straightforward from
an implementation standpoint, but if it's the wrong thing to do...

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of David Schwartz
Sent: Wednesday, May 30, 2007 1:42 PM
To: [hidden email]
Subject: RE: Saving (and restoring) cipher context


> I'm developing an application in which we're using AES to encrypt
> files as they're transferred from another system and saved to disk.  
> We'd like to provide the ability for the application to resume a
> transfer that was interrupted mid-stream, but the encryption throws a
> bit of a wrench into things because of the state associated with the
> encryption context.

> Is there a safe, supported way to stash the context somewhere on disk
> so that encryption can be resumed where it left off when the file
> transfer starts up again?  We're currently looking at the EVP
> functions; would we have to drop down to the lower-level,
> algorithm-specific routines to do this right?

Maybe you're locked into an implementation that isn't logical for your
problem set, but if not, change to a rational implementation.

If you are encrypting/decrypting during the transfer, that means the
encryption is to protect the transport. Since the resumption is over a
new transport, there is no rational reason to resume the previous
encryption.
Use a new encryption context for the new transport. If the encryption is
for storage, then resume sending the already-encrypted data.

The only case where you even have the problem you are describing is
where the data has to be encrypted differently from its normal state for
storage on the receiver. (Perhaps to be decrypted later.) Unless you're
in this very unusual situation, this shouldn't be an issue.

If you really are in this very unusual situation, I recommend one of the
following solutions:

1) Have a resync protocol that runs on both ends, re-encrypting the data
and sending block checksums. Let the other side verify the checksum
until you have a mismatch, then continue. You don't need to store the
context because this process recovers it.

2) Encrypt the file as a group of separate blocks. Simply discard any
incomplete blocks. Each block can have its own encryption context, so
there is no need to resume.

3) Use an encryption scheme that has negligible context. For example, if
you XOR each block of data with its block number and then encrypt it,
all you need to pick back up where you left off is the block number,
which you should already know.

Give us more details and tell us what you can and can't change. Your
proposed solution may not be the best one to your actual outer problem.

DS


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Saving (and restoring) cipher context

Jim Wong
Thanks for the suggestion.  I think our biggest concern about doing
encryption on the client side is exposing the underlying encryption
mechanism we're using for stored files.  Ideally, we'd like the server
to be a secure black box: files go in via SSL, are store securely, and
come back out the same way they went in.  That way, we're free to change
what happens on the server without having to update clients we have
deployed in the field.

That said, if it turns out that doing encryption on the client radically
simplifies things, it's certainly a possibility.

Also complicating things is the fact that clients other than the one
that orginally sent the file need to be able to view the data, but
that's another story...

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Jason Resch
Sent: Thursday, May 31, 2007 9:33 AM
To: [hidden email]
Subject: RE: Saving (and restoring) cipher context

If you have concerns about doing encryption client side and worry about
how to securely pass the key around you can try the oft-used strategy of
using an asymmetric (RSA) public key to encrypt a randomly generated
symmetric (AES) key.  This encrypted symmetric key can then be safely
stored with the file without worry, so long as the private RSA key is
not compromised.  Then you only need to worry about how this asymmetric
private key is managed.  The benefit of this model is that clients are
ensured that no one (not even the server) can see what data they are
uploading.

Regards,

Jason

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Jim Wong
Sent: Wednesday, May 30, 2007 6:20 PM
To: [hidden email]
Subject: RE: Saving (and restoring) cipher context

Thanks for the response.  Sorry for the ambiguity in my original
message: I thought keeping things abstract would make what we're doing
easier to understand at a high level, but I probably left things too
vague.

The short version is that we're implementing a kind of remote file
service that clients can use to store and retrieve files.  One of the
features we're trying to provide is the ability to resume uploads: for
example, if a client is storing a large file over a spotty network
connection, we don't want to have to restart the process every time the
connection drops.

For the actual file transfer, we're thinking of using vanilla SSL, for
several reasons: a) we don't want to reinvent the wheel; b) we'd like to
avoid having to deploy our own encryption code on all our client
platforms; and c) clients also transmit metadata that isn't part of the
file itself (and is stored separately from the file) that we don't want
to send over the wire in the clear.

Thus, the encryption I'm talking about is purely for the purpose of
storing the file on the server.  The idea is that even if someone were
able to gain unauthorized remote access to the storage server, he
shouldn't be able to read the contents of any of the uploaded files.  I
think this is somewhat closer to the very unusual case than the other
scenarios you described, but could be mistaken.

Fortunately, we're not locked into any particular implementation
strategy, so if we're barking up the wrong tree, there's opportunity to
change.  For example, we've also considered:

        - Having the client do the encryption, which makes resuming
uploads trivial, but complicates the client implementation (and requires
that we pass the key around).

        - Doing block-by-block encryption, as you suggest, but this adds
management overhead associated with dealing with the blocks after the
upload is complete.

        - When an upload is resumed, read back, decrypt and re-encrypt
the data already stored to get the context back to its previous state.

        - etc...

I'd love to hear what you think.  Are there philosophical issues with
saving the context across failed uploads?  Looking at the low-level AES
routines, it actually looks like it would be pretty straightforward from
an implementation standpoint, but if it's the wrong thing to do...

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of David Schwartz
Sent: Wednesday, May 30, 2007 1:42 PM
To: [hidden email]
Subject: RE: Saving (and restoring) cipher context


> I'm developing an application in which we're using AES to encrypt
> files as they're transferred from another system and saved to disk.
> We'd like to provide the ability for the application to resume a
> transfer that was interrupted mid-stream, but the encryption throws a
> bit of a wrench into things because of the state associated with the
> encryption context.

> Is there a safe, supported way to stash the context somewhere on disk
> so that encryption can be resumed where it left off when the file
> transfer starts up again?  We're currently looking at the EVP
> functions; would we have to drop down to the lower-level,
> algorithm-specific routines to do this right?

Maybe you're locked into an implementation that isn't logical for your
problem set, but if not, change to a rational implementation.

If you are encrypting/decrypting during the transfer, that means the
encryption is to protect the transport. Since the resumption is over a
new transport, there is no rational reason to resume the previous
encryption.
Use a new encryption context for the new transport. If the encryption is
for storage, then resume sending the already-encrypted data.

The only case where you even have the problem you are describing is
where the data has to be encrypted differently from its normal state for
storage on the receiver. (Perhaps to be decrypted later.) Unless you're
in this very unusual situation, this shouldn't be an issue.

If you really are in this very unusual situation, I recommend one of the
following solutions:

1) Have a resync protocol that runs on both ends, re-encrypting the data
and sending block checksums. Let the other side verify the checksum
until you have a mismatch, then continue. You don't need to store the
context because this process recovers it.

2) Encrypt the file as a group of separate blocks. Simply discard any
incomplete blocks. Each block can have its own encryption context, so
there is no need to resume.

3) Use an encryption scheme that has negligible context. For example, if
you XOR each block of data with its block number and then encrypt it,
all you need to pick back up where you left off is the block number,
which you should already know.

Give us more details and tell us what you can and can't change. Your
proposed solution may not be the best one to your actual outer problem.

DS


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Saving (and restoring) cipher context

Jim Wong
In reply to this post by Richard Koenning
Thanks, I'll look into that, particularly if I can't come up with a way to avoid a restart entirely.

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Richard Koenning
Sent: Thursday, May 31, 2007 5:54 AM
To: [hidden email]
Subject: Re: Saving (and restoring) cipher context

Jim Wong wrote:

>
> I'm developing an application in which we're using AES to encrypt
> files as they're transferred from another system and saved to disk.  
> We'd like to provide the ability for the application to resume a
> transfer that was interrupted mid-stream, but the encryption throws a
> bit of a wrench into things because of the state associated with the encryption context.
>
> Is there a safe, supported way to stash the context somewhere on disk
> so that encryption can be resumed where it left off when the file
> transfer starts up again?  We're currently looking at the EVP
> functions; would we have to drop down to the lower-level,
> algorithm-specific routines to do this right?

If you use the CBC or the CFB cipher mode, you can use the last cipher block written to the disk as IV (that means, before the restart point) for a call to EVP_EncryptInit at the restart of data transmission. Another possibility would be to implement counter mode on top of ECB, here should a restart also be simple.
Ciao,
Richard Könning
--
Dr. Richard W. Könning
Fujitsu Siemens Computers GmbH
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Saving (and restoring) cipher context

Jason Resch
In reply to this post by Jim Wong
If the server is doing all the encryption, then presumably the server is
responsible for storing the keys someplace where it has ready access to
them.  If this is the case, how is encryption protecting the data in the
event the server is compromised?  My concern is that this provides some
level of obfuscation without necessarily adding cryptographic security,
unless the clients are providing some kind of token needed by the server
to recover the keys.  Is that the case?

Best Regards,

Jason


-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Jim Wong
Sent: Thursday, May 31, 2007 1:08 PM
To: [hidden email]
Subject: RE: Saving (and restoring) cipher context

Thanks for the suggestion.  I think our biggest concern about doing
encryption on the client side is exposing the underlying encryption
mechanism we're using for stored files.  Ideally, we'd like the server
to be a secure black box: files go in via SSL, are store securely, and
come back out the same way they went in.  That way, we're free to change
what happens on the server without having to update clients we have
deployed in the field.

That said, if it turns out that doing encryption on the client radically
simplifies things, it's certainly a possibility.

Also complicating things is the fact that clients other than the one
that orginally sent the file need to be able to view the data, but
that's another story...

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Jason Resch
Sent: Thursday, May 31, 2007 9:33 AM
To: [hidden email]
Subject: RE: Saving (and restoring) cipher context

If you have concerns about doing encryption client side and worry about
how to securely pass the key around you can try the oft-used strategy of
using an asymmetric (RSA) public key to encrypt a randomly generated
symmetric (AES) key.  This encrypted symmetric key can then be safely
stored with the file without worry, so long as the private RSA key is
not compromised.  Then you only need to worry about how this asymmetric
private key is managed.  The benefit of this model is that clients are
ensured that no one (not even the server) can see what data they are
uploading.

Regards,

Jason

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Jim Wong
Sent: Wednesday, May 30, 2007 6:20 PM
To: [hidden email]
Subject: RE: Saving (and restoring) cipher context

Thanks for the response.  Sorry for the ambiguity in my original
message: I thought keeping things abstract would make what we're doing
easier to understand at a high level, but I probably left things too
vague.

The short version is that we're implementing a kind of remote file
service that clients can use to store and retrieve files.  One of the
features we're trying to provide is the ability to resume uploads: for
example, if a client is storing a large file over a spotty network
connection, we don't want to have to restart the process every time the
connection drops.

For the actual file transfer, we're thinking of using vanilla SSL, for
several reasons: a) we don't want to reinvent the wheel; b) we'd like to
avoid having to deploy our own encryption code on all our client
platforms; and c) clients also transmit metadata that isn't part of the
file itself (and is stored separately from the file) that we don't want
to send over the wire in the clear.

Thus, the encryption I'm talking about is purely for the purpose of
storing the file on the server.  The idea is that even if someone were
able to gain unauthorized remote access to the storage server, he
shouldn't be able to read the contents of any of the uploaded files.  I
think this is somewhat closer to the very unusual case than the other
scenarios you described, but could be mistaken.

Fortunately, we're not locked into any particular implementation
strategy, so if we're barking up the wrong tree, there's opportunity to
change.  For example, we've also considered:

        - Having the client do the encryption, which makes resuming
uploads trivial, but complicates the client implementation (and requires
that we pass the key around).

        - Doing block-by-block encryption, as you suggest, but this adds
management overhead associated with dealing with the blocks after the
upload is complete.

        - When an upload is resumed, read back, decrypt and re-encrypt
the data already stored to get the context back to its previous state.

        - etc...

I'd love to hear what you think.  Are there philosophical issues with
saving the context across failed uploads?  Looking at the low-level AES
routines, it actually looks like it would be pretty straightforward from
an implementation standpoint, but if it's the wrong thing to do...

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of David Schwartz
Sent: Wednesday, May 30, 2007 1:42 PM
To: [hidden email]
Subject: RE: Saving (and restoring) cipher context


> I'm developing an application in which we're using AES to encrypt
> files as they're transferred from another system and saved to disk.
> We'd like to provide the ability for the application to resume a
> transfer that was interrupted mid-stream, but the encryption throws a
> bit of a wrench into things because of the state associated with the
> encryption context.

> Is there a safe, supported way to stash the context somewhere on disk
> so that encryption can be resumed where it left off when the file
> transfer starts up again?  We're currently looking at the EVP
> functions; would we have to drop down to the lower-level,
> algorithm-specific routines to do this right?

Maybe you're locked into an implementation that isn't logical for your
problem set, but if not, change to a rational implementation.

If you are encrypting/decrypting during the transfer, that means the
encryption is to protect the transport. Since the resumption is over a
new transport, there is no rational reason to resume the previous
encryption.
Use a new encryption context for the new transport. If the encryption is
for storage, then resume sending the already-encrypted data.

The only case where you even have the problem you are describing is
where the data has to be encrypted differently from its normal state for
storage on the receiver. (Perhaps to be decrypted later.) Unless you're
in this very unusual situation, this shouldn't be an issue.

If you really are in this very unusual situation, I recommend one of the
following solutions:

1) Have a resync protocol that runs on both ends, re-encrypting the data
and sending block checksums. Let the other side verify the checksum
until you have a mismatch, then continue. You don't need to store the
context because this process recovers it.

2) Encrypt the file as a group of separate blocks. Simply discard any
incomplete blocks. Each block can have its own encryption context, so
there is no need to resume.

3) Use an encryption scheme that has negligible context. For example, if
you XOR each block of data with its block number and then encrypt it,
all you need to pick back up where you left off is the block number,
which you should already know.

Give us more details and tell us what you can and can't change. Your
proposed solution may not be the best one to your actual outer problem.

DS


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Saving (and restoring) cipher context

JoelKatz

> If the server is doing all the encryption, then presumably the server is
> responsible for storing the keys someplace where it has ready access to
> them.

That's really not something you can assume. For example, the server might
hold only public keys and when it needs to decrypt something, the private
key is provided to it by the client and stored only for the duration of the
transfer. It's possible that the keys are stored on the server, but doing
the encryption on the server means only that the server must have the
encryption key while it is encrypting. It says nothing about the decryption
key or where the keys are stored when they aren't in use.

> If this is the case, how is encryption protecting the data in the
> event the server is compromised?

It depends what your threat model is. If the only lines into the server are
serial ports with a protocol that's locked down, a server compromise is not
likely to be on your radar. Many of these schemes scope is to secure the
client and server security is outside their scope.

That said, it's best to provide structural security in the server if this is
at all possible. And it almost always is.

> My concern is that this provides some
> level of obfuscation without necessarily adding cryptographic security,
> unless the clients are providing some kind of token needed by the server
> to recover the keys.  Is that the case?

I think your argument is based on a false premise. In the majority of
real-world cases, the server is much more secure generally than the clients
are. Storing the keys on the server is likely going to be safer than storing
them on the client.

I think your implied reasoning is that each client can only compromise its
own security while the server can compromise the security of all the
clients. While this is true, there are many implementation scenarios where
it's inapplicable. Consider, for example, a scenario where all the data is
actually owned by (and disclosure would primarily harm) the server rather
than the clients. In that case, scattering the keys left and right doesn't
make things any more secure -- just much less so. At least you can secure
the server if you control it.

On the other hand, you may be dead on. We don't know enough about the OP's
scenario to say all that much. Your comments just seem to have one very
particular scenario in mind and one that I don't think is all that typical.

DS


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Saving (and restoring) cipher context

Jim Wong
In reply to this post by JoelKatz
Thanks for the information.  We were thinking about having the client
send up a hash over the entire file any time it attempted to start or
resume an upload, and if the computed checksum didn't match the
original, just force the client to start over from the beginning.

Would this work?

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of David Schwartz
Sent: Wednesday, May 30, 2007 8:01 PM
To: [hidden email]
Subject: RE: Saving (and restoring) cipher context


One huge problem with many encryption modes is that if the plaintext
changes at all, the ciphertext may become completely incomprehensible
across the resume point because the context will not be the same. So you
will probably need some resynch mechanism.

You actually have an interesting problem. If the client does the
encryption, it's not too complicated. But if you want the added
flexibility of allowing the server to totally control the encryption
algorithm, it gets trickier.
You can have the server able to request SHA1 checksums over arbitrary
byte ranges, allowing the resynchronization to be totally controlled by
the server. But there's still the issue of how the server should do it.

DS


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Saving (and restoring) cipher context

Jason Resch
In reply to this post by JoelKatz


-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of David Schwartz
Sent: Thursday, May 31, 2007 7:17 PM
To: [hidden email]
Subject: RE: Saving (and restoring) cipher context


> I think your argument is based on a false premise. In the majority of
> real-world cases, the server is much more secure generally than the
clients
> are. Storing the keys on the server is likely going to be safer than
storing them on the client.




I just wanted to raise the point that if symmetric keys are stored in an
unprotected state on the same server as the encrypted data, I see little
benefit of doing the encryption if a against compromise leads to both
the keys and ciphertexts.  If no additional protection (such as PBE,
RSA, etc.) is used to secure the symmetric keys in this scenario, then
encryption provides little additional security.

Of course it is an assumption that the symmetric keys are in no way
protected, but I thought I should mention it mention given that it was
said public key cryptography was not currently being used.

Best Regards,

Jason

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]