Can a PKCS7 encrypted object use stream ciphers (eg RC4)? Is this a
standard or implementation issue? Its not clear to me that the pkcs7
standard limits use of the streaming ciphers.
I notice that the command line tool (openssl smime) only defines block
ciphers in its documentation.
Programatically I can call PKCS7_encrypt using enveloped mode specifying
a cipher type of RC4. However it falls over on decrypt because
EVP_CIPHER_asn1_to_param(called from PKCS7_dataDecode) returns -1
because there are no parameters available for this cipher within the
envelope. But this worked in 0.9.7d.
I reviewed the CVS page for evp_lib.c (which contains
EVP_CIPHER_asn1_to_param) and a change between 0.9.7d and 0.9.8a
specifies that "Return an error if an attempt is made to encode or
decode cipher ASN1 parameters and the cipher doesn't support it."
(Check-in Number:12376, Files: openssl/crypto/evp/evp_lib.c
The return code of -1 is interpreted in PKCS7_dataDecode() as a failure
and a general decrypt error is returned.
So I guess my question is: If PKCS7 allows the use of streaming ciphers
then is the interpretation of the result of EVP_CIPHER_asn1_to_param
correct in PKCS7_dataDecode? Or, is there a reason standard or otherwise
that stream ciphers should not be used in PKCS7 enveloped encryption?