possible bug in BIO_f_base64 ?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

possible bug in BIO_f_base64 ?

Beat Jucker
context "decrypting smime message":
cat email | openssl-0.9.8 smime -inkey mykey -recip mycert

  1885:error:0D06B08E:asn1 encoding routines:ASN1_D2I_READ_BIO:not enough data:a_d2i_fp.c:238:
  1885:error:21078082:PKCS7 routines:B64_READ_PKCS7:decode error:pk7_mime.c:140:
  1885:error:2107A08B:PKCS7 routines:SMIME_read_PKCS7:pkcs7 parse error:pk7_mime.c:373:

I have already posted this problem to the openssl-users
community but didn't got an answer yet for the reason
of this error message. I also believe it is a library problem
and not an application problem.

First I thought there's an error in the ASN.1 structure because
openssl asn1parse showed me missing EOT markes at the end of file.
But than I noticed that openssl doesn't decode the BASE64 smime
file like other tools (but only on a few particular smime messages).

Given attached BASE64 encoded file openssl will write only 5280
decoded bytes instead of the original 5305 bytes as other tools
like mimencode, base64, Asn1Editor, web online base64 decoder, ...

  openssl base64 -d -in text.pem -out text.der
  --> 5280 instead of 5305 bytes!?

I couldn't find a problem with the base64 encoded source file.
However when I insert at least one "illegal" base64 character
somewhere in the base64 text (e.g. <SPACE> as first character
or insert an empty line somewhere in between the regular base64
text) I'll get the correct decoded message (5305 versus 5280 bytes)!

For me it sounds like a bug in the base64 library but my C knowlege
ist a little bit outdated to realy locate the problem in the library.

Thanks
-- Beat

PS: I don't know which tool was used to generate the base64 file

text.pem (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: possible bug in BIO_f_base64 ?

Richard Levitte - VMS Whacker
In message <[hidden email]> on Mon, 20 Jun 2005 12:11:30 +0200, Beat Jucker <[hidden email]> said:

bj> Given attached BASE64 encoded file openssl will write only 5280
bj> decoded bytes instead of the original 5305 bytes as other tools
bj> like mimencode, base64, Asn1Editor, web online base64 decoder, ...
bj>
bj>   openssl base64 -d -in text.pem -out text.der
bj>   --> 5280 instead of 5305 bytes!?

I've played with previous incarnations, and noticed that with the
latest update for 0.9.7-stable, I get 5305 bytes, while I get 5280
bytes with 0.9.8-stable.  I compared crypto/evp/bio_b64.c from both
branches, and there is virtually no difference, so the problem is
somewhere else.

I noticed something unusual about your file: the lines are 76
characters, when a PEM file usually (or at least by default when
output by OpenSSL) has 64 character lines...  I have no clue how
important that fact is, but I'm going to conduct some tests.

Cheers,
Richard

-----
Please consider sponsoring my work on free software.
See http://www.free.lp.se/sponsoring.html for details.

--
Richard Levitte                         [hidden email]
                                        http://richard.levitte.org/

"When I became a man I put away childish things, including
 the fear of childishness and the desire to be very grown up."
                                                -- C.S. Lewis
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: possible bug in BIO_f_base64 ?

Dr. Stephen Henson
On Mon, Jun 20, 2005, Richard Levitte - VMS Whacker wrote:

> In message <[hidden email]> on Mon, 20 Jun 2005 12:11:30 +0200, Beat Jucker <[hidden email]> said:
>
> bj> Given attached BASE64 encoded file openssl will write only 5280
> bj> decoded bytes instead of the original 5305 bytes as other tools
> bj> like mimencode, base64, Asn1Editor, web online base64 decoder, ...
> bj>
> bj>   openssl base64 -d -in text.pem -out text.der
> bj>   --> 5280 instead of 5305 bytes!?
>
> I've played with previous incarnations, and noticed that with the
> latest update for 0.9.7-stable, I get 5305 bytes, while I get 5280
> bytes with 0.9.8-stable.  I compared crypto/evp/bio_b64.c from both
> branches, and there is virtually no difference, so the problem is
> somewhere else.
>
> I noticed something unusual about your file: the lines are 76
> characters, when a PEM file usually (or at least by default when
> output by OpenSSL) has 64 character lines...  I have no clue how
> important that fact is, but I'm going to conduct some tests.
>

The only significant change is:

http://cvs.openssl.org/chngview?cn=12988

whether this is the problem or it has just triggered a problem elsewhere I
don't know.

Steve.
--
Dr Stephen N. Henson. Email, S/MIME and PGP keys: see homepage
OpenSSL project core developer and freelance consultant.
Funding needed! Details on homepage.
Homepage: http://www.drh-consultancy.demon.co.uk
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: possible bug in BIO_f_base64 ?

Richard Levitte - VMS Whacker
In message <[hidden email]> on Mon, 20 Jun 2005 18:53:39 +0200, "Dr. Stephen Henson" <[hidden email]> said:

steve> On Mon, Jun 20, 2005, Richard Levitte - VMS Whacker wrote:
steve>
steve> > In message <[hidden email]> on Mon, 20 Jun 2005 12:11:30 +0200, Beat Jucker <[hidden email]> said:
steve> >
steve> > bj> Given attached BASE64 encoded file openssl will write only 5280
steve> > bj> decoded bytes instead of the original 5305 bytes as other tools
steve> > bj> like mimencode, base64, Asn1Editor, web online base64 decoder, ...
steve> > bj>
steve> > bj>   openssl base64 -d -in text.pem -out text.der
steve> > bj>   --> 5280 instead of 5305 bytes!?
steve> >
steve> > I've played with previous incarnations, and noticed that with the
steve> > latest update for 0.9.7-stable, I get 5305 bytes, while I get 5280
steve> > bytes with 0.9.8-stable.  I compared crypto/evp/bio_b64.c from both
steve> > branches, and there is virtually no difference, so the problem is
steve> > somewhere else.
steve> >
steve> > I noticed something unusual about your file: the lines are 76
steve> > characters, when a PEM file usually (or at least by default when
steve> > output by OpenSSL) has 64 character lines...  I have no clue how
steve> > important that fact is, but I'm going to conduct some tests.
steve> >
steve>
steve> The only significant change is:
steve>
steve> http://cvs.openssl.org/chngview?cn=12988
steve>
steve> whether this is the problem or it has just triggered a problem
steve> elsewhere I don't know.

This specific case seems to be because of the 76 character lines.  The
attached patch seems to fix it, though.

Really, the base64 decoder is quite the pile of crap.  Why on earth
does it have dependence on where a NL will appear?  There's absolutely
no reason unless you're a PEM fetishist...  It should really be
rewritten...

-----
Please consider sponsoring my work on free software.
See http://www.free.lp.se/sponsoring.html for details.

--
Richard Levitte                         [hidden email]
                                        http://richard.levitte.org/

"When I became a man I put away childish things, including
 the fear of childishness and the desire to be very grown up."
                                                -- C.S. Lewis

Index: crypto/evp/encode.c
===================================================================
RCS file: /e/openssl/cvs/openssl/crypto/evp/encode.c,v
retrieving revision 1.14
diff -u -r1.14 encode.c
--- crypto/evp/encode.c 3 Apr 2005 16:38:22 -0000 1.14
+++ crypto/evp/encode.c 20 Jun 2005 22:01:26 -0000
@@ -313,7 +313,7 @@
  /* There will never be more than two '=' */
  }
 
- if ((v == B64_EOF) || (n >= 64))
+ if ((v == B64_EOF && (n&3) == 0) || (n >= 64))
  {
  /* This is needed to work correctly on 64 byte input
  * lines.  We process the line and then need to
Reply | Threaded
Open this post in threaded view
|

Re: possible bug in BIO_f_base64 ?

Dr. Stephen Henson
In reply to this post by Dr. Stephen Henson
On Mon, Jun 20, 2005, Dr. Stephen Henson wrote:

> On Mon, Jun 20, 2005, Richard Levitte - VMS Whacker wrote:
>
> > In message <[hidden email]> on Mon, 20 Jun 2005 12:11:30 +0200, Beat Jucker <[hidden email]> said:
> >
> > bj> Given attached BASE64 encoded file openssl will write only 5280
> > bj> decoded bytes instead of the original 5305 bytes as other tools
> > bj> like mimencode, base64, Asn1Editor, web online base64 decoder, ...
> > bj>
> > bj>   openssl base64 -d -in text.pem -out text.der
> > bj>   --> 5280 instead of 5305 bytes!?
> >
> > I've played with previous incarnations, and noticed that with the
> > latest update for 0.9.7-stable, I get 5305 bytes, while I get 5280
> > bytes with 0.9.8-stable.  I compared crypto/evp/bio_b64.c from both
> > branches, and there is virtually no difference, so the problem is
> > somewhere else.
> >
> > I noticed something unusual about your file: the lines are 76
> > characters, when a PEM file usually (or at least by default when
> > output by OpenSSL) has 64 character lines...  I have no clue how
> > important that fact is, but I'm going to conduct some tests.
> >
>
> The only significant change is:
>
> http://cvs.openssl.org/chngview?cn=12988
>
> whether this is the problem or it has just triggered a problem elsewhere I
> don't know.
>

Looks like the latter. That example causes problems because the ultimate '='
characters representing EOF are passed in separate calls to
EVP_DecodeUpdate() and EVP_DecodeBlock() ends up being called with an buffer
that is not a multiple of 4 characters in length.

The cause *may* be the line:

                if (((i+1) == inl) && (((n&3) == 0) || eof))

which causes processing if only one '=' has been reached at the end of the
input data. Removing the 'eof' check at the end seems to fix this.

Steve.
--
Dr Stephen N. Henson. Email, S/MIME and PGP keys: see homepage
OpenSSL project core developer and freelance consultant.
Funding needed! Details on homepage.
Homepage: http://www.drh-consultancy.demon.co.uk
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: possible bug in BIO_f_base64 ?

Dr. Stephen Henson
In reply to this post by Richard Levitte - VMS Whacker
On Tue, Jun 21, 2005, Richard Levitte - VMS Whacker wrote:

> In message <[hidden email]> on Mon, 20 Jun 2005 18:53:39 +0200, "Dr. Stephen Henson" <[hidden email]> said:
>
> steve> On Mon, Jun 20, 2005, Richard Levitte - VMS Whacker wrote:
> steve>
> steve> > In message <[hidden email]> on Mon, 20 Jun 2005 12:11:30 +0200, Beat Jucker <[hidden email]> said:
> steve> >
> steve> > bj> Given attached BASE64 encoded file openssl will write only 5280
> steve> > bj> decoded bytes instead of the original 5305 bytes as other tools
> steve> > bj> like mimencode, base64, Asn1Editor, web online base64 decoder, ...
> steve> > bj>
> steve> > bj>   openssl base64 -d -in text.pem -out text.der
> steve> > bj>   --> 5280 instead of 5305 bytes!?
> steve> >
> steve> > I've played with previous incarnations, and noticed that with the
> steve> > latest update for 0.9.7-stable, I get 5305 bytes, while I get 5280
> steve> > bytes with 0.9.8-stable.  I compared crypto/evp/bio_b64.c from both
> steve> > branches, and there is virtually no difference, so the problem is
> steve> > somewhere else.
> steve> >
> steve> > I noticed something unusual about your file: the lines are 76
> steve> > characters, when a PEM file usually (or at least by default when
> steve> > output by OpenSSL) has 64 character lines...  I have no clue how
> steve> > important that fact is, but I'm going to conduct some tests.
> steve> >
> steve>
> steve> The only significant change is:
> steve>
> steve> http://cvs.openssl.org/chngview?cn=12988
> steve>
> steve> whether this is the problem or it has just triggered a problem
> steve> elsewhere I don't know.
>
> This specific case seems to be because of the 76 character lines.  The
> attached patch seems to fix it, though.
>

Yes that should work. Its the '=' in separate records that seems to be the
root cause IMHO which the 76 line input would also cause.

> Really, the base64 decoder is quite the pile of crap.  Why on earth
> does it have dependence on where a NL will appear?  There's absolutely
> no reason unless you're a PEM fetishist...  It should really be
> rewritten...
>

It is bloody horrible, has been about since the dawn of time (well SSLeay
anyway) and has been patched up a couple of times where it can't handle
boundary cases.

I agree a readable robust complete rewrite would be a good idea.

Steve.
--
Dr Stephen N. Henson. Email, S/MIME and PGP keys: see homepage
OpenSSL project core developer and freelance consultant.
Funding needed! Details on homepage.
Homepage: http://www.drh-consultancy.demon.co.uk
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]