PEM_read_bio_X509:BIO_gets:unsupported method

[David Taylor-6]

Hi all,

I only just joined this list today to past this patch.

32bit Solaris has a nasty bug in fopen(3C) which prevents
opening a file once there are more than 256 descriptors
active in a process. The work-around is to use open(2)
instead. See: http://access1.sun.com/technotes/01406.html
This affects all Solaris versions, including 10, on both
Intel and Sparc.

PEM_read_bio_X509 uses BIO_gets, so if you give it an
fd BIO it fails with "BIO_gets:unsupported method" as
that routine is not implemented.

What follows is a tested and working implementation done
in 0.9.7g. This fixes the problem on Solaris. Code to
test it would be along the lines of:

fd = open(filename, O_RDONLY);
bio = BIO_new_fd(fd, BIO_NOCLOSE);
cert = PEM_read_bio_X509(bio, NULL, NULL, NULL);

Can we get this happening in the next 0.9.7 and 0.9.8
releases?

Cheers,
David

--
crypto/bio/bss_fd.c:
68a69,70
 > static int fd_gets(BIO *h, char *buf, int size);
 > static int fd_getc(BIO *bp, char *c);
79c81
<       NULL, /* fd_gets, */
---
 >       fd_gets,
217a220,251
 > static int fd_getc(BIO *bp, char *c)
 >       {
 >               return(fd_read(bp,c,1));
 >       }
 >
 > static int fd_gets(BIO *bp, char *buf, int size)
 >       {
 >       int i, ret;
 >       char c;
 >
 >       memset(buf,0,size);
 >
 >       for ( i=0 ; ((i < size)&&(fd_getc(bp,&c)==1)) ; i++ )
 >       {
 >               if (c=='\r')
 >               {
 >                       continue;
 >               }
 >               else if (c=='\n')
 >               {
 >                       break;
 >               }
 >               else
 >               {
 >                       buf[i] = c;
 >               }
 >       }
 >       buf[i] = (char)NULL;
 >       ret = strlen(buf);
 >       return ret;
 >       }
 >
--
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]

[Jean-Marc Desperrier-2]

David Taylor wrote:

> I only just joined this list today to past this patch.

So in one word :
- for technical reasons, "fd" bio are preferable to "file" bio on Solaris
- but as "fd" bio don't implement gets, they are not usable as a direct
replacement for "file" bio
- your attached patch implements gets for "fd" bio to align them
functionally with "file" bio and solve that problem

Note that this gets is implemented in a way that can become very bad
performance-wise (there's no easy solution for that, I'm mostly saying
implementers shoud beware of it).


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]

[Per Hedeland-2]

Jean-Marc Desperrier <[hidden email]> wrote:
>
>David Taylor wrote:
>
>> I only just joined this list today to past this patch.
>
>So in one word :
>- for technical reasons, "fd" bio are preferable to "file" bio on Solaris

Actually there's another workaround possible for the Solaris problem:
The description in the previous message wasn't entirely accurate,
there's no problem having more than 256 fds open and still use stdio -
the limitation is that stdio can only use fds that are below 256 (due to
the fd being kept in a char in the FILE struct, which can't be changed
without breaking binary backwards compatibility).

So, as long as you make sure that there are fds avaliable below 256, it
works fine - and this can in many/most cases be achieved by the
application making sure that other fds (notably sockets) are above that
number, through moving them there with fcntl(F_DUPFD) when needed.

This not being said as a comment on the merits of the suggested patch,
but it's probably worth noting that the problem is certainly not
OpenSSL-specific - an application can run into it whenever it tries to
use stdio (could even happen in some library outside the application's
control), and resorting to never using stdio is a pretty severe
workaround.

--Per Hedeland
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]

[Richard Salz]

> the limitation is that stdio can only use fds that are below 256 (due to
> the fd being kept in a char in the FILE struct, which can't be changed
> without breaking binary backwards compatibility).

Amazing.   I first came across this problem with INN over a decade ago.
It's still an issue?  Wow.  Sun should  be embarassed.

        /r$

--
SOA Appliance Group
IBM Application Integration Middleware


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]

[Per Hedeland-2]

Richard Salz <[hidden email]> wrote:
 [I wrote:]
>> the limitation is that stdio can only use fds that are below 256 (due to
>> the fd being kept in a char in the FILE struct, which can't be changed
>> without breaking binary backwards compatibility).
>
>Amazing.   I first came across this problem with INN over a decade ago.

Actually that's where I first came across it too:-) - in innfeed (which
uses the F_DUPFD workaround if so configured).

>It's still an issue?

I don't personally know - the OP claimed it was still there in 32-bit
mode in Solaris 10.

>  Wow.  Sun should  be embarassed.

Well, how *do* you fix it "without breaking binary backwards
compatibility"? The layout of the FILE struct is embedded in all
binaries using the stdio macros (e.g. getc()/putc()). Introducing
fopen_256(), *printf_256() etc etc to get around it doesn't sound all
that appealing.

--Per Hedeland
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]

[Richard Salz]

> Well, how *do* you fix it "without breaking binary backwards
> compatibility"? The layout of the FILE struct is embedded in all
> binaries using the stdio macros (e.g. getc()/putc()).

I don't know, but I can't believe that within a decade that they couldn't
have come up with something.  A "fixup" program that edited object files
turning inline code into function calls, e.g.

        /r$

--
SOA Appliance Group
IBM Application Integration Middleware

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]

[Jack Lloyd]

On Fri, Jan 06, 2006 at 08:37:25PM +0100, Per Hedeland wrote:

> >  Wow.  Sun should  be embarassed.
>
> Well, how *do* you fix it "without breaking binary backwards
> compatibility"? The layout of the FILE struct is embedded in all
> binaries using the stdio macros (e.g. getc()/putc()). Introducing
> fopen_256(), *printf_256() etc etc to get around it doesn't sound all
> that appealing.

Actually, it's worse than that, depending on how much you care about backwards
comptability. For example, if I pass a FILE* from a newly compiled application
to a function living in a shared library built on (say) Solaris 2.2, I might
well expect that to just work.

That said, I agree that it's laughable that a modern Unix would have a
limitation such as this. Why not at least add a macro
-DMAKE_MY_STDIO_NOT_BROKEN that gives you more than 256 files but breaks
compatability? That has proven to be acceptable for 64-bit file support on
Unix, and I don't see why this stdio problem would be different.

Anyway, this is probably getting really offtopic for the list. The bug exists
and needs to be worked around (or not). Sun's suckage or lack thereof does't
really come into that issue.

-J
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]

[Doug Royer]

FYI: Its not a bug. It is a backward binary compatibly
issue. In the BSD and older SVR days, the file descriptor
in the FILE struct is a 'char', not an 'int'. So in order
to still run the same older binary files, the size of the
FILE struct can not change. Old time UNIX's know that this
has been a limit in FILE since the late 70's.

And I seem to recall that it is in a SVID and/or POSIX
or other API definition as a 'char'.

So, other OS's may have the same 'standardized' limit.
To be portable across OS's, developers should assume
that the file descriptor in the FILE struct is a 'char'


David Taylor wrote:

> Hi all,
>
> I only just joined this list today to past this patch.
>
> 32bit Solaris has a nasty bug in fopen(3C) which prevents
> opening a file once there are more than 256 descriptors
> active in a process. The work-around is to use open(2)
> instead. See: http://access1.sun.com/technotes/01406.html
> This affects all Solaris versions, including 10, on both
> Intel and Sparc.
>

--

Doug Royer                     | http://INET-Consulting.com
-------------------------------|-----------------------------

               We Do Standards - You Need Standards