Problems with including zlib

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Problems with including zlib

grarpamp
I have a case that needs zlib statically in openssl.
But I can't seem to make that. Only dynamic is made.

For testing I put zlib125 in its own <dir>.
Then for openssl...

./config
--prefix=<path>
--with-zlib-include=<dir>/include
--with-zlib-lib=<dir>/lib
shared
zlib

make
make install

This gives...

LD_LIBRARY_PATH=../zlib125/lib ldd bin/openssl
bin/openssl:
   libssl.so.1.0.0 => /.../openssl100e/lib/libssl.so.1.0.0 (0x28102000)
   libcrypto.so.1.0.0 => /.../openssl100e/lib/libcrypto.so.1.0.0 (0x28151000)
=> libz.so.1 => ../zlib125/lib/libz.so.1 (0x282bd000)
   libthr.so.3 => /lib/libthr.so.3 (0x282d3000)
   libc.so.7 => /lib/libc.so.7 (0x282e8000)

As far as I understand things...
'zlib-dynamic' is to include zlib dynamically.
'zlib' is to include zlib statically.
But as shown above, 'zlib' is not doing that.

So I think this is a bug, is that correct?
If it is my configuration usage, how do I make zlib static?


Also, is 'enc -z' the standard zlib testing method?
Why am I getting such a poor compression ratio?
The hashes are md5 in place of wc.
I did check that adding the final reverse '-d' pipe does yield 1MiB
zeros.

dd if=/dev/zero bs=1m count=1 | LD_LIBRARY_PATH=../zlib125/lib
bin/openssl enc -aes128 -a -k 0 -S 0 -e | wc -c
1419990 f8a4638a913dedd984f987767a24796c
dd if=/dev/zero bs=1m count=1 | LD_LIBRARY_PATH=../zlib125/lib
bin/openssl enc -aes128 -a -k 0 -S 0 -e -z | wc -c
1080369 a48155fc72d937c4cfdc609e18b470ac
dd if=/dev/zero bs=1m count=1 | LD_LIBRARY_PATH=../zlib125/lib
bin/openssl zlib -aes128 -a -k 0 -S 0 -e | wc -c
1080369 a48155fc72d937c4cfdc609e18b470ac
dd if=/dev/zero bs=1m count=1 | LD_LIBRARY_PATH=../zlib125/lib
bin/openssl zlib -e | wc -c
1039 cfd12ca5dd18c8cd22cc34bb757c8b9f
dd if=/dev/zero bs=1m count=1 | gzip -n |
LD_LIBRARY_PATH=../zlib125/lib bin/openssl enc -aes128 -a -k 0 -S 0 -e
| wc -c
1455 08d52ad99d2f6edf344ef735ad62150c


There are some documentation bugs...
enc.1: -z should read: 'was compiled'.
'openssl zlib' does not appear to be documented.

Thanks.


FreeBSD RELENG_8 i386
OpenSSL 1.0.0e 6 Sep 2011
platform: BSD-x86-elf
options: bn(64,32) rc4(4x,int) des(ptr,risc1,16,long) idea(int)
 blowfish(idx)
compiler: gcc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -pthread
 -D_THREAD_SAFE -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -Wa,--noexecstack
 -DL_ENDIAN -DTERMIOS -O3 -fomit-frame-pointer -Wall
 -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_BN_ASM_MONT -DSHA1_ASM
 -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DRMD160_ASM -DAES_ASM
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Problems with including zlib

Jakob Bohm-7
On 12/19/2011 10:38 AM, grarpamp wrote:

> I have a case that needs zlib statically in openssl.
> But I can't seem to make that. Only dynamic is made.
>
> For testing I put zlib125 in its own<dir>.
> Then for openssl...
>
> ./config
> --prefix=<path>
> --with-zlib-include=<dir>/include
> --with-zlib-lib=<dir>/lib
> shared
> zlib
>
> make
> make install
>
> This gives...
>
> LD_LIBRARY_PATH=../zlib125/lib ldd bin/openssl
> bin/openssl:
>     libssl.so.1.0.0 =>  /.../openssl100e/lib/libssl.so.1.0.0 (0x28102000)
>     libcrypto.so.1.0.0 =>  /.../openssl100e/lib/libcrypto.so.1.0.0 (0x28151000)
> =>  libz.so.1 =>  ../zlib125/lib/libz.so.1 (0x282bd000)
>     libthr.so.3 =>  /lib/libthr.so.3 (0x282d3000)
>     libc.so.7 =>  /lib/libc.so.7 (0x282e8000)
>
> As far as I understand things...
> 'zlib-dynamic' is to include zlib dynamically.
> 'zlib' is to include zlib statically.
> But as shown above, 'zlib' is not doing that.
>
> So I think this is a bug, is that correct?
> If it is my configuration usage, how do I make zlib static?
1. Make sure there is a libz.a in /lib or /usr/lib, otherwise you have
no static
  zlib to link in.

2. Temporarily remove or rename the symlink named exactly "libz.so" in
    /lib, /usr/lib, /usr/local/lib and /zlib125/lib (This ensures it
cannot link to
    the dynamic zlib).

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Problems with including zlib

grarpamp
> 1. Make sure there is a libz.a in /lib or /usr/lib, otherwise you have no
> static zlib to link in.

Of course there's an old libz.a there. And it should not matter as
we're given the --with-zlib arguments to point the build elsewhere
for those libraries. And as seen in the report, it is following those
pointers. It's just not using them correctly regarding being told to
link against libz.a, not libz.so, with the 'zlib' parameter to config.

> 2. Temporarily remove or rename the symlink named exactly "libz.so" in
> /lib, /usr/lib, /usr/local/lib and /zlib125/lib (This ensures it cannot
> link to the dynamic zlib).

No, this appears to be to be a ./config build parameter setup
error. Why should user's break their perfectly sound systems
in order to work around a bug? If users wanted it to link dynamically
against libz, they would have specified 'zlib-dynamic' to ./config.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Problems with including zlib

Michael S. Zick-4
In reply to this post by grarpamp
On Mon December 19 2011, grarpamp wrote:

> I have a case that needs zlib statically in openssl.
> But I can't seem to make that. Only dynamic is made.
>
> For testing I put zlib125 in its own <dir>.
> Then for openssl...
>
> ./config
> --prefix=<path>
> --with-zlib-include=<dir>/include
> --with-zlib-lib=<dir>/lib
> shared
> zlib
>
> make
> make install
>
> This gives...
>
> LD_LIBRARY_PATH=../zlib125/lib ldd bin/openssl
> bin/openssl:
>    libssl.so.1.0.0 => /.../openssl100e/lib/libssl.so.1.0.0 (0x28102000)
>    libcrypto.so.1.0.0 => /.../openssl100e/lib/libcrypto.so.1.0.0 (0x28151000)
> => libz.so.1 => ../zlib125/lib/libz.so.1 (0x282bd000)
>    libthr.so.3 => /lib/libthr.so.3 (0x282d3000)
>    libc.so.7 => /lib/libc.so.7 (0x282e8000)
>
> As far as I understand things...
> 'zlib-dynamic' is to include zlib dynamically.
> 'zlib' is to include zlib statically.
> But as shown above, 'zlib' is not doing that.
>
> So I think this is a bug, is that correct?
> If it is my configuration usage, how do I make zlib static?
>

I had to read this thread several times before I caught onto
the context of your question.

If I restate your question a bit for the more general case:

I have an application which should link statically against
one (or more) specific library(ies) and dynamically against
the rest of the external libraries...

Then the general case answer is:

You have to make two link(er) passes;
The first to produce a "partial link" output that is your
application linked against the selected static library(ies);
The second to complete the linkage using the dynamic library(ies).

The general case answer when using a GCC/Binutils toolchain:

For many situations you can emulate multiple linker passes by using
the --start-group <archives> --end-group pair of option quotes.
From the gcc command line: -Wl,--start-group <archives> -Wl,--end-group

There is a short form of that option pair: -( archives )-

Where: "archives" may be a list of object files also.

So if Binutils linker (or GCC linker) command included only static library(ies)
within the group, then the linker would make repeated passes over that
group until no new undefined symbols where generated.
In this case, by statically linking in the specified archive(s).  ;-)

Your more specific question:

How to get the OpenSSL Makefile to recognize: zlib-static (or zlib as static)?

If using GCC/Binutils toolchain, then answering that one is a matter
of either reading the Makefile(s) or waiting for an author of the Makefile(s)
to take note of this thread and tell us.

The thing(s) to look for would be either a Makefile that produces more than
one linker run over the object files or that uses the grouping quotes (or both).

If using any other toolchain, I don't have the slightest clue.

Mike

>
> Also, is 'enc -z' the standard zlib testing method?
> Why am I getting such a poor compression ratio?
> The hashes are md5 in place of wc.
> I did check that adding the final reverse '-d' pipe does yield 1MiB
> zeros.
>
> dd if=/dev/zero bs=1m count=1 | LD_LIBRARY_PATH=../zlib125/lib
> bin/openssl enc -aes128 -a -k 0 -S 0 -e | wc -c
> 1419990 f8a4638a913dedd984f987767a24796c
> dd if=/dev/zero bs=1m count=1 | LD_LIBRARY_PATH=../zlib125/lib
> bin/openssl enc -aes128 -a -k 0 -S 0 -e -z | wc -c
> 1080369 a48155fc72d937c4cfdc609e18b470ac
> dd if=/dev/zero bs=1m count=1 | LD_LIBRARY_PATH=../zlib125/lib
> bin/openssl zlib -aes128 -a -k 0 -S 0 -e | wc -c
> 1080369 a48155fc72d937c4cfdc609e18b470ac
> dd if=/dev/zero bs=1m count=1 | LD_LIBRARY_PATH=../zlib125/lib
> bin/openssl zlib -e | wc -c
> 1039 cfd12ca5dd18c8cd22cc34bb757c8b9f
> dd if=/dev/zero bs=1m count=1 | gzip -n |
> LD_LIBRARY_PATH=../zlib125/lib bin/openssl enc -aes128 -a -k 0 -S 0 -e
> | wc -c
> 1455 08d52ad99d2f6edf344ef735ad62150c
>
>
> There are some documentation bugs...
> enc.1: -z should read: 'was compiled'.
> 'openssl zlib' does not appear to be documented.
>
> Thanks.
>
>
> FreeBSD RELENG_8 i386
> OpenSSL 1.0.0e 6 Sep 2011
> platform: BSD-x86-elf
> options: bn(64,32) rc4(4x,int) des(ptr,risc1,16,long) idea(int)
>  blowfish(idx)
> compiler: gcc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -pthread
>  -D_THREAD_SAFE -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -Wa,--noexecstack
>  -DL_ENDIAN -DTERMIOS -O3 -fomit-frame-pointer -Wall
>  -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_BN_ASM_MONT -DSHA1_ASM
>  -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DRMD160_ASM -DAES_ASM
> ______________________________________________________________________
> OpenSSL Project                                 http://www.openssl.org
> User Support Mailing List                    [hidden email]
> Automated List Manager                           [hidden email]
>
>


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Problems with including zlib

Jakob Bohm-7
In reply to this post by grarpamp
Merry Christmas, and thanks to Michael for pointing out a GNU gcc/ld
specific
option to do this in manually written Makefiles.

My replies below are about how to achieve this without GNU specific options
and without having to edit the Configure and Makefiles.  These answers do
not apply to Windows, OS/2, DOS and other non-POSIX based build
environments.

On 24-12-2011 05:31, grarpamp wrote:
>> 1. Make sure there is a libz.a in /lib or /usr/lib, otherwise you have no
>> static zlib to link in.
> Of course there's an old libz.a there. And it should not matter as
> we're given the --with-zlib arguments to point the build elsewhere
> for those libraries. And as seen in the report, it is following those
> pointers. It's just not using them correctly regarding being told to
> link against libz.a, not libz.so, with the 'zlib' parameter to config.
If you pass Configure and option to look for zlib in an additional
directory, all of these steps apply to that directory too.
>> 2. Temporarily remove or rename the symlink named exactly "libz.so" in
>> /lib, /usr/lib, /usr/local/lib and /zlib125/lib (This ensures it cannot
>> link to the dynamic zlib).
> No, this appears to be to be a ./config build parameter setup
> error. Why should user's break their perfectly sound systems
> in order to work around a bug? If users wanted it to link dynamically
> against libz, they would have specified 'zlib-dynamic' to ./config.
As partly explained by Michael, there is no portable option that
./config could tell the Makefile to pass to the compiler/linker to
get the desired effect.  It simply hasn't got a chance.

Michael's other suggestion to first use the linker to produce and
intermediary .o file with some unresolved externals is not portable
either, as only some linkers have the ability to do that.

However a general way to achieve this on almost any UNIX/POSIX
based system is to artificially present the linker with a scenario
where the linker thinks there is no shared library version of zlib
available, only a PIC-compiled static libz.a, which the linker will
then have to use when creating an OpenSSL shared library.

This is achieved by temporarily hiding the libz.so -> libz.so.N
symlinks that the linker uses, but keeping the
libz.so.N -> libz.so.N.N.N symlinks used by the dynamic linker
on your working computer.

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Problems with including zlib

Michael S. Zick-4
On Sun December 25 2011, [hidden email] wrote:

> Merry Christmas, and thanks to Michael for pointing out a GNU gcc/ld
> specific
> option to do this in manually written Makefiles.
>
> My replies below are about how to achieve this without GNU specific options
> and without having to edit the Configure and Makefiles.  These answers do
> not apply to Windows, OS/2, DOS and other non-POSIX based build
> environments.
>
> On 24-12-2011 05:31, grarpamp wrote:
> >> 1. Make sure there is a libz.a in /lib or /usr/lib, otherwise you have no
> >> static zlib to link in.
> > Of course there's an old libz.a there. And it should not matter as
> > we're given the --with-zlib arguments to point the build elsewhere
> > for those libraries. And as seen in the report, it is following those
> > pointers. It's just not using them correctly regarding being told to
> > link against libz.a, not libz.so, with the 'zlib' parameter to config.
> If you pass Configure and option to look for zlib in an additional
> directory, all of these steps apply to that directory too.
> >> 2. Temporarily remove or rename the symlink named exactly "libz.so" in
> >> /lib, /usr/lib, /usr/local/lib and /zlib125/lib (This ensures it cannot
> >> link to the dynamic zlib).
> > No, this appears to be to be a ./config build parameter setup
> > error. Why should user's break their perfectly sound systems
> > in order to work around a bug? If users wanted it to link dynamically
> > against libz, they would have specified 'zlib-dynamic' to ./config.
> As partly explained by Michael, there is no portable option that
> ./config could tell the Makefile to pass to the compiler/linker to
> get the desired effect.  It simply hasn't got a chance.
>
> Michael's other suggestion to first use the linker to produce and
> intermediary .o file with some unresolved externals is not portable
> either, as only some linkers have the ability to do that.
>
> However a general way to achieve this on almost any UNIX/POSIX
> based system is to artificially present the linker with a scenario
> where the linker thinks there is no shared library version of zlib
> available, only a PIC-compiled static libz.a, which the linker will
> then have to use when creating an OpenSSL shared library.
>
> This is achieved by temporarily hiding the libz.so -> libz.so.N
> symlinks that the linker uses, but keeping the
> libz.so.N -> libz.so.N.N.N symlinks used by the dynamic linker
> on your working computer.
>

*nix base systems (the few I know of anyway) use some variation on
a ld.so.* to do the dynamic linking.
As part of that approach, a ld.so.cache is built on the machine with
the dynamic library links pre-resolved.

You can do whatever you want with the actual links on disk of a
running system, just as long as you don't rebuild the ld.so.cache
until they are back into working condition.

As a general precaution - have the links back to their working
condition before doing: make install
Since the install step of some make files will force a ld.so.cache
rebuild as a 'feature'. ;-)

Mike
> ______________________________________________________________________
> OpenSSL Project                                 http://www.openssl.org
> User Support Mailing List                    [hidden email]
> Automated List Manager                           [hidden email]
>
>


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Problems with including zlib

Jakob Bohm-7
On 12/26/2011 1:31 AM, Michael S. Zick wrote:
On Sun December 25 2011, [hidden email] wrote:
Merry Christmas, and thanks to Michael for pointing out a GNU gcc/ld 
specific
option to do this in manually written Makefiles.

My replies below are about how to achieve this without GNU specific options
and without having to edit the Configure and Makefiles.  These answers do
not apply to Windows, OS/2, DOS and other non-POSIX based build
environments.

On 24-12-2011 05:31, grarpamp wrote:
1. Make sure there is a libz.a in /lib or /usr/lib, otherwise you have no
static zlib to link in.
Of course there's an old libz.a there. And it should not matter as
we're given the --with-zlib arguments to point the build elsewhere
for those libraries. And as seen in the report, it is following those
pointers. It's just not using them correctly regarding being told to
link against libz.a, not libz.so, with the 'zlib' parameter to config.
If you pass Configure and option to look for zlib in an additional
directory, all of these steps apply to that directory too.
2. Temporarily remove or rename the symlink named exactly "libz.so" in
/lib, /usr/lib, /usr/local/lib and /zlib125/lib (This ensures it cannot
link to the dynamic zlib).
No, this appears to be to be a ./config build parameter setup
error. Why should user's break their perfectly sound systems
in order to work around a bug? If users wanted it to link dynamically
against libz, they would have specified 'zlib-dynamic' to ./config.
As partly explained by Michael, there is no portable option that
./config could tell the Makefile to pass to the compiler/linker to
get the desired effect.  It simply hasn't got a chance.

Michael's other suggestion to first use the linker to produce and
intermediary .o file with some unresolved externals is not portable
either, as only some linkers have the ability to do that.

However a general way to achieve this on almost any UNIX/POSIX
based system is to artificially present the linker with a scenario
where the linker thinks there is no shared library version of zlib
available, only a PIC-compiled static libz.a, which the linker will
then have to use when creating an OpenSSL shared library.

This is achieved by temporarily hiding the libz.so -> libz.so.N
symlinks that the linker uses, but keeping the
libz.so.N -> libz.so.N.N.N symlinks used by the dynamic linker
on your working computer.

*nix base systems (the few I know of anyway) use some variation on
a ld.so.* to do the dynamic linking.
As part of that approach, a ld.so.cache is built on the machine with
the dynamic library links pre-resolved.

You can do whatever you want with the actual links on disk of a
running system, just as long as you don't rebuild the ld.so.cache
until they are back into working condition.

As a general precaution - have the links back to their working
condition before doing: make install
Since the install step of some make files will force a ld.so.cache
rebuild as a 'feature'. ;-)

Mike
Thanks, I was approaching this on a more basic level:

On the systems I know about (not many, sorry), ld.so.* and its caching system
looks at the symlinks from libz.so.N to libz.so.N.N.N, which is why those symlinks
tend to be included in the "runtime" shared library install packages for the system.

While /usr/bin/ld (etc.) looks at the libz.so to libz.so.N symlinks to decide which
libz.* to link to when the command line says "-lz", which is why those
symlinks tend to be included in the "development" library install packages.


      
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Problems with including zlib

Michael S. Zick-4
On Tue December 27 2011, Jakob Bohm wrote:

> On 12/26/2011 1:31 AM, Michael S. Zick wrote:
> > On Sun December 25 2011, [hidden email] wrote:
> >> Merry Christmas, and thanks to Michael for pointing out a GNU gcc/ld
> >> specific
> >> option to do this in manually written Makefiles.
> >>
> >> My replies below are about how to achieve this without GNU specific options
> >> and without having to edit the Configure and Makefiles.  These answers do
> >> not apply to Windows, OS/2, DOS and other non-POSIX based build
> >> environments.
> >>
> >> On 24-12-2011 05:31, grarpamp wrote:
> >>>> 1. Make sure there is a libz.a in /lib or /usr/lib, otherwise you have no
> >>>> static zlib to link in.
> >>> Of course there's an old libz.a there. And it should not matter as
> >>> we're given the --with-zlib arguments to point the build elsewhere
> >>> for those libraries. And as seen in the report, it is following those
> >>> pointers. It's just not using them correctly regarding being told to
> >>> link against libz.a, not libz.so, with the 'zlib' parameter to config.
> >> If you pass Configure and option to look for zlib in an additional
> >> directory, all of these steps apply to that directory too.
> >>>> 2. Temporarily remove or rename the symlink named exactly "libz.so" in
> >>>> /lib, /usr/lib, /usr/local/lib and /zlib125/lib (This ensures it cannot
> >>>> link to the dynamic zlib).
> >>> No, this appears to be to be a ./config build parameter setup
> >>> error. Why should user's break their perfectly sound systems
> >>> in order to work around a bug? If users wanted it to link dynamically
> >>> against libz, they would have specified 'zlib-dynamic' to ./config.
> >> As partly explained by Michael, there is no portable option that
> >> ./config could tell the Makefile to pass to the compiler/linker to
> >> get the desired effect.  It simply hasn't got a chance.
> >>
> >> Michael's other suggestion to first use the linker to produce and
> >> intermediary .o file with some unresolved externals is not portable
> >> either, as only some linkers have the ability to do that.
> >>
> >> However a general way to achieve this on almost any UNIX/POSIX
> >> based system is to artificially present the linker with a scenario
> >> where the linker thinks there is no shared library version of zlib
> >> available, only a PIC-compiled static libz.a, which the linker will
> >> then have to use when creating an OpenSSL shared library.
> >>
> >> This is achieved by temporarily hiding the libz.so ->  libz.so.N
> >> symlinks that the linker uses, but keeping the
> >> libz.so.N ->  libz.so.N.N.N symlinks used by the dynamic linker
> >> on your working computer.
> >>
> > *nix base systems (the few I know of anyway) use some variation on
> > a ld.so.* to do the dynamic linking.
> > As part of that approach, a ld.so.cache is built on the machine with
> > the dynamic library links pre-resolved.
> >
> > You can do whatever you want with the actual links on disk of a
> > running system, just as long as you don't rebuild the ld.so.cache
> > until they are back into working condition.
> >
> > As a general precaution - have the links back to their working
> > condition before doing: make install
> > Since the install step of some make files will force a ld.so.cache
> > rebuild as a 'feature'. ;-)
> >
> > Mike
> Thanks, I was approaching this on a more basic level:
>

I thought I was agreeing with you.  ;-)

My only contribution was the warning that some "make install"
package steps might run ldconfig as part of the install.

Dynamic linking __only__ reads ld.so.cache, so as long as
nothing (or no one) runs ldconfig - do as you wish with the links.

> On the systems I know about (not many, sorry), ld.so.* and its caching
> system
> looks at the symlinks from libz.so.N to libz.so.N.N.N, which is why
> those symlinks
> tend to be included in the "runtime" shared library install packages for
> the system.
>
> While /usr/bin/ld (etc.) looks at the libz.so to libz.so.N symlinks to
> decide which
> libz.* to link to when the command line says "-lz", which is why those
> symlinks tend to be included in the "development" library install packages.
>

Ah, another point I was not clear on:

--start-group <others> libz.a <others> --end-group
not use the (also allowed):
--start-group <others> -lz <others> --end-group

The manual only says: "explicit file name(s)" so that might allow
full pathnames also.

The OP's question was: "How to get OpenSSL build to do this?"

I have not looked at the Makefile myself, I thought one
of the Makefile authors would post to this thread.
Maybe it already recognizes a: --libz-static option.

Mike


> >> ______________________________________________________________________
> >> OpenSSL Project                                 http://www.openssl.org
> >> User Support Mailing List                    [hidden email]
> >> Automated List Manager                           [hidden email]
> >>
> >>
> > ______________________________________________________________________
> > OpenSSL Project                                 http://www.openssl.org
> > User Support Mailing List                    [hidden email]
> > Automated List Manager                           [hidden email]
>
>


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Problems with including zlib

Michael S. Zick-4
On Tue December 27 2011, Michael S. Zick wrote:

> On Tue December 27 2011, Jakob Bohm wrote:
> > On 12/26/2011 1:31 AM, Michael S. Zick wrote:
> > > On Sun December 25 2011, [hidden email] wrote:
> > >> Merry Christmas, and thanks to Michael for pointing out a GNU gcc/ld
> > >> specific
> > >> option to do this in manually written Makefiles.
> > >>
> > >> My replies below are about how to achieve this without GNU specific options
> > >> and without having to edit the Configure and Makefiles.  These answers do
> > >> not apply to Windows, OS/2, DOS and other non-POSIX based build
> > >> environments.
> > >>
> > >> On 24-12-2011 05:31, grarpamp wrote:
> > >>>> 1. Make sure there is a libz.a in /lib or /usr/lib, otherwise you have no
> > >>>> static zlib to link in.
> > >>> Of course there's an old libz.a there. And it should not matter as
> > >>> we're given the --with-zlib arguments to point the build elsewhere
> > >>> for those libraries. And as seen in the report, it is following those
> > >>> pointers. It's just not using them correctly regarding being told to
> > >>> link against libz.a, not libz.so, with the 'zlib' parameter to config.
> > >> If you pass Configure and option to look for zlib in an additional
> > >> directory, all of these steps apply to that directory too.
> > >>>> 2. Temporarily remove or rename the symlink named exactly "libz.so" in
> > >>>> /lib, /usr/lib, /usr/local/lib and /zlib125/lib (This ensures it cannot
> > >>>> link to the dynamic zlib).
> > >>> No, this appears to be to be a ./config build parameter setup
> > >>> error. Why should user's break their perfectly sound systems
> > >>> in order to work around a bug? If users wanted it to link dynamically
> > >>> against libz, they would have specified 'zlib-dynamic' to ./config.
> > >> As partly explained by Michael, there is no portable option that
> > >> ./config could tell the Makefile to pass to the compiler/linker to
> > >> get the desired effect.  It simply hasn't got a chance.
> > >>
> > >> Michael's other suggestion to first use the linker to produce and
> > >> intermediary .o file with some unresolved externals is not portable
> > >> either, as only some linkers have the ability to do that.
> > >>
> > >> However a general way to achieve this on almost any UNIX/POSIX
> > >> based system is to artificially present the linker with a scenario
> > >> where the linker thinks there is no shared library version of zlib
> > >> available, only a PIC-compiled static libz.a, which the linker will
> > >> then have to use when creating an OpenSSL shared library.
> > >>
> > >> This is achieved by temporarily hiding the libz.so ->  libz.so.N
> > >> symlinks that the linker uses, but keeping the
> > >> libz.so.N ->  libz.so.N.N.N symlinks used by the dynamic linker
> > >> on your working computer.
> > >>
> > > *nix base systems (the few I know of anyway) use some variation on
> > > a ld.so.* to do the dynamic linking.
> > > As part of that approach, a ld.so.cache is built on the machine with
> > > the dynamic library links pre-resolved.
> > >
> > > You can do whatever you want with the actual links on disk of a
> > > running system, just as long as you don't rebuild the ld.so.cache
> > > until they are back into working condition.
> > >
> > > As a general precaution - have the links back to their working
> > > condition before doing: make install
> > > Since the install step of some make files will force a ld.so.cache
> > > rebuild as a 'feature'. ;-)
> > >
> > > Mike
> > Thanks, I was approaching this on a more basic level:
> >
>
> I thought I was agreeing with you.  ;-)
>
> My only contribution was the warning that some "make install"
> package steps might run ldconfig as part of the install.
>

With libz from the current (1.2.5) source:
./configure --prefix=<whatever> --static

Only builds the static archive but the makefile is broke
for installing only a static archive. (Oops)

./configure --perfix=<whatever>

Builds both the static and the dynamic libraries.
__BUT__
The "make install" also gratuitously runs ldconfig with all
output directed to /dev/null so your only clue is when
your ld.so.cache turns to trash.  ;-)

That can be avoided by:
$ sudo make install LDCONFIG=false

- - - -

And answering my own question, why we haven't heard from
the authors -
They are waiting for one of us to read the directions.  ;-)

It reads to me as if the OpenSSL archive is already setup
to handle this situation, with: no-zlib-dynamic option.

Will try that one myself next.

Mike

>
> Dynamic linking __only__ reads ld.so.cache, so as long as
> nothing (or no one) runs ldconfig - do as you wish with the links.
>
> > On the systems I know about (not many, sorry), ld.so.* and its caching
> > system
> > looks at the symlinks from libz.so.N to libz.so.N.N.N, which is why
> > those symlinks
> > tend to be included in the "runtime" shared library install packages for
> > the system.
> >
> > While /usr/bin/ld (etc.) looks at the libz.so to libz.so.N symlinks to
> > decide which
> > libz.* to link to when the command line says "-lz", which is why those
> > symlinks tend to be included in the "development" library install packages.
> >
>
> Ah, another point I was not clear on:
>
> --start-group <others> libz.a <others> --end-group
> not use the (also allowed):
> --start-group <others> -lz <others> --end-group
>
> The manual only says: "explicit file name(s)" so that might allow
> full pathnames also.
>
> The OP's question was: "How to get OpenSSL build to do this?"
>
> I have not looked at the Makefile myself, I thought one
> of the Makefile authors would post to this thread.
> Maybe it already recognizes a: --libz-static option.
>
> Mike
>
>
> > >> ______________________________________________________________________
> > >> OpenSSL Project                                 http://www.openssl.org
> > >> User Support Mailing List                    [hidden email]
> > >> Automated List Manager                           [hidden email]
> > >>
> > >>
> > > ______________________________________________________________________
> > > OpenSSL Project                                 http://www.openssl.org
> > > User Support Mailing List                    [hidden email]
> > > Automated List Manager                           [hidden email]
> >
> >
>
>
> ______________________________________________________________________
> OpenSSL Project                                 http://www.openssl.org
> User Support Mailing List                    [hidden email]
> Automated List Manager                           [hidden email]
>
>


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Problems with including zlib

Michael S. Zick-4
On Tue December 27 2011, Michael S. Zick wrote:

> On Tue December 27 2011, Michael S. Zick wrote:
> > On Tue December 27 2011, Jakob Bohm wrote:
> > > On 12/26/2011 1:31 AM, Michael S. Zick wrote:
> > > > On Sun December 25 2011, [hidden email] wrote:
> > > >> Merry Christmas, and thanks to Michael for pointing out a GNU gcc/ld
> > > >> specific
> > > >> option to do this in manually written Makefiles.
> > > >>
> > > >> My replies below are about how to achieve this without GNU specific options
> > > >> and without having to edit the Configure and Makefiles.  These answers do
> > > >> not apply to Windows, OS/2, DOS and other non-POSIX based build
> > > >> environments.
> > > >>
> > > >> On 24-12-2011 05:31, grarpamp wrote:
> > > >>>> 1. Make sure there is a libz.a in /lib or /usr/lib, otherwise you have no
> > > >>>> static zlib to link in.
> > > >>> Of course there's an old libz.a there. And it should not matter as
> > > >>> we're given the --with-zlib arguments to point the build elsewhere
> > > >>> for those libraries. And as seen in the report, it is following those
> > > >>> pointers. It's just not using them correctly regarding being told to
> > > >>> link against libz.a, not libz.so, with the 'zlib' parameter to config.
> > > >> If you pass Configure and option to look for zlib in an additional
> > > >> directory, all of these steps apply to that directory too.
> > > >>>> 2. Temporarily remove or rename the symlink named exactly "libz.so" in
> > > >>>> /lib, /usr/lib, /usr/local/lib and /zlib125/lib (This ensures it cannot
> > > >>>> link to the dynamic zlib).
> > > >>> No, this appears to be to be a ./config build parameter setup
> > > >>> error. Why should user's break their perfectly sound systems
> > > >>> in order to work around a bug? If users wanted it to link dynamically
> > > >>> against libz, they would have specified 'zlib-dynamic' to ./config.
> > > >> As partly explained by Michael, there is no portable option that
> > > >> ./config could tell the Makefile to pass to the compiler/linker to
> > > >> get the desired effect.  It simply hasn't got a chance.
> > > >>
> > > >> Michael's other suggestion to first use the linker to produce and
> > > >> intermediary .o file with some unresolved externals is not portable
> > > >> either, as only some linkers have the ability to do that.
> > > >>
> > > >> However a general way to achieve this on almost any UNIX/POSIX
> > > >> based system is to artificially present the linker with a scenario
> > > >> where the linker thinks there is no shared library version of zlib
> > > >> available, only a PIC-compiled static libz.a, which the linker will
> > > >> then have to use when creating an OpenSSL shared library.
> > > >>
> > > >> This is achieved by temporarily hiding the libz.so ->  libz.so.N
> > > >> symlinks that the linker uses, but keeping the
> > > >> libz.so.N ->  libz.so.N.N.N symlinks used by the dynamic linker
> > > >> on your working computer.
> > > >>
> > > > *nix base systems (the few I know of anyway) use some variation on
> > > > a ld.so.* to do the dynamic linking.
> > > > As part of that approach, a ld.so.cache is built on the machine with
> > > > the dynamic library links pre-resolved.
> > > >
> > > > You can do whatever you want with the actual links on disk of a
> > > > running system, just as long as you don't rebuild the ld.so.cache
> > > > until they are back into working condition.
> > > >
> > > > As a general precaution - have the links back to their working
> > > > condition before doing: make install
> > > > Since the install step of some make files will force a ld.so.cache
> > > > rebuild as a 'feature'. ;-)
> > > >
> > > > Mike
> > > Thanks, I was approaching this on a more basic level:
> > >
> >
> > I thought I was agreeing with you.  ;-)
> >
> > My only contribution was the warning that some "make install"
> > package steps might run ldconfig as part of the install.
> >
>
> With libz from the current (1.2.5) source:
> ./configure --prefix=<whatever> --static
>
> Only builds the static archive but the makefile is broke
> for installing only a static archive. (Oops)
>
> ./configure --perfix=<whatever>
>
> Builds both the static and the dynamic libraries.
> __BUT__
> The "make install" also gratuitously runs ldconfig with all
> output directed to /dev/null so your only clue is when
> your ld.so.cache turns to trash.  ;-)
>
> That can be avoided by:
> $ sudo make install LDCONFIG=false
>
> - - - -
>
> And answering my own question, why we haven't heard from
> the authors -
> They are waiting for one of us to read the directions.  ;-)
>
> It reads to me as if the OpenSSL archive is already setup
> to handle this situation, with: no-zlib-dynamic option.
>
> Will try that one myself next.
>

This is what I tried, with the result that the Makefile
and generated opensslconf.h files looked reasonable, but...
(with a bit of \ added for the m.l.:)

$ ./config no-zlib-dynamic zlib \
 -DZLIB_INCLUDE=/opt/zlib125/include \
 -DLIBZLIB=/opt/zlib125/lib \
 shared --prefix=/opt/ssl100e \
 --openssldir=/opt/ssl100e/openssl

$ make test
- - - snip - - -
ALL TESTS SUCCESSFUL.
make[1]: Leaving directory `/Builds/BLDS/openssl-1.0.0e/test'
OPENSSL_CONF=apps/openssl.cnf util/opensslwrap.sh version -a
OpenSSL 1.0.0e 6 Sep 2011
built on: Tue Dec 27 10:58:36 CST 2011
platform: linux-x86_64
options:  bn(64,64) rc4(1x,char) des(idx,cisc,16,int) idea(int) blowfish(idx)
compiler: gcc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT \
 -DDSO_DLFCN -DHAVE_DLFCN_H -DZLIB_INCLUDE=/opt/zlib125/include \
 -DLIBZLIB=/opt/zlib125/lib -Wa,--noexecstack -m64 -DL_ENDIAN -DTERMIO \
 -O3 -Wall -DMD32_REG_T=int -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT \
 -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DWHIRLPOOL_ASM
OPENSSLDIR: "/opt/ssl100e/openssl"

/Builds/BLDS/openssl-1.0.0e/apps$ ldd openssl
    linux-vdso.so.1 =>  (0x00007fff2729d000)
    libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f16e62dc000)
    libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f16e5f16000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f16e5d11000)
    libz.so.1 => /usr/lib/libz.so.1 (0x00007f16e5afa000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f16e5776000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f16e6556000)

Which certainly looks like it is dynamically linked to the system
libz rather than the non-system libz installed in /opt/libz125

Translation: I have to agree with O.P. - It looks broke to me too.  ;-)

Mike

> Mike
> >
> > Dynamic linking __only__ reads ld.so.cache, so as long as
> > nothing (or no one) runs ldconfig - do as you wish with the links.
> >
> > > On the systems I know about (not many, sorry), ld.so.* and its caching
> > > system
> > > looks at the symlinks from libz.so.N to libz.so.N.N.N, which is why
> > > those symlinks
> > > tend to be included in the "runtime" shared library install packages for
> > > the system.
> > >
> > > While /usr/bin/ld (etc.) looks at the libz.so to libz.so.N symlinks to
> > > decide which
> > > libz.* to link to when the command line says "-lz", which is why those
> > > symlinks tend to be included in the "development" library install packages.
> > >
> >
> > Ah, another point I was not clear on:
> >
> > --start-group <others> libz.a <others> --end-group
> > not use the (also allowed):
> > --start-group <others> -lz <others> --end-group
> >
> > The manual only says: "explicit file name(s)" so that might allow
> > full pathnames also.
> >
> > The OP's question was: "How to get OpenSSL build to do this?"
> >
> > I have not looked at the Makefile myself, I thought one
> > of the Makefile authors would post to this thread.
> > Maybe it already recognizes a: --libz-static option.
> >
> > Mike
> >
> >
> > > >> ______________________________________________________________________
> > > >> OpenSSL Project                                 http://www.openssl.org
> > > >> User Support Mailing List                    [hidden email]
> > > >> Automated List Manager                           [hidden email]
> > > >>
> > > >>
> > > > ______________________________________________________________________
> > > > OpenSSL Project                                 http://www.openssl.org
> > > > User Support Mailing List                    [hidden email]
> > > > Automated List Manager                           [hidden email]
> > >
> > >
> >
> >
> > ______________________________________________________________________
> > OpenSSL Project                                 http://www.openssl.org
> > User Support Mailing List                    [hidden email]
> > Automated List Manager                           [hidden email]
> >
> >
>
>
> ______________________________________________________________________
> OpenSSL Project                                 http://www.openssl.org
> User Support Mailing List                    [hidden email]
> Automated List Manager                           [hidden email]
>
>


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Problems with including zlib

grarpamp
> Translation: I have to agree with O.P. - It looks broke to me too.  ;-)

Heh, that's precisely what I said in my report :) The front end
options to do it seem to exist, and they even have some brief
descriptions as such.  They just don't work :)

'zlib' should get us static inclusion.
'zlib-dynamic' should get us dynamic inclusion.

And then there's the failure of zlib to actually compress the data
before encryption, also in my report. (gmail stupidly wrapped the
command lines on that, sorry.)

I'll play around with the build system. And file a bug. Hopefully
something will follow.

Ps: The -D options are not needed since the --with versions of them
work fine. Again, as in my report.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Problems with including zlib

Michael S. Zick-4
On Sun January 1 2012, grarpamp wrote:
> > Translation: I have to agree with O.P. - It looks broke to me too.  ;-)
>
> Heh, that's precisely what I said in my report :) The front end
> options to do it seem to exist, and they even have some brief
> descriptions as such.  They just don't work :)
>
> 'zlib' should get us static inclusion.
>

There is an inconsistency in the output of ./config --help, the
documentation, and what is written in the config/make code.

The true meaning (as executed) is:
zlib / no-zlib (no-zlib is the default) controls the presence
of compression (except for a source file or two that isn't
properly conditioned to match that).

IF compressed streams are enabled, then:
no-zlib-dynamic (the default) should get you the compression
statically included.

> 'zlib-dynamic' should get us dynamic inclusion.
>

And that should get you a reference to the specified dynamic library.
But the build system is always referring to the system installed libraries.

Which is why I showed in my example the use of a library other
than the system installed one.

The real key here is to ignore the "--help" and the documentation
and read the top Makefile __before__ running the config step(s).
 
> And then there's the failure of zlib to actually compress the data
> before encryption, also in my report. (gmail stupidly wrapped the
> command lines on that, sorry.)
>

I was able to get the compression test to fail/pass depending on
how I (mis-)configured/built the package.

But I didn't check if compression is used outside of "make test".
;-)

> I'll play around with the build system. And file a bug. Hopefully
> something will follow.
>
> Ps: The -D options are not needed since the --with versions of them
> work fine. Again, as in my report.
>

I didn't check the --with versions, only found that the -D options are
being ignored although passed with the commands., maybe the --with
versions do work.
Which doesn't mean that the -D options are not also broke.  ;-)

Also, gcc option parsing is becoming more strict.
At the moment (v4.6) it only warns that link options are being passed
when no linking is being done.

I just am not comfortable enough with that perlized makefile build
system to even guess at where that needs to be fixed before a newer
gcc version turns that warning into a failure.

Mike
> ______________________________________________________________________
> OpenSSL Project                                 http://www.openssl.org
> User Support Mailing List                    [hidden email]
> Automated List Manager                           [hidden email]
>
>


______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]