Static linking libssl.a and libcrypto.a on Linux x64 fails

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Static linking libssl.a and libcrypto.a on Linux x64 fails

Aijaz Baig
So I posted this question over at stackoverflow (https://stackoverflow.com/questions/58771714/compiling-c-and-c-with-single-makefile) but the gist of it is as follows:

I am trying to statically link libssl.a and libcrypto.a into a static library of my own which I will be using in an application (Linux). So I first create that library (let's call it libAPP.a) and then I use that library in an application. So first things first:

1. when I checkout the contents of that library (nm libAPP.a | less) , almost all (I haven't confirmed every single one but it sure looks that way) SSL related symbols (ex. BIO_new) are left undefined in libAPP.a. Why is that?
2. perhaps this is related to question one above, but when I use libAPP.a in my application, I get a whole bunch of 'undefined reference to' errors for the SSL symbols; they disappear when I pass libSSL.a and libCrypto.a to the linker. may be I need a refresher course on libraries but what is the point of linking statically when I still have to keep those libraries?
3. Even after supplying all these libraries, my linker was complaining about a whole bunch of symbols like dlopen, dlclose etc so after some googling, I supplied -ldl and -lz to the linker. So those linker errors related to dlopen etc went but now it complains that it cannot find libz on the system when it is clearly present.

Can anyone please elaborate on the steps need for a clean static linking of libssl.a into an application? Is it even doable on Linux considering such tight coupling of libc everywhere?? If not what are the alternatives??

--

Best Regards,
Aijaz Baig
Reply | Threaded
Open this post in threaded view
|

RE: Static linking libssl.a and libcrypto.a on Linux x64 fails

Michael Wojcik
> From: openssl-users [mailto:[hidden email]] On Behalf Of Aijaz Baig
> Sent: Wednesday, November 13, 2019 01:45

> I am trying to statically link libssl.a and libcrypto.a into a static library of my own
> which I will be using in an application (Linux).

You can't link anything into a Linux static library, technically.

ELF static libraries, like the older UNIX static libraries they're descended from, are just collections of object files, possibly with some additional metadata. (In BSD 4.x, for example, libraries often had an index member added using the ranlib utility, so that the linker didn't have to search the entire library for each symbol.)

On some platforms, where objects can be relinked, the constituent object files produced by compiling source files are sometimes combined into a single large object. This is most often seen on AIX, which uses IBM's XCOFF object format (an enhanced COFF); XCOFF supports relinking objects, so you can bundle objects up this way and save some time in symbol resolution when you link against the library later. But even on AIX this is commonly seen with dynamic libraries and relatively rare for static ones.

Normally the linker isn't even involved in creating a static library. You compile sources to objects, and then use ar(1) to create the static library. The makefile you posted to StackOverflow doesn't include this step, so it's hard to tell what exactly you're doing.

But in any case, linking a static library against another static library is essentially a no-op.

What you *can* do, if you don't want to have to list your library and the OpenSSL libraries when linking your application, is combine multiple static libraries into a single one - provided the object names don't conflict. This is straightforward:

$ mkdir tmp; cd tmp
$ ar x /path/to/libssl.a
$ ar x /path/to/libcrypto.a
$ cp /path/to/your/objects/*.o .
$ ar c ../your-library.a *.o
$ cd ..
$ rm -rf tmp

(Untested, but see the ar manpage if you run into issues.)

That should create a single archive library containing all the objects from the three input libraries. Again, it relies on there being no filename clashes among the objects; if there are, you'll have to rename some of them.

--
Michael Wojcik
Distinguished Engineer, Micro Focus



Reply | Threaded
Open this post in threaded view
|

Re: Static linking libssl.a and libcrypto.a on Linux x64 fails

OpenSSL - User mailing list
On 13/11/2019 15:23, Michael Wojcik wrote:
>> From: openssl-users [mailto:[hidden email]] On Behalf Of Aijaz Baig
>> Sent: Wednesday, November 13, 2019 01:45
>> I am trying to statically link libssl.a and libcrypto.a into a static library of my own
>> which I will be using in an application (Linux).
> You can't link anything into a Linux static library, technically.
>
> ELF static libraries, like the older UNIX static libraries they're descended from, are just collections of object files, possibly with some additional metadata. (In BSD 4.x, for example, libraries often had an index member added using the ranlib utility, so that the linker didn't have to search the entire library for each symbol.)
Actually, that is also the format and mechanism with Microsoft Win32 tools,
they just use the DOS-like file name "foo.lib" instead of "libfoo.a" to
maintain makefile compatibility with their older Intel OMF-based toolchains.

The object files inside the archive are in COFF format, as they seem to
have used Unix tools to bring up the initial "NT" operating system
internally back before the initial 1993 release.

> On some platforms, where objects can be relinked, the constituent object files produced by compiling source files are sometimes combined into a single large object. This is most often seen on AIX, which uses IBM's XCOFF object format (an enhanced COFF); XCOFF supports relinking objects, so you can bundle objects up this way and save some time in symbol resolution when you link against the library later. But even on AIX this is commonly seen with dynamic libraries and relatively rare for static ones.
>
> Normally the linker isn't even involved in creating a static library. You compile sources to objects, and then use ar(1) to create the static library. The makefile you posted to StackOverflow doesn't include this step, so it's hard to tell what exactly you're doing.
>
> But in any case, linking a static library against another static library is essentially a no-op.
>
> What you *can* do, if you don't want to have to list your library and the OpenSSL libraries when linking your application, is combine multiple static libraries into a single one - provided the object names don't conflict. This is straightforward:
>
> $ mkdir tmp; cd tmp
> $ ar x /path/to/libssl.a
> $ ar x /path/to/libcrypto.a
> $ cp /path/to/your/objects/*.o .
> $ ar c ../your-library.a *.o
> $ cd ..
> $ rm -rf tmp
>
> (Untested, but see the ar manpage if you run into issues.)
>
> That should create a single archive library containing all the objects from the three input libraries. Again, it relies on there being no filename clashes among the objects; if there are, you'll have to rename some of them.
Note: I seem to recall from a long time ago that GNU ar can combine
static libraries directly (without all those temporary file names).

In BinUtils 2.25 this was apparently done by invoking ar in "MRI
compatibility mode" and using the script command "ADDLIB" inside
the provided MRI-style linker script.  For more details see the
"ar scripts" part of the full GNU BinUtils TexInfo manual.

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

Reply | Threaded
Open this post in threaded view
|

Re: Static linking libssl.a and libcrypto.a on Linux x64 fails

Aijaz Baig
In reply to this post by Michael Wojcik
Thank you for the suggestion. Will try that.

Regarding the static library, the term 'linking' I used was more tongue in cheek but nonetheless. However my current concern here is meeting libSSL and libCrypto's dependencies on host libraries on Linux platform. For instance, when I talked about 'linking' errors with respect to symbols like 'dlopen', so as I mentioned, I had to specify '-ldl' and '-lz' to the gcc linker that suggests a dynamic dependency on these platform libraries.

I was trying to understand the components that would be needed to package the whole shebang into an archive which I can later 'just run' on a similar Linux system that has been completely stripped down for purposes of size and speed

Is there a way to do that?

On Wed, Nov 13, 2019 at 8:04 PM Michael Wojcik <[hidden email]> wrote:
> From: openssl-users [mailto:[hidden email]] On Behalf Of Aijaz Baig
> Sent: Wednesday, November 13, 2019 01:45

> I am trying to statically link libssl.a and libcrypto.a into a static library of my own
> which I will be using in an application (Linux).

You can't link anything into a Linux static library, technically.

ELF static libraries, like the older UNIX static libraries they're descended from, are just collections of object files, possibly with some additional metadata. (In BSD 4.x, for example, libraries often had an index member added using the ranlib utility, so that the linker didn't have to search the entire library for each symbol.)

On some platforms, where objects can be relinked, the constituent object files produced by compiling source files are sometimes combined into a single large object. This is most often seen on AIX, which uses IBM's XCOFF object format (an enhanced COFF); XCOFF supports relinking objects, so you can bundle objects up this way and save some time in symbol resolution when you link against the library later. But even on AIX this is commonly seen with dynamic libraries and relatively rare for static ones.

Normally the linker isn't even involved in creating a static library. You compile sources to objects, and then use ar(1) to create the static library. The makefile you posted to StackOverflow doesn't include this step, so it's hard to tell what exactly you're doing.

But in any case, linking a static library against another static library is essentially a no-op.

What you *can* do, if you don't want to have to list your library and the OpenSSL libraries when linking your application, is combine multiple static libraries into a single one - provided the object names don't conflict. This is straightforward:

$ mkdir tmp; cd tmp
$ ar x /path/to/libssl.a
$ ar x /path/to/libcrypto.a
$ cp /path/to/your/objects/*.o .
$ ar c ../your-library.a *.o
$ cd ..
$ rm -rf tmp

(Untested, but see the ar manpage if you run into issues.)

That should create a single archive library containing all the objects from the three input libraries. Again, it relies on there being no filename clashes among the objects; if there are, you'll have to rename some of them.

--
Michael Wojcik
Distinguished Engineer, Micro Focus





--

Best Regards,
Aijaz Baig
Reply | Threaded
Open this post in threaded view
|

Re: Static linking libssl.a and libcrypto.a on Linux x64 fails

Aijaz Baig
> Actually, that is also the format and mechanism with Microsoft Win32 tools, they just use the DOS-like file name "foo.lib" instead of "libfoo.a" to maintain makefile compatibility with their older Intel OMF-based toolchains. The object files inside > the archive are in COFF format, as they seem to have used Unix tools to bring up the initial "NT" operating system internally back before the initial 1993 release. On some platforms, where objects can be relinked, the constituent object files
> produced by compiling source files are sometimes combined into a single large object. This is most often seen on AIX, which uses IBM's XCOFF object format (an enhanced COFF); XCOFF supports relinking objects, so you can bundle
> objects up this way and save some time in symbol resolution when you link against the library later. But even on AIX this is commonly seen with dynamic libraries and relatively rare for static ones.
As I mentioned earlier in my reply to Micheal, I would want to create
a single archive that is a self contained 'package' (if you will) that
alone would suffice for any application that requires openssl
functionality. I've created certain wrappers over the original SSL
APIs and the object files that contain this functionality (let's call
this collectively as libapp.a) would be 'bundled' along with libssl.a
and libcrypto.a. So as mike suggested, I'll combine the whole into a
new shiny 'libappssl.a' and all that the application has to be is to
pass this to the platform linker (which BTW is posix compliant) and
that's all he requires.

> Normally the linker isn't even involved in creating a static library. You compile sources to objects, and then use ar(1) to create the static library. The makefile you posted to StackOverflow doesn't include this step, so it's hard to tell what exactly you're doing.
That StackOverflow question contains yet another link that points to
another question (which I had asked about creating the library). The
Makefile for that library contains this at the final linking (or
archiving) stage:

libAPP.a: $(obj)
    @ar rcs $@ $^
    #$(LINK.c) -shared $^ -o $@

I was creating a shared library earlier but now I am creating a static
archive from the object files.

>  Note: I seem to recall from a long time ago that GNU ar can combine static libraries directly (without all those temporary file names).
>  In BinUtils 2.25 this was apparently done by invoking ar in "MRI compatibility mode" and using the script command "ADDLIB" inside the provided MRI-style linker script.
>  For more details see the "ar scripts" part of the full GNU BinUtils TexInfo manual.
> Enjoy
> Jakob

Well, I'll take a look however Mike's method is pretty easy enough to
follow. However, what I am concerned with is, which other libraries do
I need to archive (except libc which seems to be obvious I assume) to
allow libssl and libcrypto to run on a stripped down system that
resembles Linux (aka is POSIX compliant)?

Regards,
Aijaz


On Thu, Nov 14, 2019 at 6:28 AM Aijaz Baig <[hidden email]> wrote:

>
> Thank you for the suggestion. Will try that.
>
> Regarding the static library, the term 'linking' I used was more tongue in cheek but nonetheless. However my current concern here is meeting libSSL and libCrypto's dependencies on host libraries on Linux platform. For instance, when I talked about 'linking' errors with respect to symbols like 'dlopen', so as I mentioned, I had to specify '-ldl' and '-lz' to the gcc linker that suggests a dynamic dependency on these platform libraries.
>
> I was trying to understand the components that would be needed to package the whole shebang into an archive which I can later 'just run' on a similar Linux system that has been completely stripped down for purposes of size and speed
>
> Is there a way to do that?
>
> On Wed, Nov 13, 2019 at 8:04 PM Michael Wojcik <[hidden email]> wrote:
>>
>> > From: openssl-users [mailto:[hidden email]] On Behalf Of Aijaz Baig
>> > Sent: Wednesday, November 13, 2019 01:45
>>
>> > I am trying to statically link libssl.a and libcrypto.a into a static library of my own
>> > which I will be using in an application (Linux).
>>
>> You can't link anything into a Linux static library, technically.
>>
>> ELF static libraries, like the older UNIX static libraries they're descended from, are just collections of object files, possibly with some additional metadata. (In BSD 4.x, for example, libraries often had an index member added using the ranlib utility, so that the linker didn't have to search the entire library for each symbol.)
>>
>> On some platforms, where objects can be relinked, the constituent object files produced by compiling source files are sometimes combined into a single large object. This is most often seen on AIX, which uses IBM's XCOFF object format (an enhanced COFF); XCOFF supports relinking objects, so you can bundle objects up this way and save some time in symbol resolution when you link against the library later. But even on AIX this is commonly seen with dynamic libraries and relatively rare for static ones.
>>
>> Normally the linker isn't even involved in creating a static library. You compile sources to objects, and then use ar(1) to create the static library. The makefile you posted to StackOverflow doesn't include this step, so it's hard to tell what exactly you're doing.
>>
>> But in any case, linking a static library against another static library is essentially a no-op.
>>
>> What you *can* do, if you don't want to have to list your library and the OpenSSL libraries when linking your application, is combine multiple static libraries into a single one - provided the object names don't conflict. This is straightforward:
>>
>> $ mkdir tmp; cd tmp
>> $ ar x /path/to/libssl.a
>> $ ar x /path/to/libcrypto.a
>> $ cp /path/to/your/objects/*.o .
>> $ ar c ../your-library.a *.o
>> $ cd ..
>> $ rm -rf tmp
>>
>> (Untested, but see the ar manpage if you run into issues.)
>>
>> That should create a single archive library containing all the objects from the three input libraries. Again, it relies on there being no filename clashes among the objects; if there are, you'll have to rename some of them.
>>
>> --
>> Michael Wojcik
>> Distinguished Engineer, Micro Focus
>>
>>
>>
>
>
> --
>
> Best Regards,
> Aijaz Baig



--

Best Regards,
Aijaz Baig
Reply | Threaded
Open this post in threaded view
|

RE: Static linking libssl.a and libcrypto.a on Linux x64 fails

Michael Wojcik
In reply to this post by Aijaz Baig
> From: Aijaz Baig [mailto:[hidden email]]
> Sent: Wednesday, November 13, 2019 19:58

> However my current concern here is meeting libSSL and libCrypto's dependencies on
> host libraries on Linux platform. For instance, when I talked about 'linking' errors
> with respect to symbols like 'dlopen', so as I mentioned, I had to specify '-ldl'
> and '-lz' to the gcc linker that suggests a dynamic dependency on these platform
> libraries.

The -l flag can specify static as well as dynamic libraries, so its use doesn't in itself imply dynamic linking.

However, if you need libdl, then something in the application or one or more of the libraries contains references to runtime dynamic linking functions, which suggests the application may be doing explicit dynamic linking. And if memory serves, libdl is only available as a shared object, not as a static library.

> I was trying to understand the components that would be needed to package the whole
> shebang into an archive which I can later 'just run' on a similar Linux system that
> has been completely stripped down for purposes of size and speed

Ultimately, you can only statically link objects that you have available for static linking. If you only have the C runtime (glibc) or libz or whatever as a dynamic library, then dynamic linking is your only option for them.

Also, static linking generally increases the size requirements, since you duplicate object code into applications rather than having it in a single shared object.

--
Michael Wojcik
Distinguished Engineer, Micro Focus



Reply | Threaded
Open this post in threaded view
|

RE: Static linking libssl.a and libcrypto.a on Linux x64 fails

Floodeenjr, Thomas
If you want to link statically, when dynamic libraries are also available, you need to tell the linker that you want to use static libraries, otherwise it will always assume dynamic

LINK_LIBS = -Wl,-Bstatic -lstaticlibs -lcrypto -lssl -lz -Wl,-Bdynamic -ldynamiclibs

-----Original Message-----
From: openssl-users [mailto:[hidden email]] On Behalf Of Michael Wojcik
Sent: Monday, November 18, 2019 5:29 AM
To: [hidden email]
Subject: RE: Static linking libssl.a and libcrypto.a on Linux x64 fails

> From: Aijaz Baig [mailto:[hidden email]]
> Sent: Wednesday, November 13, 2019 19:58

> However my current concern here is meeting libSSL and libCrypto's
> dependencies on host libraries on Linux platform. For instance, when I
> talked about 'linking' errors with respect to symbols like 'dlopen', so as I mentioned, I had to specify '-ldl'
> and '-lz' to the gcc linker that suggests a dynamic dependency on
> these platform libraries.

The -l flag can specify static as well as dynamic libraries, so its use doesn't in itself imply dynamic linking.

However, if you need libdl, then something in the application or one or more of the libraries contains references to runtime dynamic linking functions, which suggests the application may be doing explicit dynamic linking. And if memory serves, libdl is only available as a shared object, not as a static library.

> I was trying to understand the components that would be needed to
> package the whole shebang into an archive which I can later 'just run'
> on a similar Linux system that has been completely stripped down for
> purposes of size and speed

Ultimately, you can only statically link objects that you have available for static linking. If you only have the C runtime (glibc) or libz or whatever as a dynamic library, then dynamic linking is your only option for them.

Also, static linking generally increases the size requirements, since you duplicate object code into applications rather than having it in a single shared object.

--
Michael Wojcik
Distinguished Engineer, Micro Focus