VMS building rework

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

VMS building rework

Richard Levitte - VMS Whacker
Hey,

I've been thinking for the longest time that the VMS building system
for VMS needs to be remade from the ground up.  Time hasn't been with
me, and I've taken a fairly long pause, and I gave up VMS for a while,
and...  I've got plenty of less valid excuses that I'll keep for
myself.

Time to take the bull by the horns, though.

Now that VMS seems to get a revival (if you haven't heard the news?
have a look here: http://vmssoftware.com/ ...) instead of certain
death, it might be time to update ourselves as well.

The current build system is based on the assumption that you have a
the base VMS installation with only a C compiler added.  No MMS, no
MMK, no Perl, no nothing.  The world doesn't look that way and hasn't
for a long time, time to catch up.

I've a fork of OpenSSL on github specifically for this, here:
https://github.com/levitte/openssl

Please join me, let's talk about what's needed, what tools we can
expect people to have available (more than just the basic operating
system and a C compiler), and what we can do, and make the needed
changes.  Feel free to fork my github repo, make changes and propose
them.

(I've cc'd Steven M. Schweda and Zoltan Arpadffy, as they have been
fairly vocal as well as deservingly critical, and thereby hopefully
interested, but others are free to join as well)

Cheers,
Richard

--
Richard Levitte                         [hidden email]
                                        http://richard.levitte.org/

"Life is a tremendous celebration - and I'm invited!"
-- from a friend's blog, translated from Swedish
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: VMS building rework

Zoltan Arpadffy
Hi Richard,

I absolutely welcome the idea, that the build needs to be improved on
OpenVMS.
You have my full support and I'll help as much I can appreciate time-wise.

Currently, I am struggling to include OpenVMS architectures into a jenkins
farm that would at least warn for anomalies as soon they appear.

Thank you for the positive initiative.

Regards,
Z

-----Original Message-----
From: [hidden email] [mailto:[hidden email]]
On Behalf Of Richard Levitte
Sent: den 14 augusti 2014 18:43
To: [hidden email]
Cc: [hidden email]; [hidden email]
Subject: VMS building rework

Hey,

I've been thinking for the longest time that the VMS building system for VMS
needs to be remade from the ground up.  Time hasn't been with me, and I've
taken a fairly long pause, and I gave up VMS for a while, and...  I've got
plenty of less valid excuses that I'll keep for myself.

Time to take the bull by the horns, though.

Now that VMS seems to get a revival (if you haven't heard the news?
have a look here: http://vmssoftware.com/ ...) instead of certain death, it
might be time to update ourselves as well.

The current build system is based on the assumption that you have a the base
VMS installation with only a C compiler added.  No MMS, no MMK, no Perl, no
nothing.  The world doesn't look that way and hasn't for a long time, time
to catch up.

I've a fork of OpenSSL on github specifically for this, here:
https://github.com/levitte/openssl

Please join me, let's talk about what's needed, what tools we can expect
people to have available (more than just the basic operating system and a C
compiler), and what we can do, and make the needed changes.  Feel free to
fork my github repo, make changes and propose them.

(I've cc'd Steven M. Schweda and Zoltan Arpadffy, as they have been fairly
vocal as well as deservingly critical, and thereby hopefully interested, but
others are free to join as well)

Cheers,
Richard

--
Richard Levitte                         [hidden email]
                                        http://richard.levitte.org/

"Life is a tremendous celebration - and I'm invited!"
-- from a friend's blog, translated from Swedish
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: VMS building rework

Steven M. Schweda-2
In reply to this post by Richard Levitte - VMS Whacker
From: Richard Levitte <[hidden email]>

> The current build system is based on the assumption that you have a
> the base VMS installation with only a C compiler added.  No MMS, no
> MMK, no Perl, no nothing.  The world doesn't look that way and hasn't
> for a long time, time to catch up.

   I don't get out much these days, but I suspect that it _does_ look
largely that way in some (many?) places.  If your target audience is
software developers, then you can demand more, but there may be some
people who want to build an OpenSSL kit without having to drag along a
whole, elaborate tool chain.  (If HP fails to provide current stuff, and
there's no other trusted supplier of pre-built stuff, then people may
have little choice but to build their own.)

   That said, a requirement of MMK or MMS should not be too great a
burden.  (In the stuff I work on, I believe that only Zip and UnZip
still offer DCL script builders.  Everything else has only MMK/MMS
builders.)  But I'd think very hard before demanding more than that.

> Please join me, let's talk about what's needed, what tools we can
> expect people to have available (more than just the basic operating
> system and a C compiler), and what we can do, and make the needed
> changes.  Feel free to fork my github repo, make changes and propose
> them.

   The big problem I see with OpenSSL on VMS is that, regardless of the
builders, no one tests this stuff on VMS before a release.  For example,
I just took a quick look at 0.9.8zb, and the build failed because the
builders (ssl/ssl-lib.com) hadn't been updated to include s3_cbc.c, and
crypto/symhacks.h hadn't gotten the new longer-than-31-character names.
I also ran into a problem because I extracted the "tar" kit preserving
symlinks, and the old/lame makevms.com tried to copy the header files
onto actual symlinks in [.include.openssl], which reflected them back to
their source directories.  (Stealing some code from the 1.0.1i
makevms.com helps (crudely) with that.)

   So, assuming that no one will ever reliably test this stuff on VMS
before a release (and experience suggests that this is a pretty safe
bet), I'd worry first about creating some kind of automation which might
keep the source module lists accurate in _any_ kind of builders.  _Then_
I might worry more about getting something better than the current DCL
scripts to be kept accurate.  (The names-longer-than-31-characters
problem means that keeping accurate module lists is not enough, but
adding stuff to crypto/symhacks.h is much easier than chasing through
the code to find the modules which will resolve all the %LINK-I-UDFSYM
complaints.)

   If the Grand Plan solves these problems, too, then that's fine, but
if it doesn't, then I don't see it as a significant improvement.  If
people did actual OpenSSL development on VMS, than a "make"-like
incremental build capability would have more value, but my (uninformed)
impression is that people _use_ OpenSSL on VMS, but don't do much (any?)
actual development of OpenSSL on VMS, so one giant build-all DCL script
is about as good for practically everyone as a fancy MMK/MMS scheme
would be.  (Especially if the DCL script actually worked.)


   Some changed files for 0.9.8zb should be available at:
      http://antinode.info/ftp/openssl/0_9_8zb/

   1.0.1i was not entirely happy, either:

[...]
@@@ mkshared.com
%LINK-W-NUDFSYMS, 1 undefined symbol:
%LINK-I-UDFSYM,         SSL_TEST_FUNCTIONS
%LINK-W-USEUNDEFSYMV, undefined symbol SSL_TEST_FUNCTIONS referenced
  in symbol vector option
[...]

but I haven't investigated that one.  I haven't yet tried 1.0.0n,
either, but complete success there would be amazing, too.

------------------------------------------------------------------------

   Steven M. Schweda               sms@antinode-info
   382 South Warwick Street        (+1) 651-699-9818
   Saint Paul  MN  55105-2547
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: VMS building rework

Steven M. Schweda-2
In reply to this post by Richard Levitte - VMS Whacker
>       http://antinode.info/ftp/openssl/0_9_8zb/
>
>    1.0.1i was not entirely happy, either:
> [...]
> %LINK-I-UDFSYM,         SSL_TEST_FUNCTIONS
> [...]

   Added "ssl_utst" to the module list in ssl/ssl-lib.com:

      http://antinode.info/ftp/openssl/1_0_1i/

> [...]  I haven't yet tried 1.0.0n,
> either, but complete success there would be amazing, too.

   Yup.  Would have been amazing.

   Copied the changes from 1.0.1i for dealing with %CC-W-UNKMSGID
complaints about the MAYLOSEDATA3 warning:

      http://antinode.info/ftp/openssl/1_0_0n/


   Someone who cares about the typography might want to shuffle some
things around a little, but the functionality seems to be ok.  If
there's a great demand for "diff -u" results as opposed to whole files,
I could probably be persuaded, but it might take a few days.

------------------------------------------------------------------------

   Steven M. Schweda               sms@antinode-info
   382 South Warwick Street        (+1) 651-699-9818
   Saint Paul  MN  55105-2547
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: VMS building rework

Richard Levitte - VMS Whacker
In reply to this post by Steven M. Schweda-2
In message <[hidden email]> on Thu, 21 Aug 2014 11:27:15 -0500, "Steven M. Schweda" <[hidden email]> said:

sms> From: Richard Levitte <[hidden email]>
sms>
sms> > The current build system is based on the assumption that you have a
sms> > the base VMS installation with only a C compiler added.  No MMS, no
sms> > MMK, no Perl, no nothing.  The world doesn't look that way and hasn't
sms> > for a long time, time to catch up.
sms>
sms>    I don't get out much these days, but I suspect that it _does_ look
sms> largely that way in some (many?) places.  If your target audience is
sms> software developers, then you can demand more, but there may be some
sms> people who want to build an OpenSSL kit without having to drag along a
sms> whole, elaborate tool chain.  (If HP fails to provide current stuff, and
sms> there's no other trusted supplier of pre-built stuff, then people may
sms> have little choice but to build their own.)

Philosophically speaking, of course it would be great if someone made
an installation kit for the supported VMS platforms.  It's not
necessarily going to be us (I've access to Alphas with quite up to
date VMS and C compiler versions, but that's about all from within the
team), and it's not necessarily going to be HP (or VMS Software Inc).
Someone else might be just as interested in stepping up, who's to
know?  (Vms Software Inc. might actually be interested...)

sms>    That said, a requirement of MMK or MMS should not be too great a
sms> burden.  (In the stuff I work on, I believe that only Zip and UnZip
sms> still offer DCL script builders.  Everything else has only MMK/MMS
sms> builders.)  But I'd think very hard before demanding more than that.

There is some similar package for Perl, isn't there?  Is that very
much of a pain?  The reason I ask is that there are a number of useful
tools written in Perl already, reproducing them in DCL feels like an
exercise in futility, really, and is prone to exactly the kind of
problem we have today with the current VMS build scripts (more on that
below).

sms> > Please join me, let's talk about what's needed, what tools we can
sms> > expect people to have available (more than just the basic operating
sms> > system and a C compiler), and what we can do, and make the needed
sms> > changes.  Feel free to fork my github repo, make changes and propose
sms> > them.
sms>
sms>    The big problem I see with OpenSSL on VMS is that, regardless of the
sms> builders, no one tests this stuff on VMS before a release.

So far, it's been me, and well, you and Zoltan, but that's more as an
afterthought on already released stuff rather than current development.
I took a pause, for several years, for all kinds of personal reasons,
ranging from "tired of this shit" to other reasons that don't have to
do with computing at all.  I'm getting myself back to this as we
speak.

sms>    For example, I just took a quick look at 0.9.8zb, and the
sms> build failed because the builders (ssl/ssl-lib.com) hadn't been
sms> updated to include s3_cbc.c, and crypto/symhacks.h hadn't gotten
sms> the new longer-than-31-character names.

Very true.  I've concentrated on the 1.0.0 and up branches and haven't
caught up on the 0.9.8 branch yet...  But your example is exactly the
kind of thing I want to address with a remake of the VMS build thingy.
More on that below.

sms> I also ran into a problem because I extracted the "tar" kit
sms> preserving symlinks, and the old/lame makevms.com tried to copy
sms> the header files onto actual symlinks in [.include.openssl],
sms> which reflected them back to their source directories.  (Stealing
sms> some code from the 1.0.1i makevms.com helps (crudely) with that.)

I'm behind on things, it seems...  there are symlinks in VMS these
days?  That must have happened while I looked away...  did you extract
the "tar" kit with some other tool than the trusty old VMSTAR, or has
VMSTAR learned to handle symlinks?  I haven't looked at VMSTAR in
ages, but back when I tinkered with it, it didn't have that
capability.

sms>    So, assuming that no one will ever reliably test this stuff on VMS
sms> before a release (and experience suggests that this is a pretty safe
sms> bet), I'd worry first about creating some kind of automation which might
sms> keep the source module lists accurate in _any_ kind of builders.  _Then_
sms> I might worry more about getting something better than the current DCL
sms> scripts to be kept accurate.  (The names-longer-than-31-characters
sms> problem means that keeping accurate module lists is not enough, but
sms> adding stuff to crypto/symhacks.h is much easier than chasing through
sms> the code to find the modules which will resolve all the %LINK-I-UDFSYM
sms> complaints.)

My absolutely first aim with this effort is exactly to not having to
do the silly update of modules in the DCL scripts any more.  The
information is already there in the Makefiles, there's really no good
reason why the information should be duplicated like it currently is.

Quite honestly, the tinkering with modules in the DCL scripts is pure
annoyance and something I'd like to avoid in as near a future as
possible.  "I'm tired of this shit" sure applies in this case.

sms>    If the Grand Plan solves these problems, too, then that's fine, but
sms> if it doesn't, then I don't see it as a significant improvement.  If
sms> people did actual OpenSSL development on VMS, than a "make"-like
sms> incremental build capability would have more value, but my (uninformed)
sms> impression is that people _use_ OpenSSL on VMS, but don't do much (any?)
sms> actual development of OpenSSL on VMS, so one giant build-all DCL script
sms> is about as good for practically everyone as a fancy MMK/MMS scheme
sms> would be.  (Especially if the DCL script actually worked.)

It's true that OpenSSL development doesn't primarly happen on VMS.
Not even for me, I primarly work with a laptop running Linux, and
basically only log into the VMS boxes to do builds (yup, I do that on
a semi regular basis since a few weeks) and fixing the stuff I see
needs being fixed (not a lot except for modules that need added some
times, and the occasional more-than-31-character-symbol thing).

sms>    Some changed files for 0.9.8zb should be available at:
sms>       http://antinode.info/ftp/openssl/0_9_8zb/
sms>
sms>    1.0.1i was not entirely happy, either:
sms>
sms> [...]
sms> @@@ mkshared.com
sms> %LINK-W-NUDFSYMS, 1 undefined symbol:
sms> %LINK-I-UDFSYM,         SSL_TEST_FUNCTIONS
sms> %LINK-W-USEUNDEFSYMV, undefined symbol SSL_TEST_FUNCTIONS referenced
sms>   in symbol vector option
sms> [...]

ssl_utst is missing in [.ssl]ssl-lib.com.  It's a fairly new unit
testing module that exists in the 1.0.1, 1.0.2 and master branches,
but not the older ones.

Btw, the effort that I want to pull through will primarly be directed
at the master branch for the moment being.  Just FYI.

Cheers,
Richard

--
Richard Levitte                         [hidden email]
                                        http://richard.levitte.org/

"Life is a tremendous celebration - and I'm invited!"
-- from a friend's blog, translated from Swedish
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: VMS building rework

Steven M. Schweda-2
In reply to this post by Richard Levitte - VMS Whacker
From: Richard Levitte <[hidden email]>

> There is some similar package for Perl, isn't there?  Is that very
> much of a pain?  [...]

   I expect it to be one more thing which many people won't have.  I
seem to have a Compaq/HP-sourced v5.8.6 ("Compiled at Mar  6 2008
06:07:12"), and newer stuff is probably available, but a quick search
finds much old stuff.  Of course, the tests currently expect perl,
right?  (And what I have seems to work.)  And "bc", if you want to do
the big-number tests without dragging a non-VMS system into the act.
Hence:

      http://antinode.info/dec/sw/bc.html

> I'm behind on things, it seems...  there are symlinks in VMS these
> days?

   Since V8.3, at least (which is at least seven years old), but V8.4
may have fewer bugs.

>   That must have happened while I looked away...  did you extract
> the "tar" kit with some other tool than the trusty old VMSTAR, or has
> VMSTAR learned to handle symlinks?  I haven't looked at VMSTAR in
> ages, but back when I tinkered with it, it didn't have that
> capability.

   Define "trusty old".  I adopted VMSTAR about seven years ago, adding
basic symlink support then.  It's been better than completely lame (I
claim) since about 2011.  That's pretty old, and I tend to trust it.

      http://antinode.info/dec/sw/vmstar.html

   Zip 3.0 and UnZip 6.00 (and up, also not very new) on VMS can handle
them, too.

> My absolutely first aim with this effort is exactly to not having to
> do the silly update of modules in the DCL scripts any more.  The
> information is already there in the Makefiles, there's really no good
> reason why the information should be duplicated like it currently is.

   Except that nothing on VMS can use those Makefiles directly.  So the
question becomes how to derive anything which _is_ useful on VMS from
the normal source kit.  With the auxiliary question being _what_ to
derive (.COM or .MMS, or ???).  It's very clear that getting the module
lists from any files which _are_ maintained would be nice, but what to
do with them is less clear (to me).

   SMS.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: VMS building rework

Richard Levitte - VMS Whacker
In message <[hidden email]> on Thu, 21 Aug 2014 21:32:36 -0500, "Steven M. Schweda" <[hidden email]> said:

sms> From: Richard Levitte <[hidden email]>
sms>
sms> > There is some similar package for Perl, isn't there?  Is that very
sms> > much of a pain?  [...]
sms>
sms>    I expect it to be one more thing which many people won't have.  I
sms> seem to have a Compaq/HP-sourced v5.8.6 ("Compiled at Mar  6 2008
sms> 06:07:12"), and newer stuff is probably available, but a quick search
sms> finds much old stuff.  Of course, the tests currently expect perl,
sms> right?  (And what I have seems to work.)  And "bc", if you want to do
sms> the big-number tests without dragging a non-VMS system into the act.
sms> Hence:
sms>
sms>       http://antinode.info/dec/sw/bc.html

Cool.  And for Perl installation kits, I found this:

        http://sourceforge.net/projects/vmsperlkit/files/

sms> > I'm behind on things, it seems...  there are symlinks in VMS these
sms> > days?
sms>
sms>    Since V8.3, at least (which is at least seven years old), but V8.4
sms> may have fewer bugs.

Ok.  I don't recall that makevms.com has ever been modified to handle
the presence of real symlinks, but my memory may be foggy on that one,
or it happened when I was away.

sms> >   That must have happened while I looked away...  did you extract
sms> > the "tar" kit with some other tool than the trusty old VMSTAR, or has
sms> > VMSTAR learned to handle symlinks?  I haven't looked at VMSTAR in
sms> > ages, but back when I tinkered with it, it didn't have that
sms> > capability.
sms>
sms>    Define "trusty old".  I adopted VMSTAR about seven years ago, adding
sms> basic symlink support then.  It's been better than completely lame (I
sms> claim) since about 2011.  That's pretty old, and I tend to trust it.
sms>
sms>       http://antinode.info/dec/sw/vmstar.html

Well, "trusty old" is before your go at it, so version 3.4-1.  Quite a
lot you've done with it since, cool.

sms>    Zip 3.0 and UnZip 6.00 (and up, also not very new) on VMS can handle
sms> them, too.

Good to know...

sms> > My absolutely first aim with this effort is exactly to not having to
sms> > do the silly update of modules in the DCL scripts any more.  The
sms> > information is already there in the Makefiles, there's really no good
sms> > reason why the information should be duplicated like it currently is.
sms>
sms>    Except that nothing on VMS can use those Makefiles directly.  So the
sms> question becomes how to derive anything which _is_ useful on VMS from
sms> the normal source kit.  With the auxiliary question being _what_ to
sms> derive (.COM or .MMS, or ???).  It's very clear that getting the module
sms> lists from any files which _are_ maintained would be nice, but what to
sms> do with them is less clear (to me).

Well...  those Makefiles still have useful information in them ;-)
You know the variables in the DCL scripts?  Like all the LIB_* in
[.crypto]crypto-lib.com?  They are derived from the LIBOBJ variable in
the corresponding Makefiles.  The variable ENCRYPTION_TYPES (which is
really a very silly name) is basically SDIRS with 'Basic' added in the
front to represent the modules in [.crypto].  This has been done
manually so far, but it isn't very hard to read it directly from the
Makefiles.  It just needs doing.  OR, we could use [.util]files.pl,
which is used to put together all that information into one file
(on Windows, the result is used by util/mk1mf.pl to make one big
makefile...  we could do something similar for VMS, derive a MMS...
quite a lot of the bits and pieces are already there, see)

--
Richard Levitte                         [hidden email]
                                        http://richard.levitte.org/

"Life is a tremendous celebration - and I'm invited!"
-- from a friend's blog, translated from Swedish
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]