Building recent versions on Windows with VS 2013 and MASM

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Building recent versions on Windows with VS 2013 and MASM

Steven Kneizys
I wanted to be able to build 32 and 64 bit versions of recent OpenSSL
1.0.x production and beta/snapshots using Visual Studio Express
edition.  The new Visual Studios include ml/ml64, and while NASM is
great sometimes it is nice to have the option of MASM as well.

I was trying to do things with the built-in translator and other
support code but decided to get the biggest initial return on invested
time that I would do the translation work out the build itself at
first, then integrate later, especially since 1.0.2 is already in beta
 and 1.0.1 is winding down.  The best way I could see to do that is to
use what is already standard, the NASM flavour (dialect), and go to
MASM from there.  So the strategy was to make the OpenSSL build think
it was going to send to NASM, but really I would translate to MASM.
So I created my experimental "fake nasm".

It is available here if anybody wants to play with it:
http://erpdata.com/fakenasm

If run, it will sort of say it is "nasm":
E:\usr_local_src\>nasm -v
NASM version 2.11.02 compiled on Feb 19 2014 (fakeout ver 1.0 03-MAR-2014)

Actually, it is a tiny C program that forwards the request to a ".bat"
file, and a perl script, which then does a translation of the dialect,
creating an intermediate code file ending in ".XMASM" (meaning
translated to MASM) which is then fed to ml/ml64.  Here is a sample
from a build:


E:\usr_local_src\openssl-1.0.1f_64>nmake -f ms\ntdll.mak

Microsoft (R) Program Maintenance Utility Version 12.00.21005.1
Copyright (C) Microsoft Corporation.  All rights reserved.

Building OpenSSL
        set ASM=nasm -f win64 -DNEAR -Ox -g
        perl crypto\x86_64cpuid.pl tmp32dll\x86_64cpuid.asm
        nasm -f win64 -DNEAR -Ox -g -o tmp32dll\x86_64cpuid.obj
tmp32dll\x86_64cpuid.asm
NASM version 2.11.02 compiled on Feb 19 2014 (fakeout ver 1.0 03-MAR-2014)
Assembling tmp32dll\x86_64cpuid.asm to tmp32dll\x86_64cpuid.obj (via
tmp32dll\x86_64cpuid.asm.XMASM)
ml64 /c /Cp /Cx /Zi /Fotmp32dll\x86_64cpuid.obj tmp32dll\x86_64cpuid.asm.XMASM
Microsoft (R) Macro Assembler (x64) Version 12.00.21005.1
Copyright (C) Microsoft Corporation.  All rights reserved.

 Assembling: tmp32dll\x86_64cpuid.asm.XMASM

It tells the OpenSSL build it is "nasm", but then passes the request
on to ml or ml64.  By keep the translation out of the OpenSSL tree at
first I was able to apply it to a wide variety of OpenSSL snapshots
and releases.  I was just able to build 32 and 64 bit version of:

  1.0.1f
  openssl-1.0.2-stable-SNAP-20140303
  openssl-1.0.2-stable-SNAP-20140305

All you have to do is put the "fake nasm" in your path instead of the
real one, then build as usual using Visual Studio 2013 x86 or x64
command tools prompt, and it will have ml/ml64 do the assembly parts.

While I am able to build those versions and they pass the "ms\test", I
do not have a CPU with Intel® AVX, Intel® AVX2, Intel® AVX-512 ...
anybody want to test it on those processors?  :-)

Eventually I hope to put what I am learning into the base openssl so
maybe ml/ml64 might be supported again.

Thanks,

Steve...

On Tue, Mar 4, 2014 at 9:11 AM, Steven Kneizys <[hidden email]> wrote:

> I'm working on a little fun thing to see if I can get Visual Studio to
> compile and assemble (64 bit)  the 1.0.2 snapshots ... more on that
> later on in the week.  I am trying to figure out what the MASM 12.x
> assembler wants that is different from what NASM takes right now.  I
> know things are going through the translator on OpenSSL, but the
> current MASM-generated code is not working, and I know it is
> officially unsupported.  But I am looking at it :-)
>
> On Fri, Feb 28, 2014 at 5:00 PM, Andy Polyakov <[hidden email]> wrote:
>>> I remembered seeing XMMWORD being taken into consideration if the
>>> opcode args were of the xmm variety, so out of curiosity I added a
>>> little code to my previous patch to modify the opcode's third arg as
>>> well, if present:
>>
>>
>> It will be looked into as time permits, but not for 1.0.2 release. If
>> anything one should also look into AVX2 and AVX512 support
>>
>> ______________________________________________________________________
>> OpenSSL Project                                 http://www.openssl.org
>> Development Mailing List                       [hidden email]
>> Automated List Manager                           [hidden email]
>
>
>
> --
> Steve Kneizys
> Senior Business Process Engineer
> Voice: (610) 256-1396  [For Emergency Service (888)864-3282]
> Ferrilli Information Group -- Quality Service and Solutions for Higher Education
> web: http://www.ferrilli.com/
>
> Making you a success while exceeding your expectations.



--
Steve Kneizys
Senior Business Process Engineer
Voice: (610) 256-1396  [For Emergency Service (888)864-3282]
Ferrilli Information Group -- Quality Service and Solutions for Higher Education
web: http://www.ferrilli.com/

Making you a success while exceeding your expectations.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Building recent versions on Windows with VS 2013 and MASM

Steven Kneizys
Quick question on 1.0.2 and Visual Studio win64 builds ... is ml64
supported or just nasm?

Thanks,

Steve...

On Wed, Mar 5, 2014 at 2:00 PM, Steven Kneizys <[hidden email]> wrote:

> I wanted to be able to build 32 and 64 bit versions of recent OpenSSL
> 1.0.x production and beta/snapshots using Visual Studio Express
> edition.  The new Visual Studios include ml/ml64, and while NASM is
> great sometimes it is nice to have the option of MASM as well.
>
> I was trying to do things with the built-in translator and other
> support code but decided to get the biggest initial return on invested
> time that I would do the translation work out the build itself at
> first, then integrate later, especially since 1.0.2 is already in beta
>  and 1.0.1 is winding down.  The best way I could see to do that is to
> use what is already standard, the NASM flavour (dialect), and go to
> MASM from there.  So the strategy was to make the OpenSSL build think
> it was going to send to NASM, but really I would translate to MASM.
> So I created my experimental "fake nasm".
>
> It is available here if anybody wants to play with it:
> http://erpdata.com/fakenasm
>
> If run, it will sort of say it is "nasm":
> E:\usr_local_src\>nasm -v
> NASM version 2.11.02 compiled on Feb 19 2014 (fakeout ver 1.0 03-MAR-2014)
>
> Actually, it is a tiny C program that forwards the request to a ".bat"
> file, and a perl script, which then does a translation of the dialect,
> creating an intermediate code file ending in ".XMASM" (meaning
> translated to MASM) which is then fed to ml/ml64.  Here is a sample
> from a build:
>
>
> E:\usr_local_src\openssl-1.0.1f_64>nmake -f ms\ntdll.mak
>
> Microsoft (R) Program Maintenance Utility Version 12.00.21005.1
> Copyright (C) Microsoft Corporation.  All rights reserved.
>
> Building OpenSSL
>         set ASM=nasm -f win64 -DNEAR -Ox -g
>         perl crypto\x86_64cpuid.pl tmp32dll\x86_64cpuid.asm
>         nasm -f win64 -DNEAR -Ox -g -o tmp32dll\x86_64cpuid.obj
> tmp32dll\x86_64cpuid.asm
> NASM version 2.11.02 compiled on Feb 19 2014 (fakeout ver 1.0 03-MAR-2014)
> Assembling tmp32dll\x86_64cpuid.asm to tmp32dll\x86_64cpuid.obj (via
> tmp32dll\x86_64cpuid.asm.XMASM)
> ml64 /c /Cp /Cx /Zi /Fotmp32dll\x86_64cpuid.obj tmp32dll\x86_64cpuid.asm.XMASM
> Microsoft (R) Macro Assembler (x64) Version 12.00.21005.1
> Copyright (C) Microsoft Corporation.  All rights reserved.
>
>  Assembling: tmp32dll\x86_64cpuid.asm.XMASM
>
> It tells the OpenSSL build it is "nasm", but then passes the request
> on to ml or ml64.  By keep the translation out of the OpenSSL tree at
> first I was able to apply it to a wide variety of OpenSSL snapshots
> and releases.  I was just able to build 32 and 64 bit version of:
>
>   1.0.1f
>   openssl-1.0.2-stable-SNAP-20140303
>   openssl-1.0.2-stable-SNAP-20140305
>
> All you have to do is put the "fake nasm" in your path instead of the
> real one, then build as usual using Visual Studio 2013 x86 or x64
> command tools prompt, and it will have ml/ml64 do the assembly parts.
>
> While I am able to build those versions and they pass the "ms\test", I
> do not have a CPU with Intel® AVX, Intel® AVX2, Intel® AVX-512 ...
> anybody want to test it on those processors?  :-)
>
> Eventually I hope to put what I am learning into the base openssl so
> maybe ml/ml64 might be supported again.
>
> Thanks,
>
> Steve...
>
> On Tue, Mar 4, 2014 at 9:11 AM, Steven Kneizys <[hidden email]> wrote:
>> I'm working on a little fun thing to see if I can get Visual Studio to
>> compile and assemble (64 bit)  the 1.0.2 snapshots ... more on that
>> later on in the week.  I am trying to figure out what the MASM 12.x
>> assembler wants that is different from what NASM takes right now.  I
>> know things are going through the translator on OpenSSL, but the
>> current MASM-generated code is not working, and I know it is
>> officially unsupported.  But I am looking at it :-)
>>
>> On Fri, Feb 28, 2014 at 5:00 PM, Andy Polyakov <[hidden email]> wrote:
>>>> I remembered seeing XMMWORD being taken into consideration if the
>>>> opcode args were of the xmm variety, so out of curiosity I added a
>>>> little code to my previous patch to modify the opcode's third arg as
>>>> well, if present:
>>>
>>>
>>> It will be looked into as time permits, but not for 1.0.2 release. If
>>> anything one should also look into AVX2 and AVX512 support
>>>
>>> ______________________________________________________________________
>>> OpenSSL Project                                 http://www.openssl.org
>>> Development Mailing List                       [hidden email]
>>> Automated List Manager                           [hidden email]
>>
>>
>>
>> --
>> Steve Kneizys
>> Senior Business Process Engineer
>> Voice: (610) 256-1396  [For Emergency Service (888)864-3282]
>> Ferrilli Information Group -- Quality Service and Solutions for Higher Education
>> web: http://www.ferrilli.com/
>>
>> Making you a success while exceeding your expectations.
>
>
>
> --
> Steve Kneizys
> Senior Business Process Engineer
> Voice: (610) 256-1396  [For Emergency Service (888)864-3282]
> Ferrilli Information Group -- Quality Service and Solutions for Higher Education
> web: http://www.ferrilli.com/
>
> Making you a success while exceeding your expectations.



--
Steve Kneizys
Senior Business Process Engineer
Voice: (610) 256-1396  [For Emergency Service (888)864-3282]
Ferrilli Information Group -- Quality Service and Solutions for Higher Education
web: http://www.ferrilli.com/

Making you a success while exceeding your expectations.
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Building recent versions on Windows with VS 2013 and MASM

Andy Polyakov-2
> Quick question on 1.0.2 and Visual Studio win64 builds ... is ml64
> supported or just nasm?

Quick answer: quantum mechanics. Longer answer: state is unknown till
it's measured. Long answer: reports are not rejected, but
when-in-doubt-use-nasm principle applies, and reaction to reports can as
well be avoiding problem.

As for supporting MASM in more general terms. You have to do better than
"it's *nice* to have MASM". You have to show why is it so, what is it
MASM offers that NASM doesn't. Even if there are some/any, they would be
weighed against advantages NASM has to offer (such as unconditional
availability for immediate download, possibility to affect the
functionality, cross-platform support). And even then faking nasm is
hardly appropriate way to solve the problem.

>> While I am able to build those versions and they pass the "ms\test", I
>> do not have a CPU with Intel® AVX, Intel® AVX2, Intel® AVX-512 ...
>> anybody want to test it on those processors?  :-)

http://software.intel.com/en-us/articles/intel-software-development-emulator/

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Building recent versions on Windows with VS 2013 and MASM

Steven Kneizys
>> Quick question on 1.0.2 and Visual Studio win64 builds ... is ml64
>> supported or just nasm?

>Quick answer: quantum mechanics. Longer answer: state is unknown till
>it's measured. Long answer: reports are not rejected, but
>when-in-doubt-use-nasm principle applies, and reaction to reports can as
>well be avoiding problem.

Well then I will definitely send in my more complete analysis/proposed fixes.

>As for supporting MASM in more general terms. You have to do better than
>"it's *nice* to have MASM". You have to show why is it so, what is it
>MASM offers that NASM doesn't. Even if there are some/any, they would be
>weighed against advantages NASM has to offer (such as unconditional
>availability for immediate download, possibility to affect the
>functionality, cross-platform support). And even then faking nasm is
>hardly appropriate way to solve the problem.

OpenSSL is security software.  I have clients who have auditors that make 
demands sometimes.  I just helped a client get on 1.0.1f because of an 
issue with 1.0.1e that a government auditor noted.  What if the auditors 
say "and why is it that it cannot be built for the Microsoft platform with 
the Microsoft platform tools?".  It would be SO nice to not have to have that 
conversation, ever.

And if my analysis is correct, OpenSSL is way over 99% of the way there, all
the hooks are still there, it is just a little tiny bit of tweaking.  I feel like we'd
be making a mistake to let this opportunity slip away, now that the native tool
MASM is installed even on the free "Express Edition" of Visual Studio 2012/2013
in both 32 and 64 bit.  No extra download.  Platform vendor tool.  Supportable by 
vendor.  Free.  Automatic software update mechanism.

And faking nasm, well, it is a nasm emulator ... but as the name denotes, I
really didn't want to call it a "real" tool exactly.  Just one for development
analysis work if we can get OpenSSL to use ml/ml64 natively.  But if not,
I can build it anytime I want to and have control over new features :-)

>>> While I am able to build those versions and they pass the "ms\test", I
>>> do not have a CPU with Intel® AVX, Intel® AVX2, Intel® AVX-512 ...
>>> anybody want to test it on those processors?  :-)


Yes, I tried that, testing / dev tools like this are the inspiration for the
fake nasm.  Sort of a "just inject this tool in the middle and you can learn
some things" so to speak.

On the SDE, I was seeing the AVX/AVX2 from my lil CPUID program (not the greatest
and the Intel one doesn't seem to work with the Intel SDE!)  AVX512 support didn't
seem to be turned on in the emulator, I tried the edition released yesterday and the 
previous one as  well.  I tried the command line switches but couldn't figure that part out.

When I ran my fake-nasm generated 1.0.2 snapshot 64-bit OpenSSL code 
through the SDE (while AVX/AVX2 seemed to be on) it said all tests passed. 

But I think I'd like to see it on something "real"!


On Fri, Mar 7, 2014 at 4:19 AM, Andy Polyakov <[hidden email]> wrote:
> Quick question on 1.0.2 and Visual Studio win64 builds ... is ml64
> supported or just nasm?

Quick answer: quantum mechanics. Longer answer: state is unknown till
it's measured. Long answer: reports are not rejected, but
when-in-doubt-use-nasm principle applies, and reaction to reports can as
well be avoiding problem.

As for supporting MASM in more general terms. You have to do better than
"it's *nice* to have MASM". You have to show why is it so, what is it
MASM offers that NASM doesn't. Even if there are some/any, they would be
weighed against advantages NASM has to offer (such as unconditional
availability for immediate download, possibility to affect the
functionality, cross-platform support). And even then faking nasm is
hardly appropriate way to solve the problem.

>> While I am able to build those versions and they pass the "ms\test", I
>> do not have a CPU with Intel® AVX, Intel® AVX2, Intel® AVX-512 ...
>> anybody want to test it on those processors?  :-)

http://software.intel.com/en-us/articles/intel-software-development-emulator/

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
Development Mailing List                       [hidden email]
Automated List Manager                           [hidden email]



--
Steve Kneizys
Senior Business Process Engineer
Voice: (610) 256-1396  [For Emergency Service (888)864-3282]
Ferrilli Information Group -- Quality Service and Solutions for Higher Education
web: http://www.ferrilli.com/

Making you a success while exceeding your expectations.
Reply | Threaded
Open this post in threaded view
|

Re: Building recent versions on Windows with VS 2013 and MASM

Steven Kneizys
Lots of things got in the way since March, but I've had a chance to re-look at this issue.  I had gotten as far as creating some patches but had not fully vetted them -- just this week I tried again and found that for the 9/10 snapshot generates with VS 2013 Express Edition MASM 64 bit without any changes!  Wonderful!

For the 32 bit, it still had small issues, but all that was required to get it to generate was a small variation on what is already there.  In case there was interest in having that work I am posting what I found.  I added a new file (new incarnation of a very old file I think) ms\do_masm.bat:
perl util\mkfiles.pl >MINFO
perl util\mk1mf.pl VC-WIN32 >ms\nt.mak
perl util\mk1mf.pl dll VC-WIN32 >ms\ntdll.mak
perl util\mk1mf.pl BC-NT >ms\bcb.mak

perl util\mkdef.pl 32 libeay > ms\libeay32.def
perl util\mkdef.pl 32 ssleay > ms\ssleay32.def

I then slightly modified the perlasm/x86masm.pl file to slightly change the handling of XMM pointer references, and to make sure the DB/DW/DD statements generated don't exceed MASM's buffer length and VOILA!  The MASM assembler works for 32 bit as well!  Below are those changes.  I have not checked with "ms\test.bat" on an advanced processor yet.

Thanks,

Steve...

--- openssl-1.0.2-stable-SNAP-20140910/crypto/perlasm/x86masm.pl 2013-05-19 16:00:12.000000000 -0400
+++ openssl-1.0.2-stable-SNAP-20140910_MASM/crypto/perlasm/x86masm.pl 2014-09-11 11:08:37.998095700 -0400
@@ -20,8 +20,14 @@
     { $opcode="mov"; }
     elsif ($opcode !~ /movq/)
     { # fix xmm references
- $arg[0] =~ s/\b[A-Z]+WORD\s+PTR/XMMWORD PTR/i if ($arg[1]=~/\bxmm[0-7]\b/i);
- $arg[1] =~ s/\b[A-Z]+WORD\s+PTR/XMMWORD PTR/i if ($arg[0]=~/\bxmm[0-7]\b/i);
+        my $ptrsize = 'XMMWORD';
+        if ($opcode eq 'movd') {$ptrsize = 'DWORD';}
+ $arg[0] =~ s/\b[A-Z]+WORD\s+PTR/$ptrsize PTR/i if ($arg[1]=~/\bxmm[0-7]\b/i);
+ $arg[1] =~ s/\b[A-Z]+WORD\s+PTR/$ptrsize PTR/i if ($arg[0]=~/\bxmm[0-7]\b/i);
+ if (defined($arg[2])) {
+ $arg[2] =~ s/\b[A-Z]+WORD\s+PTR/$ptrsize PTR/i if ($arg[1]=~/\bxmm[0-7]\b/i);
+ $arg[2] =~ s/\b[A-Z]+WORD\s+PTR/$ptrsize PTR/i if ($arg[0]=~/\bxmm[0-7]\b/i);
+ }
     }
 
     &::emit($opcode,@arg);
@@ -160,13 +166,13 @@
 {   push(@out,"PUBLIC\t".&::LABEL($_[0],$nmdecor.$_[0])."\n");   }
 
 sub ::data_byte
-{   push(@out,("DB\t").join(',',@_)."\n"); }
+{   push @out, (("DB\t").join(',',splice(@_, 0, 16))."\n") while @_; }
 
 sub ::data_short
-{   push(@out,("DW\t").join(',',@_)."\n"); }
+{   push @out, (("DW\t").join(',',splice(@_, 0, 8))."\n") while @_; }
 
 sub ::data_word
-{   push(@out,("DD\t").join(',',@_)."\n"); }
+{   push @out, (("DD\t").join(',',splice(@_, 0, 4))."\n") while @_; }
 
 sub ::align
 {   push(@out,"ALIGN\t$_[0]\n"); }


On Fri, Mar 7, 2014 at 10:12 AM, Steven Kneizys <[hidden email]> wrote:
>> Quick question on 1.0.2 and Visual Studio win64 builds ... is ml64
>> supported or just nasm?

>Quick answer: quantum mechanics. Longer answer: state is unknown till
>it's measured. Long answer: reports are not rejected, but
>when-in-doubt-use-nasm principle applies, and reaction to reports can as
>well be avoiding problem.

Well then I will definitely send in my more complete analysis/proposed fixes.
....



--
Steve Kneizys
Senior Business Process Engineer
Voice: (610) 256-1396  [For Emergency Service (888)864-3282]
Ferrilli Information Group -- Quality Service and Solutions for Higher Education
web: http://www.ferrilli.com/

Making you a success while exceeding your expectations.

20140910_MASM_32BIT.txt (2K) Download Attachment