Now, the "leading zero of INTEGER" part is present to indicate that following value is positive value integer. However I need to get rid of it due to some legacy reasons.
I was going through openssl source and found that through the DER construction its presence is decided based on ASN1_VALUE->type & V_ASN1_NEG, but I am unable to track down where to set generated PublicKey as V_ASN1_NEG (or influence it to be generated as negative).
Other way to handle this is to write my own TLV-DER parser and re-pack these few bytes to comply with what I need, but I would rather enforce API to do that for me, if it makes sense.
Re: Generating unsigned RSA Public Key with openssl
The API cannot do it. The encoding requires that numbers with the high-bit on have a leading zero to avoid being interpreted as negative numbers as you noticed. You could maybe generate our own RSA numbers with the high-bit off – i.e.,
make your own RSA_new kind of API. The BN code can have flags to not require the high bit on.