Although, that error is coming from one of the PEM_write_bio_* calls
(which you are not checking the error return code of), so it doesn't
explain why RSA_check_key() fails. You don't show how you generate the
RSA structure to start with, so I guess you're not generating it properly.