EVP_Decrypt_Final_ex fails on larger files

[clarksom]

I am trying to make a C++ class to make encrytion and decrytion simple
using the blowfish algorithm with openssl.  Anyways, I have modelled my
program around
http://www.linuxgazette.com/issue87/misc/vinayak/sym_funcs.c.txt which is
a C program for reading and writing files, with encrytion using openssl.
Anyways, both programs seem to fail the same way on larger files (they
will pass on smaller files) and I can't seem to get EVP_DecryptFinal_ex to
work.
Here are my encrypt and decrypt functions with my defines as follows

IP_SIZE = 1024
OP_SIZE = 1032

int Encryption::encrypt(std::ifstream &istream, std::ofstream &ostream)
{
  unsigned char outbuf[OP_SIZE];
  int olen, tlen, n;
  char inbuff[IP_SIZE];
  EVP_CIPHER_CTX ctx;
  EVP_CIPHER_CTX_init (&ctx);
  EVP_EncryptInit_ex (&ctx, EVP_bf_cbc(), NULL, key, iv);

  olen = 0;
  tlen = 0;

  for (int i = 0;;i++)
  {
    bzero(&inbuff, IP_SIZE);

    if((n = istream.readsome(inbuff, IP_SIZE)) == -1)
    {
      std::cout << "read error\n";
      break;
    }

    else if(n == 0)
      break;

    if(EVP_EncryptUpdate(&ctx, outbuf, &olen, (unsigned char*)inbuff, n)
!= 1)
    {
      std::cout << "error in encrypt update\n";
      return 0;
    }
    std::cout << "before enc. fin. ||" << i << "||\n";
    std::cout << "olen: " << olen << "\n";
    //std::cout << "EVP_MAX_BLOCK_LENGTH: " << EVP_MAX_BLOCK_LENGTH << "\n";
    if(EVP_EncryptFinal_ex(&ctx, outbuf + olen, &tlen) != 1)
    {
      std::cout << "error in encrypt final\n";
      return 0;
    }
    else
    {
      std::cout << "Final success on: " << i << "\n";
      std::cout << "tlen: " << tlen << "\n";
    }

    olen += tlen;

    ostream.write((char*)outbuf, olen);
  }
  //EVP_CIPHER_CTX_cleanup (&ctx); not needed because of _ex's
  return 1;
}

int Encryption::decrypt(std::ifstream &istream, std::ofstream &ostream)
{
  unsigned char outbuf[IP_SIZE];
  int olen, tlen, n;
  char inbuff[OP_SIZE];
  EVP_CIPHER_CTX ctx;
  EVP_CIPHER_CTX_init (&ctx);
  EVP_DecryptInit_ex (&ctx, EVP_bf_cbc (), NULL, key, iv);

  tlen = 0;

  std::cout << "OP_SIZE: " << OP_SIZE << "\n";

  for (int i = 0;; i++)
  {
    bzero(&inbuff, OP_SIZE);

    if((n = istream.readsome(inbuff, OP_SIZE)) == -1)
    {
      std::cout << "read error\n";
      break;
    }

    else if(n == 0)
      break;

    if (EVP_DecryptUpdate(&ctx, outbuf, &olen, (unsigned char*)inbuff, n)
!= 1)
    {
      std::cout << "error in decrypt update\n";
      return 0;
    }
    std::cout << "before dec. fin. ||" << i << "||\n";
    std::cout << "inbuff size: " << n << "\n";
    std::cout << "olen size: " << olen << "\n";
    if (EVP_DecryptFinal_ex(&ctx, outbuf + olen, &tlen) != 1) //outbuf + olen
    {
      std::cout << "tlen: " << tlen << "\n";
      std::cout << "error in decrypt final\n";
      return 0;
    }
    else
    {
      //std::cout << "tlen: " << tlen << "\n";
    }
    olen += tlen;


    ostream.write((char*)outbuf, olen);
  }
  //EVP_CIPHER_CTX_cleanup (&ctx); //not needed because of _ex's
  return 1;
}

The output of my program during decryption is as follows

Key
166     234     103     213     202     55      168     35      155    
193     103 2        98      36      200     0
 ------
Initialization vector
88      240     255     191     16      116     255     183
 ------
OP_SIZE: 1032
before dec. fin. ||0||
inbuff size: 1032
olen size: 1024
DecryptFinal Success: 0
tlen: 0
before dec. fin. ||1||
inbuff size: 1032
olen size: 1032
DecryptFinal Success: 1
tlen: 0
before dec. fin. ||2||
inbuff size: 1032
olen size: 1032
DecryptFinal Success: 2
tlen: 0
before dec. fin. ||3||
inbuff size: 1032
olen size: 1032
DecryptFinal Success: 3
tlen: 0
before dec. fin. ||4||
inbuff size: 1032
olen size: 1032
DecryptFinal Success: 4
tlen: 0
before dec. fin. ||5||
inbuff size: 1032
olen size: 1032
DecryptFinal Success: 5
tlen: 0
before dec. fin. ||6||
inbuff size: 1032
olen size: 1032
DecryptFinal Success: 6
tlen: 0
before dec. fin. ||7||
inbuff size: 968
olen size: 968
tlen: 0
error in decrypt final
Segmentation fault

I have spent too long (about 20 hours) trying to debug this, and I just
can't get it, I'd really appreciate it if anyone could help me or even
give me a little advice on where I am going wrong.
if anyone would like more information (like my whole class), let me know,
I didn't want to post it since it's about 3x this size of relatively
useless stuff for this problem.

Thanks in advance.

--
Matt

______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    [hidden email]
Automated List Manager                           [hidden email]