OpenSSL hash memory leak

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

OpenSSL hash memory leak

prithiraj das

Hi All,

Using OpenSSL 1.0.2g, I have written a code to generate the hash of a file in an embeddded device having linux OS and low memory capacity and the files are generally of size 44 MB or more. The first time or even the second time on some occasions, the hash of any file is successfully generated. On the 3rd or 4th time (possibly due to lack of memory/memory leak), the system reboots before the hash can be generated.  After restart, the same thing happens when the previous steps are repeated.
The stats below shows the memory usage before and after computing the hash. 

root@at91sam9m10g45ek:~# free
                      total        used          free         shared    buff/cache   available
Mem:         252180       13272      223048         280          15860          230924
Swap:                0           0               0

After computing hash :-
root@at91sam9m10g45ek:~# free
                      total        used          free       shared    buff/cache   available
Mem:         252180       13308      179308        280       59564           230868
Swap:             0                0              0

Buff/cache increases by almost 44MB (same as file size) everytime I generate the hash and free decreases. I believe the file is being loaded into buffer and not being freed. 

I am using the below code to compute the message digest. This code is part of a function ComputeHash and the file pointer here is fph.

   EVP_add_digest(EVP_sha256());
 md = EVP_get_digestbyname("sha256");
 
 if(!md) {
        printf("Unknown message digest \n");
        exit(1);
 }
 printf("Message digest algorithm successfully loaded\n");
 mdctx = EVP_MD_CTX_create();
 EVP_DigestInit_ex(mdctx, md, NULL);

 // Reading data to array of unsigned chars
 long long int bytes_read = 0;

 printf("FILE size of the file to be hashed is %ld",filesize);

 //reading image file in chunks below and fph is the file pointer to the 44MB file
 while ((bytes_read = fread (message_data, 1, BUFFER_SIZE, fph)) != 0)
  EVP_DigestUpdate(mdctx, message_data, bytes_read);
 EVP_DigestFinal_ex(mdctx, hash_data.md_value, &hash_data.md_len);
 printf("\n%d\n",EVP_MD_CTX_size(mdctx));
 printf("\n%d\n",EVP_MD_CTX_type(mdctx));
 hash_data.md_type=EVP_MD_CTX_type(mdctx);
 EVP_MD_CTX_destroy(mdctx);
 //fclose(fp);
 printf("Generated Digest is:\n ");
 for(i = 0; i < hash_data.md_len; i++)
        printf("%02x", hash_data.md_value[i]);
 printf("\n");
 EVP_cleanup();
         return hash_data;

In the the code below, I have done fclose(fp)
verify_hash=ComputeHash(fp,size1);
fclose(fp);

I believe that instead of loading the entire file all at once I am reading the 44MB file in chunks and computing the hash using the piece of code below: (fph is the file pointer)
while ((bytes_read = fread (message_data, 1, BUFFER_SIZE, fph)) != 0)
  EVP_DigestUpdate(mdctx, message_data, bytes_read);

Where I am going wrong? How can I free the buff/cache after computation of message digest?  Please suggest ways to tackle this.


Thanks and Regards,
Prithiraj

Reply | Threaded
Open this post in threaded view
|

Re: OpenSSL hash memory leak

JordanBrown
The most obvious question is "how are you allocating your message_data buffer?".  You don't show that.

On 2/22/2019 2:27 AM, prithiraj das wrote:

Hi All,

Using OpenSSL 1.0.2g, I have written a code to generate the hash of a file in an embeddded device having linux OS and low memory capacity and the files are generally of size 44 MB or more. The first time or even the second time on some occasions, the hash of any file is successfully generated. On the 3rd or 4th time (possibly due to lack of memory/memory leak), the system reboots before the hash can be generated.  After restart, the same thing happens when the previous steps are repeated.
The stats below shows the memory usage before and after computing the hash. 

root@at91sam9m10g45ek:~# free
                      total        used          free         shared    buff/cache   available
Mem:         252180       13272      223048         280          15860          230924
Swap:                0           0               0

After computing hash :-
root@at91sam9m10g45ek:~# free
                      total        used          free       shared    buff/cache   available
Mem:         252180       13308      179308        280       59564           230868
Swap:             0                0              0

Buff/cache increases by almost 44MB (same as file size) everytime I generate the hash and free decreases. I believe the file is being loaded into buffer and not being freed. 

I am using the below code to compute the message digest. This code is part of a function ComputeHash and the file pointer here is fph.

   EVP_add_digest(EVP_sha256());
 md = EVP_get_digestbyname("sha256");
 
 if(!md) {
        printf("Unknown message digest \n");
        exit(1);
 }
 printf("Message digest algorithm successfully loaded\n");
 mdctx = EVP_MD_CTX_create();
 EVP_DigestInit_ex(mdctx, md, NULL);

 // Reading data to array of unsigned chars
 long long int bytes_read = 0;

 printf("FILE size of the file to be hashed is %ld",filesize);

 //reading image file in chunks below and fph is the file pointer to the 44MB file
 while ((bytes_read = fread (message_data, 1, BUFFER_SIZE, fph)) != 0)
  EVP_DigestUpdate(mdctx, message_data, bytes_read);
 EVP_DigestFinal_ex(mdctx, hash_data.md_value, &hash_data.md_len);
 printf("\n%d\n",EVP_MD_CTX_size(mdctx));
 printf("\n%d\n",EVP_MD_CTX_type(mdctx));
 hash_data.md_type=EVP_MD_CTX_type(mdctx);
 EVP_MD_CTX_destroy(mdctx);
 //fclose(fp);
 printf("Generated Digest is:\n ");
 for(i = 0; i < hash_data.md_len; i++)
        printf("%02x", hash_data.md_value[i]);
 printf("\n");
 EVP_cleanup();
         return hash_data;

In the the code below, I have done fclose(fp)
verify_hash=ComputeHash(fp,size1);
fclose(fp);

I believe that instead of loading the entire file all at once I am reading the 44MB file in chunks and computing the hash using the piece of code below: (fph is the file pointer)
while ((bytes_read = fread (message_data, 1, BUFFER_SIZE, fph)) != 0)
  EVP_DigestUpdate(mdctx, message_data, bytes_read);

Where I am going wrong? How can I free the buff/cache after computation of message digest?  Please suggest ways to tackle this.


Thanks and Regards,
Prithiraj


-- 
Jordan Brown, Oracle ZFS Storage Appliance, Oracle Solaris
Reply | Threaded
Open this post in threaded view
|

Re: OpenSSL hash memory leak

prithiraj das
Hi,
This is how I have initialized my variables:-

EVP_MD_CTX *mdctx;
const EVP_MD *md;
int i;
HASH hash_data;
unsigned char message_data[BUFFER_SIZE];

BUFFER_SIZE has been defined as 131072
and HASH is my hash structure defined to hold the message digest, message digest length and message digest type

On Sat, 23 Feb 2019 at 00:17, Jordan Brown <[hidden email]> wrote:
The most obvious question is "how are you allocating your message_data buffer?".  You don't show that.

On 2/22/2019 2:27 AM, prithiraj das wrote:

Hi All,

Using OpenSSL 1.0.2g, I have written a code to generate the hash of a file in an embeddded device having linux OS and low memory capacity and the files are generally of size 44 MB or more. The first time or even the second time on some occasions, the hash of any file is successfully generated. On the 3rd or 4th time (possibly due to lack of memory/memory leak), the system reboots before the hash can be generated.  After restart, the same thing happens when the previous steps are repeated.
The stats below shows the memory usage before and after computing the hash. 

root@at91sam9m10g45ek:~# free
                      total        used          free         shared    buff/cache   available
Mem:         252180       13272      223048         280          15860          230924
Swap:                0           0               0

After computing hash :-
root@at91sam9m10g45ek:~# free
                      total        used          free       shared    buff/cache   available
Mem:         252180       13308      179308        280       59564           230868
Swap:             0                0              0

Buff/cache increases by almost 44MB (same as file size) everytime I generate the hash and free decreases. I believe the file is being loaded into buffer and not being freed. 

I am using the below code to compute the message digest. This code is part of a function ComputeHash and the file pointer here is fph.

   EVP_add_digest(EVP_sha256());
 md = EVP_get_digestbyname("sha256");
 
 if(!md) {
        printf("Unknown message digest \n");
        exit(1);
 }
 printf("Message digest algorithm successfully loaded\n");
 mdctx = EVP_MD_CTX_create();
 EVP_DigestInit_ex(mdctx, md, NULL);

 // Reading data to array of unsigned chars
 long long int bytes_read = 0;

 printf("FILE size of the file to be hashed is %ld",filesize);

 //reading image file in chunks below and fph is the file pointer to the 44MB file
 while ((bytes_read = fread (message_data, 1, BUFFER_SIZE, fph)) != 0)
  EVP_DigestUpdate(mdctx, message_data, bytes_read);
 EVP_DigestFinal_ex(mdctx, hash_data.md_value, &hash_data.md_len);
 printf("\n%d\n",EVP_MD_CTX_size(mdctx));
 printf("\n%d\n",EVP_MD_CTX_type(mdctx));
 hash_data.md_type=EVP_MD_CTX_type(mdctx);
 EVP_MD_CTX_destroy(mdctx);
 //fclose(fp);
 printf("Generated Digest is:\n ");
 for(i = 0; i < hash_data.md_len; i++)
        printf("%02x", hash_data.md_value[i]);
 printf("\n");
 EVP_cleanup();
         return hash_data;

In the the code below, I have done fclose(fp)
verify_hash=ComputeHash(fp,size1);
fclose(fp);

I believe that instead of loading the entire file all at once I am reading the 44MB file in chunks and computing the hash using the piece of code below: (fph is the file pointer)
while ((bytes_read = fread (message_data, 1, BUFFER_SIZE, fph)) != 0)
  EVP_DigestUpdate(mdctx, message_data, bytes_read);

Where I am going wrong? How can I free the buff/cache after computation of message digest?  Please suggest ways to tackle this.


Thanks and Regards,
Prithiraj


-- 
Jordan Brown, Oracle ZFS Storage Appliance, Oracle Solaris
Reply | Threaded
Open this post in threaded view
|

AW: OpenSSL hash memory leak

Georg Höllrigl

Hello,

 

I guess you’re not seeing a memory leak, but just normal behaviour of linux file system cache.

Buff/cache is keeping files in memory that were least accessed as long as not needed by other stuff.

You don’t need to free the buffer/cache, because linux does that automatically, when memory is needed.

 

Kind Regards,

Georg

 

Von: openssl-users <[hidden email]> Im Auftrag von prithiraj das
Gesendet: 23 February 2019 18:25
An: Jordan Brown <[hidden email]>
Cc: [hidden email]
Betreff: Re: OpenSSL hash memory leak

 

Hi,

This is how I have initialized my variables:-

 

EVP_MD_CTX *mdctx;

const EVP_MD *md;

int i;

HASH hash_data;

unsigned char message_data[BUFFER_SIZE];

 

BUFFER_SIZE has been defined as 131072

and HASH is my hash structure defined to hold the message digest, message digest length and message digest type

 

On Sat, 23 Feb 2019 at 00:17, Jordan Brown <[hidden email]> wrote:

The most obvious question is "how are you allocating your message_data buffer?".  You don't show that.

 

On 2/22/2019 2:27 AM, prithiraj das wrote:

 

Hi All,

 

Using OpenSSL 1.0.2g, I have written a code to generate the hash of a file in an embeddded device having linux OS and low memory capacity and the files are generally of size 44 MB or more. The first time or even the second time on some occasions, the hash of any file is successfully generated. On the 3rd or 4th time (possibly due to lack of memory/memory leak), the system reboots before the hash can be generated.  After restart, the same thing happens when the previous steps are repeated.

The stats below shows the memory usage before and after computing the hash. 

 

root@at91sam9m10g45ek:~# free

                      total        used          free         shared    buff/cache   available

Mem:         252180       13272      223048         280          15860          230924

Swap:                0           0               0

 

After computing hash :-

root@at91sam9m10g45ek:~# free

                      total        used          free       shared    buff/cache   available

Mem:         252180       13308      179308        280       59564           230868

Swap:             0                0              0

 

Buff/cache increases by almost 44MB (same as file size) everytime I generate the hash and free decreases. I believe the file is being loaded into buffer and not being freed. 

 

I am using the below code to compute the message digest. This code is part of a function ComputeHash and the file pointer here is fph.

 

   EVP_add_digest(EVP_sha256());

 md = EVP_get_digestbyname("sha256");

 

 if(!md) {

        printf("Unknown message digest \n");

        exit(1);

 }

 printf("Message digest algorithm successfully loaded\n");

 mdctx = EVP_MD_CTX_create();

 EVP_DigestInit_ex(mdctx, md, NULL);

 

 // Reading data to array of unsigned chars

 long long int bytes_read = 0;

 

 printf("FILE size of the file to be hashed is %ld",filesize);

 

 //reading image file in chunks below and fph is the file pointer to the 44MB file

 while ((bytes_read = fread (message_data, 1, BUFFER_SIZE, fph)) != 0)

  EVP_DigestUpdate(mdctx, message_data, bytes_read);

 EVP_DigestFinal_ex(mdctx, hash_data.md_value, &hash_data.md_len);

 printf("\n%d\n",EVP_MD_CTX_size(mdctx));

 printf("\n%d\n",EVP_MD_CTX_type(mdctx));

 hash_data.md_type=EVP_MD_CTX_type(mdctx);

 EVP_MD_CTX_destroy(mdctx);

 //fclose(fp);

 printf("Generated Digest is:\n ");

 for(i = 0; i < hash_data.md_len; i++)

        printf("%02x", hash_data.md_value[i]);

 printf("\n");

 EVP_cleanup();

         return hash_data;

 

In the the code below, I have done fclose(fp)

verify_hash=ComputeHash(fp,size1);

fclose(fp);

 

I believe that instead of loading the entire file all at once I am reading the 44MB file in chunks and computing the hash using the piece of code below: (fph is the file pointer)

while ((bytes_read = fread (message_data, 1, BUFFER_SIZE, fph)) != 0)

  EVP_DigestUpdate(mdctx, message_data, bytes_read);

 

Where I am going wrong? How can I free the buff/cache after computation of message digest?  Please suggest ways to tackle this.

 

 

Thanks and Regards,

Prithiraj

 

 

-- 
Jordan Brown, Oracle ZFS Storage Appliance, Oracle Solaris
Reply | Threaded
Open this post in threaded view
|

Re: OpenSSL hash memory leak

prithiraj das
Hi All,

Apart from my code posted in this mailchain, I tried testing using the OpenSSL commands. I ran openssl dgst -sha256 Test_blob. Test_blob and all files mentioned below are almost 44 MB (or more).

The first time buff/cache value increased by 44MB (size of the file)
                    total        used           free        shared  buff/cache   available
Mem:         252180       12984      181544         284       57652         231188
Swap:                  0           0           0

I ran the same OpenSSL command again with the same file, and the result of free command remained the same
                    total        used           free        shared  buff/cache   available
Mem:         252180       12984      181544         284       57652        231188
Swap:                  0           0           0

Next I ran the same command with a different file (let's say Test_blob2) and ran the free command after it, result:-
                         total        used        free        shared  buff/cache   available
Mem:            252180       12948      137916      284      101316         231200
Swap:                    0           0           0

The buff/cache value has increased by the size of the file concerned (almost 44MB)
If I run the same command the 3rd time with another file not previously used (let's say Test_blob3), the following happens

Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
Rebooting in 15 seconds..

Is there a way to resolve this problem, How do I clear the buff/cache?

On Sun, 24 Feb 2019 at 03:15, Georg Höllrigl <[hidden email]> wrote:

Hello,

 

I guess you’re not seeing a memory leak, but just normal behaviour of linux file system cache.

Buff/cache is keeping files in memory that were least accessed as long as not needed by other stuff.

You don’t need to free the buffer/cache, because linux does that automatically, when memory is needed.

 

Kind Regards,

Georg

 

Von: openssl-users <[hidden email]> Im Auftrag von prithiraj das
Gesendet: 23 February 2019 18:25
An: Jordan Brown <[hidden email]>
Cc: [hidden email]
Betreff: Re: OpenSSL hash memory leak

 

Hi,

This is how I have initialized my variables:-

 

EVP_MD_CTX *mdctx;

const EVP_MD *md;

int i;

HASH hash_data;

unsigned char message_data[BUFFER_SIZE];

 

BUFFER_SIZE has been defined as 131072

and HASH is my hash structure defined to hold the message digest, message digest length and message digest type

 

On Sat, 23 Feb 2019 at 00:17, Jordan Brown <[hidden email]> wrote:

The most obvious question is "how are you allocating your message_data buffer?".  You don't show that.

 

On 2/22/2019 2:27 AM, prithiraj das wrote:

 

Hi All,

 

Using OpenSSL 1.0.2g, I have written a code to generate the hash of a file in an embeddded device having linux OS and low memory capacity and the files are generally of size 44 MB or more. The first time or even the second time on some occasions, the hash of any file is successfully generated. On the 3rd or 4th time (possibly due to lack of memory/memory leak), the system reboots before the hash can be generated.  After restart, the same thing happens when the previous steps are repeated.

The stats below shows the memory usage before and after computing the hash. 

 

root@at91sam9m10g45ek:~# free

                      total        used          free         shared    buff/cache   available

Mem:         252180       13272      223048         280          15860          230924

Swap:                0           0               0

 

After computing hash :-

root@at91sam9m10g45ek:~# free

                      total        used          free       shared    buff/cache   available

Mem:         252180       13308      179308        280       59564           230868

Swap:             0                0              0

 

Buff/cache increases by almost 44MB (same as file size) everytime I generate the hash and free decreases. I believe the file is being loaded into buffer and not being freed. 

 

I am using the below code to compute the message digest. This code is part of a function ComputeHash and the file pointer here is fph.

 

   EVP_add_digest(EVP_sha256());

 md = EVP_get_digestbyname("sha256");

 

 if(!md) {

        printf("Unknown message digest \n");

        exit(1);

 }

 printf("Message digest algorithm successfully loaded\n");

 mdctx = EVP_MD_CTX_create();

 EVP_DigestInit_ex(mdctx, md, NULL);

 

 // Reading data to array of unsigned chars

 long long int bytes_read = 0;

 

 printf("FILE size of the file to be hashed is %ld",filesize);

 

 //reading image file in chunks below and fph is the file pointer to the 44MB file

 while ((bytes_read = fread (message_data, 1, BUFFER_SIZE, fph)) != 0)

  EVP_DigestUpdate(mdctx, message_data, bytes_read);

 EVP_DigestFinal_ex(mdctx, hash_data.md_value, &hash_data.md_len);

 printf("\n%d\n",EVP_MD_CTX_size(mdctx));

 printf("\n%d\n",EVP_MD_CTX_type(mdctx));

 hash_data.md_type=EVP_MD_CTX_type(mdctx);

 EVP_MD_CTX_destroy(mdctx);

 //fclose(fp);

 printf("Generated Digest is:\n ");

 for(i = 0; i < hash_data.md_len; i++)

        printf("%02x", hash_data.md_value[i]);

 printf("\n");

 EVP_cleanup();

         return hash_data;

 

In the the code below, I have done fclose(fp)

verify_hash=ComputeHash(fp,size1);

fclose(fp);

 

I believe that instead of loading the entire file all at once I am reading the 44MB file in chunks and computing the hash using the piece of code below: (fph is the file pointer)

while ((bytes_read = fread (message_data, 1, BUFFER_SIZE, fph)) != 0)

  EVP_DigestUpdate(mdctx, message_data, bytes_read);

 

Where I am going wrong? How can I free the buff/cache after computation of message digest?  Please suggest ways to tackle this.

 

 

Thanks and Regards,

Prithiraj

 

 

-- 
Jordan Brown, Oracle ZFS Storage Appliance, Oracle Solaris
Reply | Threaded
Open this post in threaded view
|

Re: OpenSSL hash memory leak

prithiraj das
If it helps, sometimes I do get the following errors for the same and subsequent reboot:

Alignment trap: sh (601) PC=0xb6e008f8 Instr=0x4589c0d7 Address=0x000000d7 FSR 0x801
Alignment trap: login (584) PC=0xb6e6ab00 Instr=0xe5951000 Address=0xd27cdc63 FSR 0x001
Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b


On Sun, 24 Feb 2019 at 15:58, prithiraj das <[hidden email]> wrote:
Hi All,

Apart from my code posted in this mailchain, I tried testing using the OpenSSL commands. I ran openssl dgst -sha256 Test_blob. Test_blob and all files mentioned below are almost 44 MB (or more).

The first time buff/cache value increased by 44MB (size of the file)
                    total        used           free        shared  buff/cache   available
Mem:         252180       12984      181544         284       57652         231188
Swap:                  0           0           0

I ran the same OpenSSL command again with the same file, and the result of free command remained the same
                    total        used           free        shared  buff/cache   available
Mem:         252180       12984      181544         284       57652        231188
Swap:                  0           0           0

Next I ran the same command with a different file (let's say Test_blob2) and ran the free command after it, result:-
                         total        used        free        shared  buff/cache   available
Mem:            252180       12948      137916      284      101316         231200
Swap:                    0           0           0

The buff/cache value has increased by the size of the file concerned (almost 44MB)
If I run the same command the 3rd time with another file not previously used (let's say Test_blob3), the following happens

Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
Rebooting in 15 seconds..

Is there a way to resolve this problem, How do I clear the buff/cache?

On Sun, 24 Feb 2019 at 03:15, Georg Höllrigl <[hidden email]> wrote:

Hello,

 

I guess you’re not seeing a memory leak, but just normal behaviour of linux file system cache.

Buff/cache is keeping files in memory that were least accessed as long as not needed by other stuff.

You don’t need to free the buffer/cache, because linux does that automatically, when memory is needed.

 

Kind Regards,

Georg

 

Von: openssl-users <[hidden email]> Im Auftrag von prithiraj das
Gesendet: 23 February 2019 18:25
An: Jordan Brown <[hidden email]>
Cc: [hidden email]
Betreff: Re: OpenSSL hash memory leak

 

Hi,

This is how I have initialized my variables:-

 

EVP_MD_CTX *mdctx;

const EVP_MD *md;

int i;

HASH hash_data;

unsigned char message_data[BUFFER_SIZE];

 

BUFFER_SIZE has been defined as 131072

and HASH is my hash structure defined to hold the message digest, message digest length and message digest type

 

On Sat, 23 Feb 2019 at 00:17, Jordan Brown <[hidden email]> wrote:

The most obvious question is "how are you allocating your message_data buffer?".  You don't show that.

 

On 2/22/2019 2:27 AM, prithiraj das wrote:

 

Hi All,

 

Using OpenSSL 1.0.2g, I have written a code to generate the hash of a file in an embeddded device having linux OS and low memory capacity and the files are generally of size 44 MB or more. The first time or even the second time on some occasions, the hash of any file is successfully generated. On the 3rd or 4th time (possibly due to lack of memory/memory leak), the system reboots before the hash can be generated.  After restart, the same thing happens when the previous steps are repeated.

The stats below shows the memory usage before and after computing the hash. 

 

root@at91sam9m10g45ek:~# free

                      total        used          free         shared    buff/cache   available

Mem:         252180       13272      223048         280          15860          230924

Swap:                0           0               0

 

After computing hash :-

root@at91sam9m10g45ek:~# free

                      total        used          free       shared    buff/cache   available

Mem:         252180       13308      179308        280       59564           230868

Swap:             0                0              0

 

Buff/cache increases by almost 44MB (same as file size) everytime I generate the hash and free decreases. I believe the file is being loaded into buffer and not being freed. 

 

I am using the below code to compute the message digest. This code is part of a function ComputeHash and the file pointer here is fph.

 

   EVP_add_digest(EVP_sha256());

 md = EVP_get_digestbyname("sha256");

 

 if(!md) {

        printf("Unknown message digest \n");

        exit(1);

 }

 printf("Message digest algorithm successfully loaded\n");

 mdctx = EVP_MD_CTX_create();

 EVP_DigestInit_ex(mdctx, md, NULL);

 

 // Reading data to array of unsigned chars

 long long int bytes_read = 0;

 

 printf("FILE size of the file to be hashed is %ld",filesize);

 

 //reading image file in chunks below and fph is the file pointer to the 44MB file

 while ((bytes_read = fread (message_data, 1, BUFFER_SIZE, fph)) != 0)

  EVP_DigestUpdate(mdctx, message_data, bytes_read);

 EVP_DigestFinal_ex(mdctx, hash_data.md_value, &hash_data.md_len);

 printf("\n%d\n",EVP_MD_CTX_size(mdctx));

 printf("\n%d\n",EVP_MD_CTX_type(mdctx));

 hash_data.md_type=EVP_MD_CTX_type(mdctx);

 EVP_MD_CTX_destroy(mdctx);

 //fclose(fp);

 printf("Generated Digest is:\n ");

 for(i = 0; i < hash_data.md_len; i++)

        printf("%02x", hash_data.md_value[i]);

 printf("\n");

 EVP_cleanup();

         return hash_data;

 

In the the code below, I have done fclose(fp)

verify_hash=ComputeHash(fp,size1);

fclose(fp);

 

I believe that instead of loading the entire file all at once I am reading the 44MB file in chunks and computing the hash using the piece of code below: (fph is the file pointer)

while ((bytes_read = fread (message_data, 1, BUFFER_SIZE, fph)) != 0)

  EVP_DigestUpdate(mdctx, message_data, bytes_read);

 

Where I am going wrong? How can I free the buff/cache after computation of message digest?  Please suggest ways to tackle this.

 

 

Thanks and Regards,

Prithiraj

 

 

-- 
Jordan Brown, Oracle ZFS Storage Appliance, Oracle Solaris
Reply | Threaded
Open this post in threaded view
|

Re: OpenSSL hash memory leak

Georg Höllrigl
That pretty much sounds like a hardware problem. I'd expect that you see similar behaviour when you md5sum the files?

Openssl mailing list might be the wrong place for that topic.


-------- Ursprüngliche Nachricht --------
Von: prithiraj das <[hidden email]>
Datum: 24.02.19 11:34 (GMT+01:00)
An: Georg Höllrigl <[hidden email]>, [hidden email], Jordan Brown <[hidden email]>
Betreff: Re: OpenSSL hash memory leak

If it helps, sometimes I do get the following errors for the same and subsequent reboot:

Alignment trap: sh (601) PC=0xb6e008f8 Instr=0x4589c0d7 Address=0x000000d7 FSR 0x801
Alignment trap: login (584) PC=0xb6e6ab00 Instr=0xe5951000 Address=0xd27cdc63 FSR 0x001
Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b


On Sun, 24 Feb 2019 at 15:58, prithiraj das <[hidden email]> wrote:
Hi All,

Apart from my code posted in this mailchain, I tried testing using the OpenSSL commands. I ran openssl dgst -sha256 Test_blob. Test_blob and all files mentioned below are almost 44 MB (or more).

The first time buff/cache value increased by 44MB (size of the file)
                    total        used           free        shared  buff/cache   available
Mem:         252180       12984      181544         284       57652         231188
Swap:                  0           0           0

I ran the same OpenSSL command again with the same file, and the result of free command remained the same
                    total        used           free        shared  buff/cache   available
Mem:         252180       12984      181544         284       57652        231188
Swap:                  0           0           0

Next I ran the same command with a different file (let's say Test_blob2) and ran the free command after it, result:-
                         total        used        free        shared  buff/cache   available
Mem:            252180       12948      137916      284      101316         231200
Swap:                    0           0           0

The buff/cache value has increased by the size of the file concerned (almost 44MB)
If I run the same command the 3rd time with another file not previously used (let's say Test_blob3), the following happens

Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
Rebooting in 15 seconds..

Is there a way to resolve this problem, How do I clear the buff/cache?

On Sun, 24 Feb 2019 at 03:15, Georg Höllrigl <[hidden email]> wrote:

Hello,

 

I guess you’re not seeing a memory leak, but just normal behaviour of linux file system cache.

Buff/cache is keeping files in memory that were least accessed as long as not needed by other stuff.

You don’t need to free the buffer/cache, because linux does that automatically, when memory is needed.

 

Kind Regards,

Georg

 

Von: openssl-users <[hidden email]> Im Auftrag von prithiraj das
Gesendet: 23 February 2019 18:25
An: Jordan Brown <[hidden email]>
Cc: [hidden email]
Betreff: Re: OpenSSL hash memory leak

 

Hi,

This is how I have initialized my variables:-

 

EVP_MD_CTX *mdctx;

const EVP_MD *md;

int i;

HASH hash_data;

unsigned char message_data[BUFFER_SIZE];

 

BUFFER_SIZE has been defined as 131072

and HASH is my hash structure defined to hold the message digest, message digest length and message digest type

 

On Sat, 23 Feb 2019 at 00:17, Jordan Brown <[hidden email]> wrote:

The most obvious question is "how are you allocating your message_data buffer?".  You don't show that.

 

On 2/22/2019 2:27 AM, prithiraj das wrote:

 

Hi All,

 

Using OpenSSL 1.0.2g, I have written a code to generate the hash of a file in an embeddded device having linux OS and low memory capacity and the files are generally of size 44 MB or more. The first time or even the second time on some occasions, the hash of any file is successfully generated. On the 3rd or 4th time (possibly due to lack of memory/memory leak), the system reboots before the hash can be generated.  After restart, the same thing happens when the previous steps are repeated.

The stats below shows the memory usage before and after computing the hash. 

 

root@at91sam9m10g45ek:~# free

                      total        used          free         shared    buff/cache   available

Mem:         252180       13272      223048         280          15860          230924

Swap:                0           0               0

 

After computing hash :-

root@at91sam9m10g45ek:~# free

                      total        used          free       shared    buff/cache   available

Mem:         252180       13308      179308        280       59564           230868

Swap:             0                0              0

 

Buff/cache increases by almost 44MB (same as file size) everytime I generate the hash and free decreases. I believe the file is being loaded into buffer and not being freed. 

 

I am using the below code to compute the message digest. This code is part of a function ComputeHash and the file pointer here is fph.

 

   EVP_add_digest(EVP_sha256());

 md = EVP_get_digestbyname("sha256");

 

 if(!md) {

        printf("Unknown message digest \n");

        exit(1);

 }

 printf("Message digest algorithm successfully loaded\n");

 mdctx = EVP_MD_CTX_create();

 EVP_DigestInit_ex(mdctx, md, NULL);

 

 // Reading data to array of unsigned chars

 long long int bytes_read = 0;

 

 printf("FILE size of the file to be hashed is %ld",filesize);

 

 //reading image file in chunks below and fph is the file pointer to the 44MB file

 while ((bytes_read = fread (message_data, 1, BUFFER_SIZE, fph)) != 0)

  EVP_DigestUpdate(mdctx, message_data, bytes_read);

 EVP_DigestFinal_ex(mdctx, hash_data.md_value, &hash_data.md_len);

 printf("\n%d\n",EVP_MD_CTX_size(mdctx));

 printf("\n%d\n",EVP_MD_CTX_type(mdctx));

 hash_data.md_type=EVP_MD_CTX_type(mdctx);

 EVP_MD_CTX_destroy(mdctx);

 //fclose(fp);

 printf("Generated Digest is:\n ");

 for(i = 0; i < hash_data.md_len; i++)

        printf("%02x", hash_data.md_value[i]);

 printf("\n");

 EVP_cleanup();

         return hash_data;

 

In the the code below, I have done fclose(fp)

verify_hash=ComputeHash(fp,size1);

fclose(fp);

 

I believe that instead of loading the entire file all at once I am reading the 44MB file in chunks and computing the hash using the piece of code below: (fph is the file pointer)

while ((bytes_read = fread (message_data, 1, BUFFER_SIZE, fph)) != 0)

  EVP_DigestUpdate(mdctx, message_data, bytes_read);

 

Where I am going wrong? How can I free the buff/cache after computation of message digest?  Please suggest ways to tackle this.

 

 

Thanks and Regards,

Prithiraj

 

 

-- 
Jordan Brown, Oracle ZFS Storage Appliance, Oracle Solaris
Reply | Threaded
Open this post in threaded view
|

Re: OpenSSL hash memory leak

Hubert Kario
In reply to this post by prithiraj das
On Sunday, 24 February 2019 11:34:18 CET prithiraj das wrote:
> If it helps, sometimes I do get the following errors for the same and
> subsequent reboot:
>
> Alignment trap: sh (601) PC=0xb6e008f8 Instr=0x4589c0d7 Address=0x000000d7
> FSR 0x801
> Alignment trap: login (584) PC=0xb6e6ab00 Instr=0xe5951000
> Address=0xd27cdc63 FSR 0x001
> Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b

that doesn't look like openssl problem at all, openssl may trigger it, but
only because it's using the system to its fullest potential, not because there
are issues in openssl

I'd suggest trying memtest86 and trying to capture full kernel stacktrace with
netconsole, in this order. But this mailing list is not a good place for
follow up on this.
--
Regards,
Hubert Kario
Senior Quality Engineer, QE BaseOS Security team
Web: www.cz.redhat.com
Red Hat Czech s.r.o., Purkyňova 115, 612 00  Brno, Czech Republic

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: OpenSSL hash memory leak

OpenSSL - User mailing list
On 25/02/2019 15:05, Hubert Kario wrote:

> On Sunday, 24 February 2019 11:34:18 CET prithiraj das wrote:
>> If it helps, sometimes I do get the following errors for the same and
>> subsequent reboot:
>>
>> Alignment trap: sh (601) PC=0xb6e008f8 Instr=0x4589c0d7 Address=0x000000d7
>> FSR 0x801
>> Alignment trap: login (584) PC=0xb6e6ab00 Instr=0xe5951000
>> Address=0xd27cdc63 FSR 0x001
>> Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
> that doesn't look like openssl problem at all, openssl may trigger it, but
> only because it's using the system to its fullest potential, not because there
> are issues in openssl
>
> I'd suggest trying memtest86 and trying to capture full kernel stacktrace with
> netconsole, in this order. But this mailing list is not a good place for
> follow up on this.

Just FYI.  "Alignment trap" is not usually a hardware issue.  It
is virtually always a software error (specifically, accessing a
16, 32, 64, 80 or 128 bit value through an insufficiently aligned
pointer).

A stack trace is needed to determine if this is a kernel or user
mode issue, and if so where.

Of cause there is the remote possibility that a hardware error
caused a pointer to have a value it shouldn't have according to
the code.

However unless the error is actually in OpenSSL code, there is
little that this list can do to fix the problem.

Given the specific text of the other error message, I hope you
are not somehow running OpenSSL itself as process 1 (init), as
that would be highly unusual.


Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

Reply | Threaded
Open this post in threaded view
|

Re: OpenSSL hash memory leak

Joe Browning
   EVP_DigestFinal_ex(mdctx, hash_data.md_value, &hash_data.md_len)

Missing reference there for value?
Joe

On Mon, Feb 25, 2019, 09:31 Jakob Bohm via openssl-users <[hidden email]> wrote:
On 25/02/2019 15:05, Hubert Kario wrote:
> On Sunday, 24 February 2019 11:34:18 CET prithiraj das wrote:
>> If it helps, sometimes I do get the following errors for the same and
>> subsequent reboot:
>>
>> Alignment trap: sh (601) PC=0xb6e008f8 Instr=0x4589c0d7 Address=0x000000d7
>> FSR 0x801
>> Alignment trap: login (584) PC=0xb6e6ab00 Instr=0xe5951000
>> Address=0xd27cdc63 FSR 0x001
>> Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
> that doesn't look like openssl problem at all, openssl may trigger it, but
> only because it's using the system to its fullest potential, not because there
> are issues in openssl
>
> I'd suggest trying memtest86 and trying to capture full kernel stacktrace with
> netconsole, in this order. But this mailing list is not a good place for
> follow up on this.

Just FYI.  "Alignment trap" is not usually a hardware issue.  It
is virtually always a software error (specifically, accessing a
16, 32, 64, 80 or 128 bit value through an insufficiently aligned
pointer).

A stack trace is needed to determine if this is a kernel or user
mode issue, and if so where.

Of cause there is the remote possibility that a hardware error
caused a pointer to have a value it shouldn't have according to
the code.

However unless the error is actually in OpenSSL code, there is
little that this list can do to fix the problem.

Given the specific text of the other error message, I hope you
are not somehow running OpenSSL itself as process 1 (init), as
that would be highly unusual.


Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded