0

The embedded Linux system I have has 28K RAM and 31.6M of flash which the file system is mounted on. I developed an application that needs to run continuously inside the embedded Linux. The problem is that the virtual storage size (VSZ in top command) for the application keeps increasing until the system crashes. If I understand things right and correct me If I'm wrong, VSZ should vary depending on the memory allocation done by the program and if the memory is freed it should go to the previous value and stay constant.

The program uses the following declaration in the source code:

1- A vector of struct pointers:

vector<Struct*> someVector; // Struct pointers are initialized with new

// After the previous vector has some values, it is freed like this
for(unsigned i = 0; i < someVector.size(); i++)
{
    delete someVector[i];
}
someVector.clear();
vector<Struct*>().swap(someVector);

2- A vector of strings, same as before but instead of the delete inside the loop I do: someVector[i] = "";

3- char*

char* somePointer = (char*) malloc(100);
free(somePointer);
somePointer = NULL;

4- Sometimes I use fstream* initialized with new I don't think it will make any difference as C++ objects will be destroyed once out of the scope.

5- char** array = new char*[100]; This is deleted by delete[] array. I used delete on the internal elements but valgrind gives me Mismatched free/delete or something similar.

NOTE: The source is big and these are the most data structure that the application use. If there is anything else suspicious I will do an update

I compile the application and move it to the embedded device using SSH, I run the application and monitor top command output; the VSZ value for application's process keeps increasing until a large value like 11K and the system crashes. I'm pretty sure every allocation I do I have the corresponding free mechanism (free, delete, etc).

valgrind ./myApplication tool does not show even a warning message, Should I assume that allocation and deallocation is done safely ?

Any help is appreciated. I'm available immediately for any info required. Thanks In advance!

Update 8/4/2015: The process that increases the VSZ of the application is gathering data and storing them in files on the flash memory continuously, there is a mechanism to check for exceeding size but I do not use it now. Could this be related ?

3bdalla
  • 387
  • 1
  • 9
  • 27
  • You probably meant 28M RAM, not 28K? – amdn Apr 08 '15 at 10:13
  • *"Sometimes I use `fstream*` initialized with `new` I don't think it will make any difference as C++ objects will be destroyed once out of the scope."* - not unless you call `delete` on them explicitly, or use a smart pointer. *"I used delete on the internal elements but valgrind..."* - you should indeed use `delete[]` if they're allocated ala `new char[]`: are you sure you were only deleting the subset of array elements you'd populated though? You're better off using `vector`, and generally things that deallocate themselves. No need for 2 `someVector[i] = "";` – Tony Delroy Apr 08 '15 at 10:16
  • @amdn It might be 28Kb as well. It's embedded system after all. I was working with a system with 192Kb RAM and 32Kb were taken permanently. My friend was said it is quite much for this processor anyways, so I find it totally possible. – luk32 Apr 08 '15 at 10:16
  • @luk32, could be, but later OP says VSZ keeps increasing until a large value like 11K - that seems small even for embedded. – amdn Apr 08 '15 at 10:20
  • 1
    Doesn't `valgrind` complain about #2? That is clearly a memory leak unless the strings previously pointed by vector elemenst are static. And for #5, you should delete internal elements with `delete[]`, not `delete` – simurg Apr 08 '15 at 10:20
  • @simurg It is not a memory leak. It is a vector of strings not pointers to strings. Run-time library should clean everything when the vector goes out of scope. – luk32 Apr 08 '15 at 10:33
  • #4 is very wrong. As in C, only objects with automatic storage are handled automatically. (Objects don't have scope, only variables do.) – molbdnilo Apr 08 '15 at 10:34
  • see http://stackoverflow.com/questions/5994543/problem-usage-memory-in-c. If your libc supports , you can use malloc_stats to get the correct memory usage by the process. The values shown by "top" are not always the correct values. – k0n3ru Apr 08 '15 at 10:35
  • @luk32 You're right that it is not a memory leak as re-assignment is handled by the `string` class. However, `str = ""` is not freeing memory. It's is just (possibly) reducing the memory usage. And it is not clear if the vector is ever going out of scope. – simurg Apr 08 '15 at 10:53
  • @amdn: It is 28K RAM, it is a product that is already in production. – 3bdalla Apr 08 '15 at 10:54
  • @TonyD: The internal arrays are allocated by assigning values from other character arrays or strings. That part that uses the `char**` is a black-box for me. I did not implement and I do not know how it works but it serves a purpose in the code, it was taken from another project. If I delete the internal arrays the warning message in #5 will appear on valgrind. – 3bdalla Apr 08 '15 at 10:54
  • @simurg: No. Not even a warning message. But why would it be a memory leak ? If I delete the internal arrays the warning message in #5 will appear on valgrind. – 3bdalla Apr 08 '15 at 10:54
  • @simurg Yea, that is what I pointed out in my "answer", comment was too short for code example. It doesn't even reduce memory usage, at least not with implementations that I tested. – luk32 Apr 08 '15 at 10:58
  • @3bdalla: valgrind's mismatched free/delete message implies you're trying `delete[]` but they were allocated with malloc or realloc and need to be deallocated with free. – Tony Delroy Apr 08 '15 at 10:58
  • @molbdnilo: I did not get your point :\ – 3bdalla Apr 08 '15 at 11:02
  • @3bdalla The lifetimes of objects created with `new` are unrelated to scope (the objects don't even belong to any scope). For instance, in `int* p = new int(63);` only `p`'s lifetime is delimited by the current scope; the lifetime of the `int` it points to lasts until you `delete` it. – molbdnilo Apr 08 '15 at 11:12
  • I'm sure for every new/malloc there is delete/free – 3bdalla Apr 08 '15 at 11:17

3 Answers3

2

There's absolutely no reason to expect delete/free to return virtual memory to the system - they usually keep it in-process for later re-use. Some allocation patterns will cause growth even with no leaks due to free store fragmentation.

The first step is still to make sure you have no leaks, both using valgrind, and moving those dynamic allocations under smart pointer control where possible, and using valgrind's massif tool to see where your allocated memory is going (this will help identify problems like the one luk32 mentions).

The next step, if still required, is to use specialized allocators: fixed-size object allocators (which eliminate fragmentation and minimize overhead), arena allocators for objects (of whatever size) with well-known lifetime, and mmap-based backing stores for explicit, deterministic return of VM to the OS.

Note that you do have one built-in arena allocator already available - it's used for local variables with automatic scope. Using this instead of newing everything - where applicable - makes life much simpler.

Useless
  • 55,472
  • 5
  • 73
  • 117
0

someVector[i] = ""; this does not necessarily mean that the internal buffer will be trimmed. You can check it with capacity() In c++11 there is shrink_to_fit, other methods do not seem to work.

#include <iostream>
using namespace std;

int main() {
    string s("Some example string. Let's give it few more bytes.");
    cout << s.size() << '\n';
    cout << s.capacity() << '\n';
    s = "";
    cout << s.capacity() << '\n';
    s.clear();
    cout << s.capacity() << '\n';
    s.resize(0);
    cout << s.capacity() << '\n';
    s.shrink_to_fit();
    cout << s.capacity() << '\n';
    return 0;
}

Gives 50 for every method except shrink_to_fit, on ideone: http://ideone.com/nwIzYz. It also won't be reported as "lost" by valgrind. So until you delete those objects they will hold on their internal buffers. If you use it on some global scope, and then try to free memory with those calls, it most probably won't work.

luk32
  • 15,002
  • 33
  • 58
  • `shrink_to_fit` requires C++11 and my compiler does not support it. – 3bdalla Apr 08 '15 at 11:04
  • I'm just saying that if count that `someVector[i] = "";` will immediately release memory you are wrong. If you want to release it, you need to use `shrink_to_fit` or delete the string, `someVector.clear()` should do. I don't know your case so I can't give a specific answer. You need to draw conclusion, I believe I gave enough info on it. For example if you leave the vector around, for further re-use it will still occupy space. But I don't know how you use it. – luk32 Apr 08 '15 at 11:14
0

New/delete internally use malloc/free. Both functions are wrappers around kernel functions like mmap or bkr/sbrk (on linux). Those functions keep track of pages to optimize the memory fragmentation in case of a lot of small allocations are done.

When you call free, those pages maybe are not returned to the OS, in the same way when you query the OS for memory, the OS will not give you the memory until the memory is used or looks like you are going to use it.

If you want to return the memory to the OS, on linux, you can use malloc_trim.

I use this method on c++03 compilers to shrink to fit.

template<typename T, class Allocator>
void shrink_capacity(std::vector<T,Allocator>& m)
{
    std::vector<T,Allocator>(m.begin(),m.end()).swap(m);
}
Jose Palma
  • 747
  • 6
  • 13