1

I have C Linux application which continuously allocates and frees memory (around 200 alloc/free per sec) using malloc, calloc, realloc & free functions. Even though all allocated memory are freed (verified by wrapping *alloc and free), the VmSize, VmRSS & VmData numbers are keep on increasing and finally application is getting killed by OOM killer.

Why the VmSize, VmRSS & VmData are keep on increasing? if it is Memory management issue, any pointers to avoid this?

I saw this Problem usage memory in C, but the answers are not explaining the OOM behavior.

Community
  • 1
  • 1
Jibin Scaria
  • 428
  • 4
  • 12
  • 5
    I *strongly* suggest you run it through **valgrind** rather than relying on macro'd log output. It is remarkably efficient at pointing out where potential leaks are, and it sounds like you have one, whether you agree or not, its worth checking. – WhozCraig May 16 '13 at 10:11
  • 1
    The first tool I used was valgrind, but it didn't help as there is no leak from application code – Jibin Scaria May 16 '13 at 10:14
  • Are you using **any** 3rd-party libs in your code outside of the runtime library implementation? – WhozCraig May 16 '13 at 10:15
  • no only standard libs, libc, xml2, pthread, z, m, rt etc. One more thing I forgot to mention is that the allocations are of random sizes – Jibin Scaria May 16 '13 at 10:18
  • The random sizes shouldn't matter unless your random sizes are statistically ever-increasing, as you would end up with a likely pretty putrid heap fragmented to the heavens. – WhozCraig May 16 '13 at 10:29
  • Libxml2 is a 3rd party library and does like to allocate memory. – thuovila May 16 '13 at 11:22
  • And what kind of application is this? – Basile Starynkevitch May 16 '13 at 12:07
  • Just curious, is it not possible to allocate some memory which would suit the application's needs in the beginning once and reuse it while expanding when insufficient? I agree with WhozCraig in that your code might be causing severe fragmentation. – Anish Ramaswamy May 16 '13 at 14:06
  • @AnishRam I am trying out this option. already under testing. The VmSize and VmData are constant as-of now but the VmRSS is increasing by 4KB. Need to wait and see what happens when VmRSS catches VmSize. – Jibin Scaria May 16 '13 at 17:55

1 Answers1

0

You should first use valgrind (to debug potential hard to find memory leaks or misbehavior). Don't forget to compile with gcc -Wall -g (then, when it works, use -Wall -O); of course, improve your code till no warnings are given.

You could probably (if the algorithms fit) try to (usefully) allocate memory zone of e.g. either power of two, or 3 times a power of two [perhaps minus 2 or 3 words]; at least try to avoid too many different random sizes of allocation.

You may want to try using Boehm's conservative garbage collector - i.e. replacing all your malloc with GC_MALLOC (or GC_MALLOC_ATOMIC & strdupwith GC_STRDUP), your free with GC_FREE, etc...

At least for testing purposes, use setrlimit(2) perhaps thru the bash ulimit builtin. You want RLIMIT_AS - possibly with RLIMIT_DATA (setting these limits sensibly avoids the OOM killer and makes your mmap -called by malloc- fail when memory is exhausted).

You may want to compile with GCC 4.8 which accepts -fsanitize=address.

You could also implement your own application specific garbage collector (read that wikipage, it gives you insights and terminology); a mark & compact algorithm will fight fragmentation.

See also this question about memory fragmentation. Look into the Plug tool.

Community
  • 1
  • 1
Basile Starynkevitch
  • 1
  • 16
  • 251
  • 479