5

I have an application that reads from large amount of MSMQ queues(about 10000 at the moment). I use queue.BeginPeek with UInt32.MaxValue timeout to receive message from queue. When the message appears in queue, I process it and call queue.BeginPeek again. So I listen to all the queues, but message processing is done on Thread Pool.

I noticed that memory usage slowly grows(two weeks of work cause growth from 200 MB to 800 MB). After investigating the dump file I see typical heap fragmentation picture with many free objects (some of them have size about several megabytes). And there are pinned objects between the holes.

This seems to be common situation when working with calls to unmanaged code, that create pinned objects. But I did not find any solution in the internet.

So is memory management in .NET so pure, that it does not allow to complete even such simple scenarios, or I miss something?

Edit : I've made some investigations in sample applications. The holes(free memory zones, so called free objects) between pinned objects are reused by GC when allocating memory for new objects. But in my production application, pinned objects are long living, and they finally appear in 2nd generation with the holes between them(because GC just shifts the border which separates generations). As I have really little normal long-living objects, I see this holes in 2nd generation in dump file.

So memory consumption of my application can grow to 10000*(average size of the hole). (10000 is the number of queues that can also increase in future). I have no ideas how to fix this at the moment. The only way is to restart application from time to time.

Once again I can only ask, why .NET does not have separate heap for pinned objects? (maybe this is newbie question). At the moment I see that calling async operations that work with unmanaged code can cause memory issues.

struhtanov
  • 416
  • 2
  • 10
  • 1
    Yes, lots of pinning going on when you've got 10,000 BeginPeek() calls going. Don't expect miracles. Growth doesn't mean much, you've only got a problem if that ultimately never stabilizes and crashes your app with OOM. – Hans Passant Oct 01 '12 at 14:33
  • I've edited my post, I got linear growth (from 200MB to 800MB in two weeks). I'm not sure when I'll get OOM, if this will happen, but this is still huge amount of memory. I can only repeat -- this scenario is typical in my opinion. I'm newbie in memory management tools, but why they can't implement separate heap for pinned objects? Another option is to create cache with OverlappedData structures, but as System.Messaging is not open source, it is hard to implement such behavior. With current GC implementation pinned objects can be used only in simple scenarios – struhtanov Oct 01 '12 at 15:20
  • How many concurrent calls to BeginPeek do you have. And you always call EndPeek, right? In my estimation nothing should be left pinned one you quiesce the system (no outstanding calls). – usr Oct 01 '12 at 15:21
  • As I always need to listen to the queue, I call BeginPeek again, after handling the message. So I always have 10000 concurrent BeginPeek's. – struhtanov Oct 01 '12 at 16:05
  • Pinned objects don't go on a separate heap because they are not pinned in the first place. They are only pinned while calling an unmanaged function that needs access to the memory during the call. In your case these unmanaged calls are long lived meaning that you have many pinned objects. – Martin Liversage Oct 05 '12 at 11:10
  • Why not to move object in another heap when it actually gets pinned? – struhtanov Oct 08 '12 at 12:45
  • You can try to use the Server GC, which can be configured in the App.config. It seems to have better steady state behavior (less time on GC and less memory used) when this happens. – Pablo Montilla Jun 26 '15 at 13:06

2 Answers2

4

After looking at the source code for the managed wrapper for MSMQ it seems that you have stumbled upon a genuine problem using the API. Calling BeginPeek will create a collection of properties that are then pinned before passed to the unmanaged API. Only when a message is received are these properties unpinned but to continue receiving messages you have to call BeginPeek at this point leading to memory fragmentation over time.

If this fragmentation is a real concern the only hack I can come up with is every hour or so you should cancel all calls to BeginPeek, force a garbage collection and then resume normal listening operations. This should allow the garbage collector to handle the fragmentation.

Martin Liversage
  • 96,855
  • 20
  • 193
  • 238
  • This is interesting. Upon EndPeek, the memory will be freed and able to be reclaimed right? The problem is that it has moved into the area used for longer-lived objects such that it won't get cleaned as quickly right? Wouldn't an occasional forced garbage collection clean up those unpinned areas? – tcarvin Oct 05 '12 at 12:21
  • The source is somewhat complex but if I didn't misread it then the properties are unpinned when a message is received from the queue (you wait with infinte timeout). The properties are not freed at that time but as long as they are pinned they cannot be garbage collected and **not moved** either. The garbage collector compacts the heap by moving managed objects on each GC but in your case with X*10,000 pinned objects I guess compacting is hard. – Martin Liversage Oct 05 '12 at 13:22
  • Your suggestion should work as I see. I also thought about such solution, but synchronizing Peek's is much more complex than restarting the service(as I have not GUI application, this is not such a big deal). If there will be no other good suggestions I will mark this as answer. Thanks. – struhtanov Oct 08 '12 at 12:44
1

Well, if it is an issue with the pinned objects being long-lived then a simple work-around would be to use a timeout on the BeginPeek with a shorter duration to keep them from moving into the next generation. After each timeout and EndPeek you can reissue the BeginPeek. Depending on when the properties that Martin refered to are created and disposed, you may have to recreate the queue object (not the queue itself obviously, just the amanged wrapper). Though with luck you won't have to take it that far.

Martin Liversage
  • 96,855
  • 20
  • 193
  • 238
tcarvin
  • 10,293
  • 3
  • 27
  • 50
  • This does no help. Moreover such solution cause faster heap fragmentation(I changed timeout to infinite just because of this observation). As I can't synchronize BeginPeek calls, garbage collection may (and it is) occur between reissuing BeginPeeks, so some of them will go 2nd generation, and heap will not be compacted, and also this pinned objects will be on the top of the 2nd generation, so memory usage will only increase. Each 10 seconds we will increase memory usage at least on the size of pinned objects, that were in the 0 gen when GC starts. – struhtanov Oct 08 '12 at 12:40