7

I have a recursive problem where The consumers do some work at each level of a tree, then need to recurse down the tree and perform that same work at the next level.

I want to use ConcurrentBag/BlockingCollection etc to run this in parallel. In this scenario, the consumers of the queue, are also the producers for the queue!

My problem is this : Using BlockingCollection, I can write very simple foreach logic to dequeue items, and queue new ones - When the queue is empty, the blocking collection will block correctly, and wait for new work to be produced by one of the other consumers.

But how do I know if all of the consumers are blocking?!

I know about CompleteAdding(), but that does not seem to serve, as the only time you are actually complete is when all of the producers are done producing and the queue is empty - and since they would all be blocking, there is nobody "free" to set CompleteAdding(). Is there a way to detect this? (Perhaps an event that can fire when blocking, and fire again when unblocked?)

I can deal with this manually, by not using a foreach, but manually having a while(!complete) loop, and using TryTake, but then I need to manually sleep, which seems inefficent (the whole reason to have the blocking collection vs just the concurrent collections in the first place!) Each time through the loop, if TryTake is false, I could set an Idle flag, and then have a Master check if the queue is empty, and all threads are idle, set a complete flag, but again, this seems kludgy.

Intuition is telling me there is some way to use Blocking Collection to do this, but I cant quite get there.

Anyway, anyone have a good pattern for when the consumers are the producers and being able to detect when to release all the blocks would be awesome

Jason Coyne
  • 6,179
  • 7
  • 35
  • 64
  • 1
    Good question. Anything with external flags or events seems ripe for race conditions. – Damien_The_Unbeliever Jun 01 '12 at 14:45
  • Do the processors (combined consumers/producers) have a lot of state or require a lot of resource? Could you re-cast the problem in terms of creating `Task`s that each do one iteration only? – Damien_The_Unbeliever Jun 01 '12 at 14:48
  • @Damien_The_Unbeliever : Yes, I can do single iteration, and in fact already have that working, but am trying to use a producer/consumer pattern as this is code that may be migrated to the cloud in the future, where worker roles will be using the Azure Queue storage in the same way, and I would like to keep the overall logic as similar as possible between the two implementations. In that scenario I will be forced to check if the Workers are all idle to decide that the queue will is done, but it seems like I should be efficient where possible locally - also, I just want to figure this out :) – Jason Coyne Jun 01 '12 at 14:55
  • Here is an article that _almost_ answers the question, but not quite, since it assumes all production will be done before consumption begins. Im thikning I may have to manually sleep and check :( http://www.informit.com/guides/content.aspx?g=dotnet&seqNum=842 – Jason Coyne Jun 01 '12 at 15:17

0 Answers0