ListGet Chunk Number being executed

Question

I am breaking a list into chunks and processing it as below:

foreach (var partialist in breaklistinchunks(chunksize))
{
  try
   {
       do something
  }
catch
{
print error
}

}



public static class IEnumerableExtensions
    {
        public static IEnumerable<List<T>> BreakListinChunks<T>(this IEnumerable<T> sourceList, int chunkSize)
        {
            List<T> chunkReturn = new List<T>(chunkSize);
            foreach (var item in sourceList)
            {
                chunkReturn.Add(item);
                if (chunkReturn.Count == chunkSize)
                {
                    yield return chunkReturn;
                    chunkReturn = new List<T>(chunkSize);
                }
            }
            if (chunkReturn.Any())
            {
                yield return chunkReturn;
            }
        }
    }

If there is an error, I wish to run the chunk again. Is it possible to find the particular chunk number where we received the error and run that again ? The batches have to be executed in sequential order .So if batch#2 generates an error, then I need to be able to run 2 again, if it fails again. I just need to get out of the loop for good .

I have edited your title. Please see, "[Should questions include “tags” in their titles?](http://meta.stackexchange.com/questions/19190/)", where the consensus is "no, they should not". — John Saunders, Oct 25 '12 at 01:13
The short answer is that you cannot with foreach without using an external variable. Here's a long thread on the question: http://stackoverflow.com/questions/43021/how-do-you-get-the-index-of-the-current-iteration-of-a-foreach-loop — Candide, Oct 25 '12 at 01:14

score 4 · Answer 1 · answered Oct 25 '12 at 01:17

4

List<Chunk> failedChunks = new List<Chunk>();
foreach (var partialist in breaklistinchunks(chunksize))
{
    try
    {
        //do something
    }
    catch
    {
        //print error
        failedChunks.Add(partiallist);
    }

}

// attempt to re-process failed chunks here

answered Oct 25 '12 at 01:17

Heather

2,448
1
21
32

Based on his post, I think "Chunk" should be List. Also, if you explicitly want the index in a foreach loop you can do BreakListIntoChunks(size).Select((chunk, index) => new { chunk, index }) – ChaseMedallion Oct 25 '12 at 01:22

score 2 · Accepted Answer · edited May 23 '17 at 10:24

I propose this answer based on your comment to Aaron's answer.

The batches have to be executed in sequential order .So if 2 is a problem , then I need to be able to run 2 again, if it fails again. I just need to get out of the loop for good.

foreach (var partialist in breaklistinchunks(chunksize))
{
    int fails = 0;
    bool success = false;

    do
    {
        try
        {
            // do your action
            success = true; // should be on the last line before the 'catch'
        }
        catch
        {
            fails += 1;
            // do something about error before running again
        }
    }while (!success && fails < 2);

    // exit the iteration if not successful and fails is 2
    if (!success && fails >= 2)
        break;
}

Aaron Anodide · Answer 3 · 2012-10-25T01:57:26.250

I made a possible solution for you if you don't mind switching from Enumerable to Queue, which kind of fits given the requirements...

void Main()
{
    var list = new Queue<int>();
    list.Enqueue(1);
    list.Enqueue(2);
    list.Enqueue(3);
    list.Enqueue(4);
    list.Enqueue(5);

    var random = new Random();

    int chunksize = 2;
    foreach (var chunk in list.BreakListinChunks(chunksize))
    {
        foreach (var item in chunk)
        {   
            try
            {           
                if(random.Next(0, 3) == 0) // 1 in 3 chance of error
                    throw new Exception(item + " is a problem");
                else
                    Console.WriteLine (item + " is OK");
            }
            catch (Exception ex)
            {
                Console.WriteLine (ex.Message);
                list.Enqueue(item);
            }
        }
    }
}

public static class IEnumerableExtensions
{   
    public static IEnumerable<List<T>> BreakListinChunks<T>(this Queue<T> sourceList, int chunkSize)
    {
        List<T> chunkReturn = new List<T>(chunkSize);
        while(sourceList.Count > 0)
        {
            chunkReturn.Add(sourceList.Dequeue());
            if (chunkReturn.Count == chunkSize || sourceList.Count == 0)
            {
                yield return chunkReturn;
                chunkReturn = new List<T>(chunkSize);
            }       
        }
    }
}

Outputs

1 is a problem
2 is OK
3 is a problem
4 is a problem
5 is a problem
1 is a problem
3 is OK
4 is OK
5 is OK
1 is a problem
1 is OK

The batches have to be executed in sequential order .So if 2 is a problem , then I need to be able to run 2 again, if it fails again. I just need to get out of the loop for good. — user1110790, Oct 25 '12 at 01:52
I'm not sure that can fit with the idea of yielding lists which is the only way I know of to process in chunks... it's interesting to me because it seems i'm always writing code that does this type of thing though so if I have any ideas I'll let you know — Aaron Anodide, Oct 25 '12 at 02:05

score 0 · Answer 4 · answered Oct 25 '12 at 01:12

0

One possibility would be to use a for loop instead of a foreach loop and use the counter as a means to determine where an error occurred. Then you could continue from where you left off.

answered Oct 25 '12 at 01:12

Inisheer

19,061
9
47
81

Do you mean something like ? for (int i=0 ; i – user1110790 Oct 25 '12 at 01:15
Yes, although your return type must be accessible by index (ie. a List<> for example). – Inisheer Oct 25 '12 at 01:18

score 0 · Answer 5 · answered Oct 25 '12 at 02:45

You can use break to exit out of the loop as soon as a chunk fails twice:

foreach (var partialList in breaklistinchunks(chunksize))
{
    if(!TryOperation(partialList) && !TryOperation(partialList))
    {
        break;
    }
}

private bool TryOperation<T>(List<T> list)
{
    try
    {
        // do something
    }
    catch
    {
        // print error
        return false;
    }
    return true;
}

You could even make the loop into a one-liner with LINQ, but it is generally bad practice to combine LINQ with side-effects, and it's not very readable:

breaklistinchunks(chunksize).TakeWhile(x => TryOperation(x) || TryOperation(x));

ListGet Chunk Number being executed

5 Answers5