Filtering duplicates from a type collection using LINQ

Question

I'm filtering a list by grouping on two parameters and selecting the most recent type in the sub-grouplist based on createdate (using first()). This eliminates the duplicates on the x.application and x.externalid properties.

var list = ((List<SomeType>)xDic)
            .GroupBy(x => new {x.Application, x.ExternalID})
            .OrderByDescending(z => z.First().CreateDate)
            .Select(y => y.First()).ToList();

What I am having trouble with is defining yet another combination of properties (x.application and x.externaldisplayid) to filter with and group by to take the first one.

To summarize, I need to get a unique List of SomeTypes by filtering out any duplicates based on the ((x.application/x.externalid) OR (x.application/x.externaldisplayid)) combinations.

Example set:
{ "extID": 1234, "extDspID" : 111, "App" : "Test", "CreateDate": 2/01/2015}
{ "extID": 1234, "extDspID" : 5, "App" : "Test", "CreateDate": 1/01/2015}
{ "extID": 012, "extDspID" : 90, "App" : "Mono", "CreateDate": 6/06/2015}
{ "extID": 999, "extDspID" : 78, "App" : "Epic", "CreateDate": 8/08/2015}
{ "extID": 333, "extDspID" : 78, "App" : "Epic", "CreateDate": 8/12/2015}
{ "extID": 345, "extDspID" : 33, "App" : "Test", "CreateDate": 2/01/2015}
{ "extID": 666, "extDspID" : 33, "App" : "Test", "CreateDate": 1/01/2015}

desired result:
{ "extID": 1234, "extDspID" : 111, "App" : "Test", "CreateDate": 2/01/2015}
{ "extID": 012, "extDspID" : 90, "App" : "Mono", "CreateDate": 6/06/2015}
{ "extID": 333, "extDspID" : 78, "App" : "Epic", "CreateDate": 8/12/2015}
{ "extID": 345, "extDspID" : 33, "App" : "Test", "CreateDate": 2/01/2015}

Answer 1

First, declare two equality compareres to specify your two conditions like this:

public class MyEqualityComparer1 : IEqualityComparer<SomeType>
{
    public bool Equals(SomeType x, SomeType y)
    {
        return x.Application == y.Application && x.ExternalID == y.ExternalID;
    }

    public int GetHashCode(SomeType obj)
    {
        return (obj.Application + obj.ExternalID).GetHashCode();
    }
}

public class MyEqualityComparer2 : IEqualityComparer<SomeType>
{
    public bool Equals(SomeType x, SomeType y)
    {
        return x.Application == y.Application && x.ExternalDisplayId == y.ExternalDisplayId;
    }

    public int GetHashCode(SomeType obj)
    {
        return (obj.Application + obj.ExternalDisplayId).GetHashCode();
    }
}

Then, order your list by CreatedDate and then use Distinct to filter your list like this:

var result = xDic
    .OrderByDescending(x => x.CreateDate)
    .Distinct(new MyEqualityComparer1())
    .Distinct(new MyEqualityComparer2());

The Distinct method should remove the later items, so we should be able to depend on the fact that we used OrderByDescending to make sure that Distinct will remove items with the less recent CreatedTime.

However, since the documentation of Distinct do not guarantee this, you can use a custom distinct method like this:

public static class Extensions
{
    public static IEnumerable<T> OrderedDistinct<T>(this IEnumerable<T> enumerable, IEqualityComparer<T> comparer)
    {
        HashSet<T> hash_set = new HashSet<T>(comparer);

        foreach(var item in enumerable)
            if (hash_set.Add(item))
                yield return item;
    }
}

And use it like this:

var result = xDic
    .OrderByDescending(x => x.CreateDate)
    .OrderedDistinct(new MyEqualityComparer1())
    .OrderedDistinct(new MyEqualityComparer2());

Answer 2

The current accepted answer will not sort your "SomeType" objects correctly and so won't produce your desired result set.

I've implemented a solution here:

https://dotnetfiddle.net/qBkIXo

I too based my solution on Distinct (see MSDN documentation here). The way I generate the hash is based on this neat approach which uses an anonymous type e.g.

public int GetHashCode(SomeType sometype)
{
 //Calculate the hash code for the SomeType.
 return new { sometype.Application, sometype.ExternalID }.GetHashCode();
}

To achieve the correct desired results a combination of grouping, ordering and use of distinct needs to be applied e.g.

    var noduplicates = products.GroupBy(x => new {x.Application, x.ExternalDisplayId})
        .Select(y => y.OrderByDescending(x => x.CreateDate).First())
        .ToList()
        .Distinct(new ApplicationExternalDisplayIdComparer())
        .GroupBy(x => new {x.Application, x.ExternalID})
        .Select(y => y.OrderByDescending(x => x.CreateDate).First())
        .ToList()
        .Distinct(new ApplicationExternalIDComparer());

As you'll see in the fiddle output, this gives the results you are expecting.