LINQ query sometimes (randomly, for the exact same data) throwing NullReferenceException

Question

I am using LINQ-to-Entities (EF 6.1.3) to perform the following query:

var users = msgList.Select(m => m.From)
                    .Union(msgList.Select(m => m.To))
                    .Distinct()
                    .Where(u => u.ID != userId) //userId is an assigned local var.
                    .ToList();

msgList is a List (already fetched, not a queryable and lazy loading is off) of Messages which consists of some fields like From and To which are guaranteed to be non-null. Both From and To were Included in the original query, so they are guaranteed to be non-null.

My User object is also guaranteed to be non-null, so there's nothing that can actually be null.

However, this line is sometimes throwing a null pointer exception, and sometimes executing perfectly with the exact same user, exact same database, exactly same data (nothing altered). Load is not an issue as it's a code not yet in production and I'm the only one testing it.

The exception seems to be thrown at the Where call:

at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext() 
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)

How can this happen?

UPDATE: This is of course not a duplicate of What is a NullReferenceException, and how do I fix it?. Any sane developer with even a little knowledge in .NET/C#/OOP knows what that error is and that this question has nothing to do with it, even though it involves that exception as a part of it.

UPDATE 2: I've switched it to assigning to a list each line, as suggested below:

var msgListSelection = msgList.Select(m => m.From).ToList();
var union = msgListSelection.Union(msgList.Select(m => m.To)).ToList();
var distinct = union.Distinct().ToList();
var where = distinct.Where(u => u.ID != userId).ToList();
var users = where;

The exception occurs at the where line:

var where = distinct.Where(u => u.ID != User.ID).ToList();

If distinct returned null, it would have been thrown on ToList call of var distinct = union.Distinct().ToList(); on the line above.

Am I missing something?

UPDATE 2: My User class is a POCO C# type mapped to an Entity type in my database which has an ID property of long, and my Message class is again a POCO type mapped in Entity Framework, with navigation properties From and To to some User instances guaranteed to be non-null. They are annotated as Required and I've also checked them at the database level just to be sure.

UPDATE 3: My EF context lives from the beginning of the request (set at a delegating handler in the beginning of the request) to the end. I don't think the problem is related to the lifespan of the DbContext as there are many controllers with the same mechanism with tens of methods that access the context, and I'm only having such problem with this particular method.

UPDATE 4: I've added a null check on distincts:

var distinct = union.Distinct().ToList();
if(distinct == null)
{
    throw new Exception("distinct was null");
}
var where = distinct.Where(u => u.ID != userId).ToList();

It seems to pass that point with no problem, but throw the null pointer exception at the last line var where = distinct.Where(u => u.ID != userId).ToList(); which sorts out the possibility that distinct may be null.

UPDATE 5: I've wrote an API testing tool and sent about 250 requests to the same endpoint with the same user. The first one failed with this error, and all the rest succeeded successfully. There seems to be a problem with the first request.

Answer 1

You may be experiencing what is caused by the closure principle. You reference the User property in your LINQ query. Because the LINQ query in itself is executed as an (anonymous) method delegate, the closure principle applies.

Quoting the above link:

In essence, a closure is a block of code which can be executed at a later time, but which maintains the environment in which it was first created - i.e. it can still use the local variables etc of the method which created it, even after that method has finished executing.

The usage of the User property is subject to this principle. Its value can have changed upon the execution of the LINQ query. To protect against this, the User property should be copied to a local variable and that referenced in the LINQ query. Like so:

var user = User;
var users = msgList.Select(m => m.From)
                   .Union(msgList.Select(m => m.To))
                   .Distinct()
                   .Where(u => u.ID != user.ID)
                   .ToList();

Update

When using a local reference copy to the user property, another possibility for the NullReferenceException may lie with the Select-Union-Distinct methods. When calling ToList, the Where clause is executed on all items in the union of the two Select clauses. By default, Distinct executes the Equals method from the IQuality interface, which would be called on the elements from Select(m => m.From) . If this element is null, it would cause the NullReferenceException.