21

I've run into a case where something that worked fairly well with LINQ to SQL seems to be very obtuse (or maybe impossible) with the Entity Framework. Specifically, I've got an entity that includes a rowversion property (both for versioning and concurrency control). Something like:

public class Foo
{
  [Key]
  [MaxLength(50)]
  public string FooId { get; set; }

  [Timestamp]
  [ConcurrencyCheck]
  public byte[] Version { get; set; }
}

I would like to be able to take a entity as input, and find all of the other entities that are more recently updated. Something like:

Foo lastFoo = GetSomeFoo();
var recent = MyContext.Foos.Where(f => f.Version > lastFoo.Version);

Now, in the database this would work: two rowversion values can be compared to one another without any problems. And I've done a similar thing before using LINQ to SQL, which maps the rowversion to System.Data.Linq.Binary, which can be compared. (At least to the extent that the expression tree can be mapped back to the database.)

But in Code First, the type of the property must be byte[]. And two arrays can't be compared with the regular comparison operators. Is there some other way to write the comparison of the arrays that LINQ to Entities will understand? Or to coerce the arrays into other types so that the comparison can get past the compiler?

Sixten Otto
  • 14,750
  • 3
  • 46
  • 60
  • If you can tolerate the potential for an OCC overwrite (sub-millesecond fetch/update on the same column), you could avoid the rowversion type and use DateTime2 for your timestamps. This would allow you to perform the comparisons you indicated and also give you a realistic last-modified timestamp. – Brent M. Spell Sep 16 '11 at 02:20
  • If it comes to that, I'll probably keep the `rowversion` for concurrency, and add a date field for the querying I need to do. It's highly unlikely that I'd be updating more than one entity in the same millisecond, but I like the unambiguous nature of the `rowversion`. – Sixten Otto Sep 16 '11 at 16:13
  • Unfortunately, in that case, you'll also have to manually update the date/time field whenever you update the row. ConcurrencyCheck on the date/time property would do this automatically, and I don't think you can have more than one ConcurrencyCheck property on a class. – Brent M. Spell Sep 16 '11 at 20:55
  • That's a good point, but the updates in this particular app will be pretty controlled, so that won't be a very big burden. But, yeah, if we were doing piecemeal updates of the data in various places, it might be a pain to remember that. – Sixten Otto Sep 17 '11 at 15:02
  • [Opened an issue](https://github.com/aspnet/EntityFramework/issues/5936) for EF Core. – jnm2 Jul 01 '16 at 14:59

10 Answers10

12

Found a workaround that works perfectly! Tested on Entity Framework 6.1.3.

There's no way to use the < operator with byte arrays because the C# type system prevents that (as it should). But what you can do is build the exact same syntax using expressions, and there is a loophole that allows you to pull this off.

First step

If you don't want the full explanation, you can skip to the Solution section.

If you aren't familiar with expressions, here is MSDN's crash course.

Basically, when you type queryable.Where(obj => obj.Id == 1) the compiler really outputs the same thing as if you had typed:

var objParam = Expression.Parameter(typeof(ObjType));
queryable.Where(Expression.Lambda<Func<ObjType, bool>>(
    Expression.Equal(
        Expression.Property(objParam, "Id"),
        Expression.Constant(1)),
    objParam))

And that expression is what the database provider parses to create your query. This is obviously much more verbose than the original, but it also allows you do do meta-programming just like when you do reflection. The verbosity is the only downside to this method. It's a better downside than other answers here, like having to write raw SQL or not being able to use parameters.

In my case, I was already using expressions, but in your case the first step is to rewrite your query using expressions:

Foo lastFoo = GetSomeFoo();
var fooParam = Expression.Parameter(typeof(Foo));
var recent = MyContext.Foos.Where(Expression.Lambda<Func<Foo, bool>>(
    Expression.LessThan(
        Expression.Property(fooParam, nameof(Foo.Version)),
        Expression.Constant(lastFoo.Version)),
    fooParam));

This is how we get around the compiler error we get if we try to use < on byte[] objects. Now instead of a compiler error, we get a runtime exception because Expression.LessThan tries to find byte[].op_LessThan and fails at runtime. This is where the loophole comes in.

Loophole

To get rid of that runtime error, we will tell Expression.LessThan what method to use so that it doesn't try to find the default one (byte[].op_LessThan) which doesn't exist:

var recent = MyContext.Foos.Where(Expression.Lambda<Func<Foo, bool>>(
    Expression.LessThan(
        Expression.Property(fooParam, nameof(Foo.Version)),
        Expression.Constant(lastFoo.Version),
        false,
        someMethodThatWeWrote), // So that Expression.LessThan doesn't try to find the non-existent default operator method
    fooParam));

Great! Now all we need is MethodInfo someMethodThatWeWrote created from a static method with the signature bool (byte[], byte[]) so that the types match at runtime with our other expressions.

Solution

You need a small DbFunctionExpressions.cs. Here's a truncated version:

public static class DbFunctionExpressions
{
    private static readonly MethodInfo BinaryDummyMethodInfo = typeof(DbFunctionExpressions).GetMethod(nameof(BinaryDummyMethod), BindingFlags.Static | BindingFlags.NonPublic);
    private static bool BinaryDummyMethod(byte[] left, byte[] right)
    {
        throw new NotImplementedException();
    }

    public static Expression BinaryLessThan(Expression left, Expression right)
    {
        return Expression.LessThan(left, right, false, BinaryDummyMethodInfo);
    }
}

Usage

var recent = MyContext.Foos.Where(Expression.Lambda<Func<Foo, bool>>(
    DbFunctionExpressions.BinaryLessThan(
        Expression.Property(fooParam, nameof(Foo.Version)),
        Expression.Constant(lastFoo.Version)),            
    fooParam));
  • Enjoy.

Notes

Does not work on Entity Framework Core 1.0.0, but I opened an issue there for fuller support without the need for expressions anyway. (EF Core doesn't work because it goes through a stage where it copies the LessThan expression with the left and right parameters but doesn't copy the MethodInfo parameter we use for the loophole.)

jnm2
  • 7,157
  • 5
  • 57
  • 92
  • Other than the need to convert fairly concise LINQ expressions to much more verbose trees of Expression, this works beautifully in my isolated tests in LinqPad. The solution does *not* work out of the box with our in-memory unit tests (where it will actually call `BinaryDummyMethod`, and still needs some help to order the results). Finding workarounds for those things, though. – Sixten Otto Sep 13 '16 at 22:23
  • Thanks! The simplest solution is to implement `BinaryDummyMethod` via the built in array IStructuralComparer. All you need is a lexicographical comparer. However, if this `byte[]` represents a rowversion, [see this answer](http://stackoverflow.com/a/38149945) because there's something else that needs to be taken into consideration for rowversion comparisons. – jnm2 Sep 14 '16 at 02:14
  • Yeah, implementing a functional `BinaryDummyMethod` wasn't too rough (good call about endianness, though!), but the sorting is turning out to be an issue: doing it in memory requires a comparer, but LINQ can't map that to a store expression. That's outside the scope of this question though. – Sixten Otto Sep 14 '16 at 20:33
  • Follow up: http://stackoverflow.com/questions/39500470/order-entity-framework-query-by-rowversion-property-in-store-and-in-memory – Sixten Otto Sep 14 '16 at 22:16
  • By happenstance, I came across this answer yesterday to a similar question: http://stackoverflow.com/a/19402565/16892 . The approach is very similar to yours in principle, but without having to resort to direct construction of expression trees. Is there a flaw I'm missing? – Sixten Otto Sep 16 '16 at 21:59
  • @SixtenOtto that's cool! I can't say whether that approach has a flaw but my guess is it doesn't. – jnm2 Dec 08 '16 at 13:23
  • @divega Can you look at a second EF6 workaround someone found (http://stackoverflow.com/a/19402565) and confirm that there are no downsides to using it? If so, I'll update my answer. – jnm2 Dec 08 '16 at 13:26
  • hi, both of you provided excellent solution for this problem, thx a lot! There are some differences though. Solution from @jnm2 generates value directly in SQL and is not using EF parameter `@p__linq__0` - this avoids using parametrized SQL execution cache. On the other hand solution from @jnm2 allows creating generic version of this method like this `public static IQueryable RowVersionGreaterThenBinary(this IQueryable query, byte[] rowVersion) where T : IRowVersionedDbOject` (this does not work with solution from @BootGenius) Solution from @BootGenius is way simpler though. – Lukas K Mar 09 '17 at 19:31
  • additional difference is that the solution from @BootGenius generates more complicated SELECT ... FROM (SELECT ... FROM ... ). There is not that much performance impact of this, but someone might dislike it. – Lukas K Mar 09 '17 at 19:54
  • in Linq2Sql, I had to rename the `BinaryDummyMethod` to `op_LessThan' to make Linq2Sql believe that it's a real ` – Mahmood Dehghan Jan 26 '20 at 07:41
5

You can use SqlQuery to write the raw SQL instead of having it generated.

MyContext.Foos.SqlQuery("SELECT * FROM Foos WHERE Version > @ver", new SqlParameter("ver", lastFoo.Version));
Josh
  • 2,180
  • 3
  • 21
  • 24
  • Technically, yes. I have a few reservations about this, though. First, the API I'm exposing has to be the concrete `DbSet` and `DbContext`, and not the interfaces. Second, that in a Code First model, doing SQL queries smells pretty bad to me. More importantly, it seems that you have to select _exactly_ the set of rows needed to populate the entity, no more or less. And with Code First generating the schema automatically from a non-contrived model, that's potentially _much_ more complicated than just "select * from Foos". – Sixten Otto Sep 16 '11 at 20:30
  • This does suggest a possible workaround, though: add a method to my repository that does a query for the set of PK values, and then select the entities with the matching PKs in a second step. – Sixten Otto Sep 16 '11 at 20:34
  • Still don't feel like this is an optimal solution, but I did end up using the workaround I mentioned earlier, so I'm marking this as the answer. – Sixten Otto Nov 14 '11 at 16:37
  • 1
    @SixtenOtto I found something that is more optimal. It allows the use of much more complicated queries than the raw SQL method. [Check this out](http://stackoverflow.com/a/38152016/521757) and see if you prefer it. – jnm2 Jul 01 '16 at 18:41
3

You can accomplish this in EF 6 code-first by mapping a C# function to a database function. It took some tweaking and doesn't produce the most efficient SQL, but it gets the job done.

First, create a function in the database to test for a newer rowversion. Mine is

CREATE FUNCTION [common].[IsNewerThan]
(
    @CurrVersion varbinary(8),
    @BaseVersion varbinary(8)
) ...

When constructing your EF context, you'll have to manually define the function in the store model, like this:

private static DbCompiledModel GetModel()
{
    var builder = new DbModelBuilder();
    ... // your context configuration
    var model = builder.Build(...); 
    EdmModel store = model.GetStoreModel();
    store.AddItem(GetRowVersionFunctionDef(model));
    DbCompiledModel compiled = model.Compile();
    return compiled;
}

private static EdmFunction GetRowVersionFunctionDef(DbModel model)
{
    EdmFunctionPayload payload = new EdmFunctionPayload();
    payload.IsComposable = true;
    payload.Schema = "common";
    payload.StoreFunctionName = "IsNewerThan";
    payload.ReturnParameters = new FunctionParameter[]
    {
        FunctionParameter.Create("ReturnValue", 
            GetStorePrimitiveType(model, PrimitiveTypeKind.Boolean), ParameterMode.ReturnValue)
    };
    payload.Parameters = new FunctionParameter[]
    {
        FunctionParameter.Create("CurrVersion",  GetRowVersionType(model), ParameterMode.In),
        FunctionParameter.Create("BaseVersion",  GetRowVersionType(model), ParameterMode.In)
    };
    EdmFunction function = EdmFunction.Create("IsRowVersionNewer", "EFModel",
        DataSpace.SSpace, payload, null);
    return function;
}

private static EdmType GetStorePrimitiveType(DbModel model, PrimitiveTypeKind typeKind)
{
    return model.ProviderManifest.GetStoreType(TypeUsage.CreateDefaultTypeUsage(
        PrimitiveType.GetEdmPrimitiveType(typeKind))).EdmType;
}

private static EdmType GetRowVersionType(DbModel model)
{
    // get 8-byte array type
    var byteType = PrimitiveType.GetEdmPrimitiveType(PrimitiveTypeKind.Binary);
    var usage = TypeUsage.CreateBinaryTypeUsage(byteType, true, 8);

    // get the db store type
    return model.ProviderManifest.GetStoreType(usage).EdmType;
}

Create a proxy for the method by decorating a static method with the DbFunction attribute. EF uses this to associate the method with the named method in the store model. Making it an extension method produces cleaner LINQ.

[DbFunction("EFModel", "IsRowVersionNewer")]
public static bool IsNewerThan(this byte[] baseVersion, byte[] compareVersion)
{
    throw new NotImplementedException("You can only call this method as part of a LINQ expression");
}

Example

Finally, call the method from LINQ to entities in a standard expression.

    using (var db = new OrganizationContext(session))
    {
        byte[] maxRowVersion = db.Users.Max(u => u.RowVersion);
        var newer = db.Users.Where(u => u.RowVersion.IsNewerThan(maxRowVersion)).ToList();
    }

This generates the T-SQL to achieve what you want, using the context and entity sets you have defined.

WHERE ([common].[IsNewerThan]([Extent1].[RowVersion], @p__linq__0)) = 1',N'@p__linq__0 varbinary(8000)',@p__linq__0=0x000000000001DB7B
drew
  • 176
  • 11
  • I wasn't able to get this working in EF 5. There is little documentation I could find on manipulating the store model in code. It'd be much nicer if EF natively recognized that SQL can do the comparisons and allow GreaterThan expressions in LINQ for appropriately configured concurrency fields. Or, at least provide a way to map a LINQ expression to an inline SQL expression, which appears to be possible in Model-First. It'd be nice to have EF spit out the 'WHERE RowVersion > ...' automatically. – drew Nov 26 '13 at 19:11
2

I extended jnm2’s answer to hide the ugly expression code in a extension method

Usage:

ctx.Foos.WhereVersionGreaterThan(r => r.RowVersion, myVersion);

Extension Method:

public static class RowVersionEfExtensions
{


    private static readonly MethodInfo BinaryGreaterThanMethodInfo = typeof(RowVersionEfExtensions).GetMethod(nameof(BinaryGreaterThanMethod), BindingFlags.Static | BindingFlags.NonPublic);
    private static bool BinaryGreaterThanMethod(byte[] left, byte[] right)
    {
        throw new NotImplementedException();
    }

    private static readonly MethodInfo BinaryLessThanMethodInfo = typeof(RowVersionEfExtensions).GetMethod(nameof(BinaryLessThanMethod), BindingFlags.Static | BindingFlags.NonPublic);
    private static bool BinaryLessThanMethod(byte[] left, byte[] right)
    {
        throw new NotImplementedException();
    }

    /// <summary>
    /// Filter the query to return only rows where the RowVersion is greater than the version specified
    /// </summary>
    /// <param name="query">The query to filter</param>
    /// <param name="propertySelector">Specifies the property of the row that contains the RowVersion</param>
    /// <param name="version">The row version to compare against</param>
    /// <returns>Rows where the RowVersion is greater than the version specified</returns>
    public static IQueryable<T> WhereVersionGreaterThan<T>(this IQueryable<T> query, Expression<Func<T, byte[]>> propertySelector, byte[] version)
    {
        var memberExpression = propertySelector.Body as MemberExpression;
        if (memberExpression == null) { throw new ArgumentException("Expression should be of form r=>r.RowVersion"); }
        var propName = memberExpression.Member.Name;

        var fooParam = Expression.Parameter(typeof(T));
        var recent = query.Where(Expression.Lambda<Func<T, bool>>(
            Expression.GreaterThan(
                Expression.Property(fooParam, propName),
                Expression.Constant(version),
                false,
                BinaryGreaterThanMethodInfo),
            fooParam));
        return recent;
    }


    /// <summary>
    /// Filter the query to return only rows where the RowVersion is less than the version specified
    /// </summary>
    /// <param name="query">The query to filter</param>
    /// <param name="propertySelector">Specifies the property of the row that contains the RowVersion</param>
    /// <param name="version">The row version to compare against</param>
    /// <returns>Rows where the RowVersion is less than the version specified</returns>
    public static IQueryable<T> WhereVersionLessThan<T>(this IQueryable<T> query, Expression<Func<T, byte[]>> propertySelector, byte[] version)
    {
        var memberExpression = propertySelector.Body as MemberExpression;
        if (memberExpression == null) { throw new ArgumentException("Expression should be of form r=>r.RowVersion"); }
        var propName = memberExpression.Member.Name;

        var fooParam = Expression.Parameter(typeof(T));
        var recent = query.Where(Expression.Lambda<Func<T, bool>>(
            Expression.LessThan(
                Expression.Property(fooParam, propName),
                Expression.Constant(version),
                false,
                BinaryLessThanMethodInfo),
            fooParam));
        return recent;
    }



}
jnm2
  • 7,157
  • 5
  • 57
  • 92
innominate227
  • 9,681
  • 1
  • 14
  • 16
1

This method works for me and avoids tampering with the raw SQL:

var recent = MyContext.Foos.Where(c => BitConverter.ToUInt64(c.RowVersion.Reverse().ToArray(), 0) > fromRowVersion);

I would guess however raw SQL would be more efficient.

Adsborough
  • 53
  • 1
  • 5
  • EF6 gives NotSupportedException: "LINQ to Entities does not recognize the method 'UInt64 ToUInt64(Byte[], Int32)' method, and this method cannot be translated into a store expression." – jnm2 Jul 01 '16 at 12:56
  • 1
    It almost works, but you have to materialize the collection (ex: ToList) before execute this linq expression. – rd_rscan Jan 20 '17 at 20:12
0

I found this workaround usefull:

byte[] rowversion = BitConverter.GetBytes(revision);

var dbset = (DbSet<TEntity>)context.Set<TEntity>();

string query = dbset.Where(x => x.Revision != rowversion).ToString()
    .Replace("[Revision] <> @p__linq__0", "[Revision] > @rowversion");

return dbset.SqlQuery(query, new SqlParameter("rowversion", rowversion)).ToArray();
Jekas
  • 578
  • 4
  • 15
0

I ended up executing a raw query:
ctx.Database.SqlQuery("SELECT * FROM [TABLENAME] WHERE(CONVERT(bigint,@@DBTS) >" + X)).ToList();

  • you should not use SELECT * - this blocks usage of indexes on the table. Just a side note, that has nothing to do with the topic here :) – Lukas K Mar 09 '17 at 16:27
0

That is the best solution, but have a performance issue. The parameter @ver will be cast. Cast columns in where clause are bad to the database.

Type conversion in expression may affect "SeekPlan" in query plan choice

MyContext.Foos.SqlQuery("SELECT * FROM Foos WHERE Version > @ver", new SqlParameter("ver", lastFoo.Version));

Without cast. MyContext.Foos.SqlQuery("SELECT * FROM Foos WHERE Version > @ver", new SqlParameter("ver", lastFoo.Version).SqlDbType = SqlDbType.Timestamp);

Andre Mendonca
  • 516
  • 5
  • 11
0

Here is yet another workaround available to EF 6.x that doesn't require creating functions in the database but uses model defined functions instead.

Function definitions (this goes inside the section in your CSDL file, or inside section if you are using EDMX files):

<Function Name="IsLessThan" ReturnType="Edm.Boolean" >
  <Parameter Name="source" Type="Edm.Binary" MaxLength="8" />
  <Parameter Name="target" Type="Edm.Binary" MaxLength="8" />
  <DefiningExpression>source &lt; target</DefiningExpression>
</Function>
<Function Name="IsLessThanOrEqualTo" ReturnType="Edm.Boolean" >
  <Parameter Name="source" Type="Edm.Binary" MaxLength="8" />
  <Parameter Name="target" Type="Edm.Binary" MaxLength="8" />
  <DefiningExpression>source &lt;= target</DefiningExpression>
</Function>
<Function Name="IsGreaterThan" ReturnType="Edm.Boolean" >
  <Parameter Name="source" Type="Edm.Binary" MaxLength="8" />
  <Parameter Name="target" Type="Edm.Binary" MaxLength="8" />
  <DefiningExpression>source &gt; target</DefiningExpression>
</Function>
<Function Name="IsGreaterThanOrEqualTo" ReturnType="Edm.Boolean" >
  <Parameter Name="source" Type="Edm.Binary" MaxLength="8" />
  <Parameter Name="target" Type="Edm.Binary" MaxLength="8" />
  <DefiningExpression>source &gt;= target</DefiningExpression>
</Function>

Note that I haven't written the code to create the functions using the APIs available in Code First, but similar to code to what Drew proposed or the model conventions I wrote some time ago for UDFs https://github.com/divega/UdfCodeFirstSample, should work

Method definition (this goes in your C# source code):

using System.Collections;
using System.Data.Objects.DataClasses;

namespace TimestampComparers
{
    public static class TimestampComparers
    {

        [EdmFunction("TimestampComparers", "IsLessThan")]
        public static bool IsLessThan(this byte[] source, byte[] target)
        {
            return StructuralComparisons.StructuralComparer.Compare(source, target) == -1;
        }

        [EdmFunction("TimestampComparers", "IsGreaterThan")]
        public static bool IsGreaterThan(this byte[] source, byte[] target)
        {
            return StructuralComparisons.StructuralComparer.Compare(source, target) == 1;
        }

        [EdmFunction("TimestampComparers", "IsLessThanOrEqualTo")]
        public static bool IsLessThanOrEqualTo(this byte[] source, byte[] target)
        {
            return StructuralComparisons.StructuralComparer.Compare(source, target) < 1;
        }

        [EdmFunction("TimestampComparers", "IsGreaterThanOrEqualTo")]
        public static bool IsGreaterThanOrEqualTo(this byte[] source, byte[] target)
        {
            return StructuralComparisons.StructuralComparer.Compare(source, target) > -1;
        }
    }
}

Note also that I have defined the methods as extension methods over byte[], although this is not necessary. I also provided implementations for the methods so that they work if you evaluate them outside queries, but you can choose as well to throw NotImplementedException. When you use these methods in LINQ to Entities queries, we will never really invoke them. Also not that I have made the first argument for EdmFunctionAttribute “TimestampComparers”. This has to match the namespace specified in the section of your conceptual model.

Usage:

using System.Linq;

namespace TimestampComparers
{
    class Program
    {
        static void Main(string[] args)
        {
            using (var context = new OrdersContext())
            {
                var stamp = new byte[] { 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, };

                var lt = context.OrderLines.FirstOrDefault(l => l.TimeStamp.IsLessThan(stamp));
                var lte = context.OrderLines.FirstOrDefault(l => l.TimeStamp.IsLessThanOrEqualTo(stamp));
                var gt = context.OrderLines.FirstOrDefault(l => l.TimeStamp.IsGreaterThan(stamp));
                var gte = context.OrderLines.FirstOrDefault(l => l.TimeStamp.IsGreaterThanOrEqualTo(stamp));

            }
        }
    }
}
divega
  • 5,771
  • 26
  • 30
  • 1
    Code first is the only thing I'm interested in, and I'm guessing the same goes for a lot of people. It's not obvious to me how to go from the UDF sample to a custom `DefiningExpression` that isn't related to a SQL function. – jnm2 Jul 21 '16 at 22:39
  • I investigated a bit and remembered that MDFs are actually blocked in Code First by a model serialization bug. The PR at http://entityframework.codeplex.com/SourceControl/network/forks/BrandonDahler/EntityFramework/contribution/8302 contains a fix for the issue and an example of how it would work. Hopefully we will include this PR in EF 6.2. – divega Jul 30 '16 at 00:10
0

(The following answer by Damon Warren is copied over from here):

Here is what we did to solve this:

Use a compare extension like this:

public static class EntityFrameworkHelper
    {
        public static int Compare(this byte[] b1, byte[] b2)
        {
            throw new Exception("This method can only be used in EF LINQ Context");
        }
    }

Then you can do

byte[] rowversion = .....somevalue;
_context.Set<T>().Where(item => item.RowVersion.Compare(rowversion) > 0);

The reason this works without a C# implementation is because the compare extension method is never actually called, and EF LINQ simplifies x.compare(y) > 0 down to x > y

John
  • 5,749
  • 3
  • 36
  • 75