I am building an app that will store stock tick data. one of my methods needs to take an array of items, currently c. 100,000 items but will be over 1,000,000 at a time, and add them to an entity database. this is my current add logic:-
var tickContext = new DataAccess();
// array is created, add it to the entity database
for (int j = 0; j < tickDataArray.Length; j++)
{
MarketTickData temp2 = new MarketTickData();
if(j > 1)
temp2 = tickDataArray[j-1];
MarketTickData temp = tickDataArray[j];
tickContext.TickBarData.Add(tickDataArray[j]);
tickContext.Configuration.AutoDetectChangesEnabled = false;
tickContext.Configuration.ValidateOnSaveEnabled = false;
if (j % 200 == 0)
{
tickContext.SaveChanges();
tickContext.Dispose();
tickContext = new DataAccess();
}
}
// add remaining items to database
tickContext.SaveChanges();
tickContext.Configuration.AutoDetectChangesEnabled = true;
tickContext.Configuration.ValidateOnSaveEnabled = true;
When I test my logic on various sizes of tickContext before saving I am not seeing huge improvments in preformance. At a Context size of 50 my adds are taking 47 Seconds, 1000 is 54 Seconds and the best is 200 items taking 44 seconds. But it is still relatively slow.
Is there a way to improve this ?