[EDIT]
Thanks to @VilleKrumlinde I have fixed a bug that I accidentally introduced earlier when trying to avoid a Code Analysis warning. I was accidentally turning on "overlapped" file handling, which kept resetting the file length. That is now fixed, and you can call FastWrite() multiple times for the same stream without issues.
[End Edit]
Overview
I'm doing some timing tests to compare two different ways of writing arrays of structs to disk. I believe that the perceived wisdom is that I/O costs are so high compared to other things that it isn't worth spending too much time optimising the other things.
However, my timing tests seem to indicate otherwise. Either I'm making a mistake (which is entirely possible), or my optimisation really is quite significant.
History
First some history: This FastWrite() method was originally written years ago to support writing structs to a file that was consumed by a legacy C++ program, and we are still using it for this purpose. (There's also a corresponding FastRead() method.) It was written primarily to make it easier to write arrays of blittable structs to a file, and its speed was a secondary concern.
I've been told by more than one person that optimisations like this aren't really much faster than just using a BinaryWriter, so I've finally bitten the bullet and performed some timing tests. The results have surprised me...
It appears that my FastWrite() method is 30 - 50 times faster than the equivalent using BinaryWriter. That seems ridiculous, so I'm posting my code here to see if anyone can find the errors.
System Specification
- Tested an x86 RELEASE build, run from OUTSIDE the debugger.
- Running on Windows 8, x64, 16GB memory.
- Run on a normal hard drive (not an SSD).
- Using .Net 4 with Visual Studio 2012 (so .Net 4.5 is installed)
Results
My results are:
SlowWrite() took 00:00:02.0747141
FastWrite() took 00:00:00.0318139
SlowWrite() took 00:00:01.9205158
FastWrite() took 00:00:00.0327242
SlowWrite() took 00:00:01.9289878
FastWrite() took 00:00:00.0321100
SlowWrite() took 00:00:01.9374454
FastWrite() took 00:00:00.0316074
As you can see, that seems to show that the FastWrite() is 50 times faster on that run.
Here's my test code. After running the test, I did a binary comparison of the two files to verify that they were indeed identical (i.e. FastWrite() and SlowWrite() produced identical files).
See what you can make of it. :)
using System;
using System.ComponentModel;
using System.Diagnostics;
using System.IO;
using System.Runtime.InteropServices;
using System.Text;
using System.Threading;
using Microsoft.Win32.SafeHandles;
namespace ConsoleApplication1
{
internal class Program
{
[StructLayout(LayoutKind.Sequential, Pack = 1)]
struct TestStruct
{
public byte ByteValue;
public short ShortValue;
public int IntValue;
public long LongValue;
public float FloatValue;
public double DoubleValue;
}
static void Main()
{
Directory.CreateDirectory("C:\\TEST");
string filename1 = "C:\\TEST\\TEST1.BIN";
string filename2 = "C:\\TEST\\TEST2.BIN";
int count = 1000;
var array = new TestStruct[10000];
for (int i = 0; i < array.Length; ++i)
array[i].IntValue = i;
var sw = new Stopwatch();
for (int trial = 0; trial < 4; ++trial)
{
sw.Restart();
using (var output = new FileStream(filename1, FileMode.Create))
using (var writer = new BinaryWriter(output, Encoding.Default, true))
{
for (int i = 0; i < count; ++i)
{
output.Position = 0;
SlowWrite(writer, array, 0, array.Length);
}
}
Console.WriteLine("SlowWrite() took " + sw.Elapsed);
sw.Restart();
using (var output = new FileStream(filename2, FileMode.Create))
{
for (int i = 0; i < count; ++i)
{
output.Position = 0;
FastWrite(output, array, 0, array.Length);
}
}
Console.WriteLine("FastWrite() took " + sw.Elapsed);
}
}
static void SlowWrite(BinaryWriter writer, TestStruct[] array, int offset, int count)
{
for (int i = offset; i < offset + count; ++i)
{
var item = array[i]; // I also tried just writing from array[i] directly with similar results.
writer.Write(item.ByteValue);
writer.Write(item.ShortValue);
writer.Write(item.IntValue);
writer.Write(item.LongValue);
writer.Write(item.FloatValue);
writer.Write(item.DoubleValue);
}
}
static void FastWrite<T>(FileStream fs, T[] array, int offset, int count) where T: struct
{
int sizeOfT = Marshal.SizeOf(typeof(T));
GCHandle gcHandle = GCHandle.Alloc(array, GCHandleType.Pinned);
try
{
uint bytesWritten;
uint bytesToWrite = (uint)(count * sizeOfT);
if
(
!WriteFile
(
fs.SafeFileHandle,
new IntPtr(gcHandle.AddrOfPinnedObject().ToInt64() + (offset*sizeOfT)),
bytesToWrite,
out bytesWritten,
IntPtr.Zero
)
)
{
throw new IOException("Unable to write file.", new Win32Exception(Marshal.GetLastWin32Error()));
}
Debug.Assert(bytesWritten == bytesToWrite);
}
finally
{
gcHandle.Free();
}
}
[DllImport("kernel32.dll", SetLastError=true)]
[return: MarshalAs(UnmanagedType.Bool)]
private static extern bool WriteFile
(
SafeFileHandle hFile,
IntPtr lpBuffer,
uint nNumberOfBytesToWrite,
out uint lpNumberOfBytesWritten,
IntPtr lpOverlapped
);
}
}
Follow Up
I have also tested the code proposed by @ErenErsönmez, as follows (and I verified that all three files are identical at the end of the test):
static void ErenWrite<T>(FileStream fs, T[] array, int offset, int count) where T : struct
{
// Note: This doesn't use 'offset' or 'count', but it could easily be changed to do so,
// and it doesn't change the results of this particular test program.
int size = Marshal.SizeOf(typeof(TestStruct)) * array.Length;
var bytes = new byte[size];
GCHandle gcHandle = GCHandle.Alloc(array, GCHandleType.Pinned);
try
{
var ptr = new IntPtr(gcHandle.AddrOfPinnedObject().ToInt64());
Marshal.Copy(ptr, bytes, 0, size);
fs.Write(bytes, 0, size);
}
finally
{
gcHandle.Free();
}
}
I added a test for that code, and at the same time removed the lines output.Position = 0; so that the files now grow to 263K (which is a reasonable size).
With those changes, the results are:
NOTE Look at how much slower the FastWrite() times are when you don't keep resetting the file pointer back to zero!:
SlowWrite() took 00:00:01.9929327
FastWrite() took 00:00:00.1152534
ErenWrite() took 00:00:00.2185131
SlowWrite() took 00:00:01.8877979
FastWrite() took 00:00:00.2087977
ErenWrite() took 00:00:00.2191266
SlowWrite() took 00:00:01.9279477
FastWrite() took 00:00:00.2096208
ErenWrite() took 00:00:00.2102270
SlowWrite() took 00:00:01.7823760
FastWrite() took 00:00:00.1137891
ErenWrite() took 00:00:00.3028128
So it looks like you can achieve almost the same speed using Marshaling without having to use the Windows API at all. The only drawback is that Eren's method has to make a copy of the entire array of structs, which could be an issue if memory is limited.
