I'm writing a program that will do high speed data acquisition. The acquisition card can run at up 6.8 GB/s (It's on PCIe3 x8). Right now I'm trying to stream to a RAM disk to see the max write speed I can achieve with Python.
The card is going to give me 5-10 MB blocks, which I can then write somewhere.
I wrote this piece of code, which writes a 10MB block 500 times to a binary file. I'm using Anaconda2 on Windows 7 64-bit, and I used the profiler from Anaconda's accelerate.
block = 'A'*10*1024*1024
filename = "R:\\test"
f = os.open(filename, os.O_CREAT| os.O_BINARY|os.O_TRUNC|os.O_WRONLY|os.O_SEQUENTIAL)
p = profiler.Profile(signatures=False)
p.enable()
start = time.clock()
for x in range(500):
os.write(f,block)
transferTime_sec = time.clock() - start
p.disable()
p.print_stats()
print('\nwrote %f MB' % (os.stat(filename).st_size/(1024*1024)))
I tested this on a RAM disk (R:\) and I got the following output:
So I figured, I'm getting something around 2.5 GB/s on RAM. which is not bad but far from max RAM throughput still, but the numbers are consistent. So the low throughput is one problem.
The second problem is, when I test this code with a PCIe SSD (which I had benchmarked with another software at 1090 MB/s sequential write), it gives comparable figures.
This makes me think that it's caching and/or buffering (?) and so I'm just not measuring actual IO. I'm not sure what's going on really as I'm fairly new to python.
So my main question is how to achieve max write speeds, and a side question is why am I getting these numbers?