0

I have a Delphi app that references a datafile of 28-byte records. The file is written sequentially but read randomly. The datafile is split into N physical files which are rolled over at 10 megs or so to provide some insurance against disk problems, and because we are only ever writing to the most recent one, and I found it became slower and slower to write to if it were allowed to grow to big. On startup I read the entire file set and build an index so that I can quickly know which file to seek into given a virtual record number.

As part of the splitting into N files I implemented a read cache. I realise now that Windows does a fair amount of caching on it's own, and I wonder if I'm gaining anything by sticking another cache between myself and the disk files.

Any thoughts appreciated.

skaffman
  • 381,978
  • 94
  • 789
  • 754
rossmcm
  • 5,184
  • 7
  • 51
  • 108

1 Answers1

2

No. Use file mappings to use the existing file cache more efficiently.

Ignacio Vazquez-Abrams
  • 699,552
  • 132
  • 1,235
  • 1,283
  • Thanks @Ignacio. Why is file mapping any faster than just relying on the Windows native disk file caching? And how would I interface to it from Delphi - presumably there is a Windows API? – rossmcm Feb 08 '11 at 12:58
  • It *isn't* faster than file caching; it *is* the file cache. Instead of reading from and writing to the file, you manipulate the data within the file cache directly, and the OS handles the rest. I'm sure Delphi exposes the appropriate API calls in `Windows`; if not, complain to . – Ignacio Vazquez-Abrams Feb 08 '11 at 13:12
  • @Ignacio, Hmmm.. so where is the advantage? At present I might write Seek (f, RecordNo) ; Read (f, FRecord) ;. To use file mapping, presumably I would set up a file map for each of my datafiles, and replace the seek/read by a file mapping API call. Why will that be any faster than a seek/read (assuming that the data I want is in the Windows cache)? – rossmcm Feb 08 '11 at 13:30
  • The advantage is that you *don't* have to seek, read, or write; it's all done via memory manipulation, which is much faster. – Ignacio Vazquez-Abrams Feb 08 '11 at 13:31
  • @Ignacio, My understanding now of file mappings is that if I set up mappings for each of the physical datafiles that make up my big virtual datafile, memory will only actually be allocated when I reference those, i.e. creating file mappings for 1Gb of datafiles will not immediately use up 1Gb of memory. Eventually however, I will reference all of the files. What limits the amount of memory allocated before the OS starts evicting existing blocks of memory? – rossmcm Feb 13 '11 at 08:00
  • 1
    Your limit is the total amount of physical memory, as always. – Ignacio Vazquez-Abrams Feb 13 '11 at 12:49
  • @Ignacio, Say that had occurred - my app had used mapping up to the point where there was no physical memory left. If another app requested memory, Windows would presumably evict the oldest of my blocks. In other words, would the two apps happily co-exist and eventually share the available physical memory? – rossmcm Feb 13 '11 at 15:34
  • 1
    Sure, part of the mapping would be swapped out or discarded, and reloaded when a page fault occurred trying to access the missing part, with some other chunk of memory swapped out or discarded in turn. – Ignacio Vazquez-Abrams Feb 13 '11 at 18:49