Using Pipes to read and write binary data in Haskell

Question

I am trying to read and write very many ints in constant memory. I have figured out how to write the ints to memory but have not figured out how to read them back.

import Control.Lens (zoom)
import System.IO (IOMode(..), withFile)
import Pipes
import qualified Pipes.Prelude as P
import qualified Pipes.ByteString as PB    
import qualified Pipes.Parse as P
import qualified Pipes.Binary as P

intStream :: Monad m => Proxy x' x () Int m b
intStream = go (0 :: Int) where
   go i = yield i >> go (i + 1)

decoder :: Monad m => Int ->  P.Parser P.ByteString m [Int]
decoder n = zoom (P.decoded . P.splitAt n) P.drawAll

main :: IO ()
main = do
    withFile "ints" WriteMode $ \h -> do
         runEffect $ for intStream P.encode >-> P.take 10000 >-> PB.toHandle h
    withFile "ints" ReadMode $ \h -> do
         xs <- P.evalStateT (decoder 10000000) (PB.fromHandle h)
         print xs

I got the decoder function from the documentation for Pipes.Binary. However it uses drawAll which according to the documentation drawAll is not idiomatic use of Pipes and is provided for testing purposes.

My question is how to modify decoder so that it doesn't use drawAll and thus does not load all the values of xs into memory. So instead of printing the list of xs I could P.map print over a stream of decoded ints being read from the file.

score 6 · Accepted Answer · answered Jul 22 '15 at 17:51

The docs say that decoded is a lens from a stream of bytes to a stream of decoded values. We can get the latter out of the former using view from lens:

decoder :: Monad m => Int -> Producer P.ByteString m a -> Producer Int m ()
decoder n p = void (view P.decoded p) >-> P.take n

main :: IO ()
main = do
    withFile "ints" WriteMode $ \h -> do
         runEffect $ for intStream P.encode >-> P.take 10000 >-> PB.toHandle h
    withFile "ints" ReadMode $ \h -> do
         runEffect $ decoder 10000 (PB.fromHandle h) >-> P.print

I don't have much experience with pipes, I just followed the types here. The program seems to function as intended though.

I try to follow the types but sometimes it gets difficult when the types start looking like alphabet soup ;) — Justin Raymond, Jul 22 '15 at 18:35

Using Pipes to read and write binary data in Haskell

1 Answers1