Numpy and 16-bit PGM

Question

What is an efficient and clear way to read 16-bit PGM images in Python with numpy?

I cannot use PIL to load 16-bit PGM images due to a PIL bug. I can read in the header with the following code:

dt = np.dtype([('type', 'a2'),
               ('space_0', 'a1', ),
               ('x', 'a3', ),
               ('space_1', 'a1', ),
               ('y', 'a3', ),
               ('space_2', 'a1', ),
               ('maxval', 'a5')])
header = np.fromfile( 'img.pgm', dtype=dt )
print header

This prints the correct data: ('P5', ' ', '640', ' ', '480', ' ', '65535') But I have a feeling that is not quite the best way. And beyond that, I'm having trouble how to figure out how to read in the following data of x by y (in this case 640x480) by 16-bit with the offset of size(header).

EDIT: IMAGE ADDED

MATLAB code to read and display the image is:

I = imread('foo.pgm'); 
imagesc(I);

And looks like this:

Answer 1

import re
import numpy

def read_pgm(filename, byteorder='>'):
    """Return image data from a raw PGM file as numpy array.

    Format specification: http://netpbm.sourceforge.net/doc/pgm.html

    """
    with open(filename, 'rb') as f:
        buffer = f.read()
    try:
        header, width, height, maxval = re.search(
            b"(^P5\s(?:\s*#.*[\r\n])*"
            b"(\d+)\s(?:\s*#.*[\r\n])*"
            b"(\d+)\s(?:\s*#.*[\r\n])*"
            b"(\d+)\s(?:\s*#.*[\r\n]\s)*)", buffer).groups()
    except AttributeError:
        raise ValueError("Not a raw PGM file: '%s'" % filename)
    return numpy.frombuffer(buffer,
                            dtype='u1' if int(maxval) < 256 else byteorder+'u2',
                            count=int(width)*int(height),
                            offset=len(header)
                            ).reshape((int(height), int(width)))


if __name__ == "__main__":
    from matplotlib import pyplot
    image = read_pgm("foo.pgm", byteorder='<')
    pyplot.imshow(image, pyplot.cm.gray)
    pyplot.show()

Answer 2

I'm not terribly familar with the PGM format, but generally speaking you'd just use numpy.fromfile. fromfile will start at whatever position the file pointer you pass to it is at, so you can simply seek (or read) to the end of the header, and then use fromfile to read the rest in.

You'll need to use infile.readline() instead of next(infile).

import numpy as np

with open('foo.pgm', 'r') as infile:
    header = infile.readline()
    width, height, maxval = [int(item) for item in header.split()[1:]]
    image = np.fromfile(infile, dtype=np.uint16).reshape((height, width))

On a side note, the "foo.pgm" file you pointed to in your comment appears to specify the wrong number of rows in the header.

If you're going to be reading in a lot of files that potentially have that problem, you can just pad the array with zeros or truncate it, like this.

import numpy as np

with open('foo.pgm', 'r') as infile:
    header = next(infile)
    width, height, maxval = [int(item) for item in header.split()[1:]]
    image = np.fromfile(infile, dtype=np.uint16)
    if image.size < width * height:
        pad = np.zeros(width * height - image.size, dtype=np.uint16)
        image = np.hstack([image, pad])
    if image.size > width * height:
        image = image[:width * height]
    image = image.reshape((height, width))

Answer 3

from here I understand that the header information can be separated by either spaces, carriage returns or others. If yours is separated by spaces (inform me if otherwise) you can do:

with open('img.pgm') as f:
    lines = f.readlines()
    data = np.array([line.split() for line in lines[1:]], dtype=np.int16).T

your data is now an array in int16 format!

Suppose you are still interested in the header information, you can do:

class Header(object):
    def __init__(self, type, width, height, maxval):
        self.type = type
        self.width = int(width)
        self.height = int(height)
        self.maxval = int(maxval)

h = Header(*lines[0].split()[:4])

so that you can check the image data against the read lines:

assert (h.width, h.height) == data.shape    
assert h.maxval >= data.max()

Edit: with the image data being binary, the file has to be opened as 'rb' and read from after the header information:

import numpy as np

def as_array(filepath):
    f = open(filepath, 'r')
    w, h = size = tuple(int(v) for v in next(f).split()[1:3])
    data_size = w * h * 2

    f.seek(0, 2)
    filesize = f.tell()
    f.close()
    i_header_end = filesize - (data_size)

    f = open(filepath, 'rb')
    f.seek(i_header_end)
    buffer = f.read()
    f.close()

    # convert binary data to an array of the right shape
    data = np.frombuffer(buffer, dtype=np.uint16).reshape((w, h))

    return data

a = as_array('foo.pgm')

Answer 4

Indeed, the 'string' after the header is a binary in your file. I solved that below (found the following: ndarray: [2047 2047 2047 ..., 540 539 539]) but there is another problem: the file is not long enough; counts only 289872 numbers instead of 640*480...

I am terribly sorry for my exageration by making a class for it...

import numpy as np
import Image

class PGM(object):
    def __init__(self, filepath):

        with open(filepath) as f:

            # suppose all header info in first line:
            info = f.readline().split()
            self.type = info[0]
            self.width, self.height, self.maxval = [int(v) for v in info[1:]]
            size = self.width * self.height

            lines = f.readlines()
            dt = [np.int8, np.int16][self.maxval > 255]
            try:
                # this will work if lines are integers separated by e.g. spaces
                self.data = np.array([l.split() for l in lines], dtype=dt).T
            except ValueError:
                # data is binary
                data = np.fromstring(lines[0], dtype=dt)
                if data.size < size:
                    # this is the case for the 'db.tt/phaR587 (foo.pgm)'
                    #raise ValueError('data binary string probably uncomplete')
                    data = np.hstack((data, np.zeros(size-data.size)))
                self.data = data[:size].reshape((self.width, self.height))

            assert (self.width, self.height) == self.data.shape
            assert self.maxval >= self.data.max()

        self._img = None

    def get_img(self):
        if self._img is None:
            # only executed once
            size = (self.width, self.height)
            mode = 'L'
            data = self.data
            self.img = Image.frombuffer(mode, size, data)

        return self.img

    Image = property(get_img)

mypgm = PGM('foo.pgm')

mypgm.Image

edit: great Idea from Joe Kington to fill image with zeros!

Answer 5

Thanks to the answer by @joe-kington for helping figure this out. The solution follows.

There is a little bit of extra work to not hard-code the known header length (17 bytes in this case), but to determine it from the header. The PGM standard says that the header usually ends with a newline but can end with any whitespace. I think this code will break on a PGM that uses non-newline whitespace for the end-of-header delimeter. Header size in this case would be determined by the size of variables holding width, height, and maxsize, plus two bytes for 'P5', plus 4 bytes of whitespace.

Other cases where this might break are if the width or height are larger than an int (very big image). Or if the PGM is 8-bit rather than 16-bit (which can be determined from maxval, and possible width, height, and the filesize).

#!/usr/bin/python
import numpy as np
import matplotlib.pyplot as plt

file='foo.pgm'
infile = open(file,'r')
header = next(infile)
width, height, maxval = [int(item) for item in header.split()[1:]]
infile.seek(len(header))
image = np.fromfile(infile, dtype=np.uint16).reshape((height, width))
print width, height, maxval
plt.figimage(image)