24

Summary

I want to write a .png file as quickly as possible, without a concern for compression. That is, I don't care much about the file size, but I do care that the write happens as quickly as possible.

Motivation

I am making a web-based map application using OpenLayers on the client side and python/C++ on the back end. The application needs to be able to draw dynamic content quickly as the user moves around the map. I have both tile-based (256x256 tile) and single-image-based ("single tile") versions working, but in both cases the slowest part of the back-end render is actually saving the image as a png file (whether on-disk or in-memory). For instance, I may be able to generate a "raw", "tga", or "tiff" version of a certain view in about 200 ms, but it takes more like 1.2 seconds to generate the .png version just because the .png save takes almost a full second, whereas the time to actually save the other formats is 100 ms or less (and even though the "raw" file is five times the size of the .png file). And this file save time is also significantly more than the time to transfer the resulting image from the server to the client. (One important property of my app is that in general the "back end" will be running on the same machine as the browser, so transfer times are negligible, even for largeish files.)

I thought I could make .png writes fast (when using libpng from C++) by calling

    png_set_compression_level( png_ptr, 0 );

before calling any png_write_... functions. However, while that call does indeed seem to stop libpng from compressing the file (the resulting file is about the same size as the .raw file), it doesn't make saving the .png noticeably faster.

Please help

I need to use .png for these image, because I need them to be transparent overlays on top of the base map, and I need more than the 256 colors offered by GIF. OpenLayers is simply using html img tags, so my understanding is that I can only use valid img formats.

I would think there would be a way of writing a .png file quickly by not doing any real compression (I understand that .png is "always compressed" but I imagined this could include "null compression"). It seems like you should be able to write like a simple fixed header, followed by the uncompressed data, followed by some fixed footer. Or maybe that same idea but in a line-by-line way. The point being that I can do all sorts of looping through this 2.5 MB of raw data in memory in C++ very quickly, and can dump it to various file formats very quickly, so it seems like I should be able to dump it in a fixed, uncompressed .png format quickly also.

Does that sound right? Do you know where I can find examples of code that does that?

M Katz
  • 4,604
  • 3
  • 36
  • 59
  • 2
    Your disk has a fixed / max write speed, and it's generally going to be slower than your processor's ability to do math. – Brian Roach Oct 30 '11 at 00:54
  • 2
    +1 mostly because this may be the first "do this FAST" question I've seen in a while that (1) has a good reason for caring about speed and (2) includes hard numbers and (3) actually has seen a honest attempt before being posted here. –  Oct 30 '11 at 00:56
  • 1
    @ Brian Roach: Nope. Please read the post again. Disk speed is *not* the limiting factor here at all. I can write a 2.5 MB .raw file in 100 ms. When I write a .png file that is five times smaller, it takes a full second. Plus it doesn't matter if I actually write it to disk or write it to an in-memory file. Same slowness, due to the speed of actually building/compressing the data. – M Katz Oct 30 '11 at 01:08
  • If you have already disabled compression and the file is then larger and it's still as slow as before, have you considered the possibility that compression is not the culprit here? – Pascal Cuoq Oct 30 '11 at 01:59
  • @ Pascal Cuoq: Well, it may not be compression per se. But I don't believe it's sheer number of bytes either. I need to learn more about .png, but I know there is "filtering" and some other stuff going on. I believe it's some process that's happening with libpng spending time arranging the bytes, whether it's "compression" or something else. – M Katz Oct 30 '11 at 02:06
  • Did you manage to do this? I have the same problem, I want to write PNG very fast to RAM. – Phil Jan 15 '14 at 08:33

2 Answers2

31

what you want is an implementation that is specialized for your purpose; you are going to have to write your own encoder. it's actually not too hard and the specs are free.

the format isn't too complex and should be easy to implement an encoder

note: all values are unsigned. Multiple-byte integers are in "network byte order" (most significant byte first).

the format is composed of chunks. chunk structure:

  • length of chunk contents, 4 bytes
  • chunk identifier (ASCII), 4 bytes
  • chunk contents, "length of chunk contents" bytes
  • CRC of the identifier and the contents (i.e. excluding the length), 4 bytes

your implementation should only need the magic number and three chunks:

detailed layout:

  • { 137, 80, 78, 71, 13, 10, 26, 10 } (magic number), 8 bytes
  • byte length of IHDR chunk, 4 bytes (value: 13)
  • { 73, 72, 68, 82 } ("IHDR"), 4 bytes
  • width, 4 bytes
  • height, 4 bytes
  • bit depth (per color), 1 byte (8 = 24/32 bit color)
  • color type, 1 byte (2 = RGB)
  • compression method, 1 byte (0 = DEFLATE algo which allows no compression)
  • filter method, 1 byte (0 = no filter)
  • interlace method, 1 byte (0 = no interlace)
  • CRC of IHDR chunk, 4 bytes
  • byte length of IDAT chunk contents, 4 bytes
  • { 73, 68, 65, 84 } ("IDAT"), 4 bytes
  • raw image data with DEFLATE algo wrapping, "byte length of IDAT chunk contents" bytes
  • CRC of IDAT chunk, 4 bytes
  • byte length of IEND chunk, 4 bytes (value: 0)
  • { 73, 69, 78, 68 } ("IEND"), 4 bytes
  • CRC of IEND chunk, 4 bytes (can be precomputed)

DEFLATE data encoding with no compression

your data will be split into chunks of 65535 bytes the format is simple:

  • first X chunks
    • header, 1 byte (value: 0)
    • data, 65535 bytes
  • final chunk
    • header, 1 byte (value: 1)
    • data, 65535 bytes or less

that's it

so that is how you make a fast png file

Glenn Randers-Pehrson
  • 10,432
  • 2
  • 31
  • 57
Gravis
  • 941
  • 11
  • 16
  • Thanks for the detailed info. That's just what I needed. – M Katz Nov 14 '11 at 02:17
  • 1
    Some details from when I needed the same thing: – Řrřola Jan 27 '12 at 18:44
  • 1
    The IDAT contents look like this: byte 0x78, byte 0x01, chunks, Adler32 checksum of the image data. The chunks have a 5-byte header in little-endian: byte (1=final chunk, otherwise 0), 2 bytes chunk_size, 2 bytes complement of chunk_size. – Řrřola Jan 27 '12 at 18:53
  • 1
    Also, every image row first needs to be prefixed with byte 0 (=no filter). – Řrřola Jan 27 '12 at 18:55
  • 1
    The crc32("IEND") == 0xAE426082 – Michaelangel007 Feb 11 '14 at 23:56
  • The PNG file specification is really well written and an enjoyable read. The file format is very flexible and is generally well designed. – Drew Noakes Feb 24 '14 at 23:45
  • Thanks for your outline. I've put some crappy code up at https://github.com/orthopteroid/img-tools/blob/master/png256x256x1.h –  Jun 05 '15 at 23:16
  • This is a great answer, although I feel it kind of understates the complexity of creating deflated IDAT data. I ended up rolling my own javascript PNG encoder after reading this answer, but using a deflate library. – Aerik Jun 07 '17 at 18:56
  • @Gravis please don't be insulted or anything, it's a GREAT answer and it was very helpful. But I struggled with the deflate bit. Even for uncompressed blocks, there seems to be more than just the first byte being a 1 or zero - I struggled through part of the spec (https://tools.ietf.org/html/rfc1951) and found, at a minimum, there's also something about encoding block lengths. – Aerik Jun 08 '17 at 19:16
  • @Gravis - from the DEFLATE spec: "3.2.4. Non-compressed blocks (BTYPE=00) Any bits of input up to the next byte boundary are ignored. The rest of the block consists of the following information: | LEN, NLEN, LEN bytes of literal data... | LEN is the number of data bytes in the block. NLEN is the one's complement of LEN." – Aerik Jun 12 '17 at 18:18
  • @Gravis Okay, here is more info, from no less than Mark Adler, PNG doesn't just deflate, but uses zlib format: https://stackoverflow.com/questions/21816646/png-deflate-and-zlib#21820311 and then I also found it here: https://www.w3.org/TR/PNG-Compression.html – Aerik Jun 14 '17 at 01:50
15

Compression speeed for PNG is influenced mainly by two parameters:

  1. Compression level of ZLIB compression. Setting it to 0, with png_set_compression_level esentially amounts to disabling this compression.

  2. Pixel filtering. This can vary for each line, and the choosing is often done by some heuristic, which can be efficient for size, but can be time-consuming. See png_set_filter and png_set_filter_heuristics If speed is the main concern (and, more to the point, if ZLIB compression is disabled) you should select a single filter: PNG_FILTER_NONE

There is little more to do to optimize speed, I think. Compared to a "raw" format like BMP or TGA, a non-compressed PNG will still have the (small) burden of computing the CRC32 for each chunk, plus the internal Adler CRC for ZLIB. And that's pretty much all.

(For the record, I've coded a full pure Java coder/encoder which strives for -memory and CPU- efficiency: PNGJ)

leonbloy
  • 65,169
  • 19
  • 130
  • 176