5

I have a very weird problem.. I really do hope someone has an answer because I wouldn't know where else to ask.

I am writing a cgi application in C++ which is executed by Apache and outputs HTML code. I am compressing the HTML output myself - from within my C++ application - since my web host doesn't support mod_deflate for some reason.

I tested this with Firefox 2, Firefox 3, Opera 9, Opera 10, Google Chrome, Safari, IE6, IE7, IE8, even wget.. It works with ANYTHING except IE8.

IE8 just says "Internet Explorer cannot display the webpage", with no information whatsoever. I know it's because of the compression only because it works if I disable it.

Do you know what I'm doing wrong?

I use zlib to compress it, and the exact code is:

    /* Compress it */
int compressed_output_size = content.length() + (content.length() * 0.2) + 16;
char *compressed_output = (char *)Alloc(compressed_output_size);
int compressed_output_length;
Compress(compressed_output, compressed_output_size, (void *)content.c_str(), content.length(), &compressed_output_length);

/* Send the compressed header */
cout << "Content-Encoding: deflate\r\n";
cout << boost::format("Content-Length: %d\r\n") % compressed_output_length;
cgiHeaderContentType("text/html");
cout.write(compressed_output, compressed_output_length);


static void Compress(void *to, size_t to_size, void *from, size_t from_size, int *final_size)
{
int ret;
z_stream stream;

stream.zalloc = Z_NULL;
stream.zfree = Z_NULL;
stream.opaque = Z_NULL;

if ((ret = deflateInit(&stream, CompressionSpeed)) != Z_OK)
    COMPRESSION_ERROR("deflateInit() failed: %d", ret);

stream.next_out = (Bytef *)to;
stream.avail_out = (uInt)to_size;
stream.next_in = (Bytef *)from;
stream.avail_in = (uInt)from_size;

if ((ret = deflate(&stream, Z_NO_FLUSH)) != Z_OK)
    COMPRESSION_ERROR("deflate() failed: %d", ret);

if (stream.avail_in != 0)
    COMPRESSION_ERROR("stream.avail_in is not 0 (it's %d)", stream.avail_in);

if ((ret = deflate(&stream, Z_FINISH)) != Z_STREAM_END)
    COMPRESSION_ERROR("deflate() failed: %d", ret);

if ((ret = deflateEnd(&stream)) != Z_OK)
    COMPRESSION_ERROR("deflateEnd() failed: %d", ret);

if (final_size)
    *final_size = stream.total_out;
return;
}
Thomas Bonini
  • 40,716
  • 28
  • 117
  • 153
  • ok, this is weird. If I send "gzip" as the content-encoding it works on IE8, but on any other browser..: Content Encoding Error The page you are trying to view cannot be shown because it uses an invalid or unsupported form of compression. – Thomas Bonini Jul 03 '09 at 05:28
  • 1
    After more tries.. If I send the content-encoding as gzip (it's not! it's deflate) it works on Internet explorer only any version, including ie8 while it doesn't work on any other browser. If I send deflate, the correct one, it works on any browser including ie6 and ie7 but not on ie8 <.> – Thomas Bonini Jul 03 '09 at 05:36
  • Looks like it's time to perform user-agent detection. Yes, it's an ugly hack, but in my limited experience web development is full of them. – j_random_hacker Jul 03 '09 at 05:43
  • cout << boost::format("Content-Encoding: %s\r\n") % ((UserAgent.GetBrowser() == INTERNET_EXPLORER && UserAgent.GetVersion() >= 8) ? "gzip" : "deflate"); // Well that fixes it. Still looking for a real answer though if anyone can come up with one. – Thomas Bonini Jul 03 '09 at 06:09
  • 1
    Any clues in IE8's Accept-Encoding header? – John Kugelman Jul 03 '09 at 06:43
  • It sends both deflate and gzip as Accept-Encoding. I check if it contains "deflate" before sending any deflated output. – Thomas Bonini Jul 03 '09 at 09:40
  • Also, make sure you've read this SO question http://stackoverflow.com/questions/388595/why-use-deflate-instead-of-gzip-for-text-files-served-by-apache. Better to stick with gzip. – Eye Nov 12 '12 at 12:40

2 Answers2

6

The gzip and deflate methods aren't the same... they are very close, but there are some subtle differences with the header, so, if you change your content-encoding, you should also change your parameters to the encoding method (specifically, the window size)!

See: http://apcmag.com/improve_your_site_with_http_compression.htm

Probably the other browsers are ignoring your content-encoding specification and doing some automatic recognition, but IE8 is not...

See: http://www.zlib.net/manual.html#deflateInit2

Try to use:

method=Z_DEFLATED
windowBits=-15  (negative so that the header is suppressed)

And use "gzip" as the content-encoding

e.tadeu
  • 4,136
  • 2
  • 16
  • 21
  • 2
    wow, thanks! I love you :D A little correction, though. There are 3 (!) compression formats: zlib - the one I was using, which I thought was "deflate" -, deflate, and gzip. As if it wasn't confusing enough, all 3 of them are apparently created using the library zlib. Not specifying the windowBits (like I was doing) outputs the zlib format, using a negative value (like you suggested) outputs a **DEFLATED** format (not gzip!). I'm not sure yet how to output the gzip format (and I don't care, now it works with deflate on all browsers). THANKS AGAIN!! :) – Thomas Bonini Jul 03 '09 at 21:36
5

I wanted to clarify what I've discovered on this, as I've written my own deflate algorithm, my own HTTP server, and to my dismay IE8 also failed to recognize my deflated content:

HTTP RFC is http://www.faqs.org/ftp/rfc/rfc2616.pdf. Page 17 states both RFC 1950 and RFC 1951 is used when performing a deflate in the HTTP headers. RFC 1950 is simply defining the header and trailer bytes; the deflate algorithm is defined in RFC 1951. When I programmed this to spec, IE8 failed.

When I ignored RFC 1950 and only did RFC 1951, it passed.

I would assume, then, that IE8 isn't following RFC 2616 page 17 correctly, and all of the other browsers are nice enough to accept either format.

Andrew
  • 71
  • 1
  • 1
  • I've just come across a customer's web caching proxy that is forcing deflate, and guess which implementation they've chosen? The wrong one, probably to support IE8. Unfortunately, our Java client, using standard HTTP libraries, is expecting the correct deflate implementation. I'll have to turn off deflate support in our client, and force them to use gzip or nothing. – Marcus Adams Feb 08 '11 at 20:39
  • @MarcusAdams try `inflateInit2(&strm, -MAX_WBITS)` instead of `inflateInit(&strm)` in zlib. – asuka Aug 07 '20 at 16:40