Accessing Compressed Data for Cloudflare Workers Sites

Question

My cloudflare workers site houses binary data that a react app fetches. This binary data is stored as gzip compressed as it compresses extremely well (we're talking about a 20-25x reduction and uncompressed it is too large to fit in the 10MB KV limit). The problem I'm running into is that either the worker returns the data without the appropriate header:

Content-Encoding: gzip

or if I have the worker add the header, cloudflare will doubly compress the response. So how do I store gzip compressed data within cloudflare KV such that I can return it with the proper content encoding and without cloudflare doubly compressing the response?

Reference

For a minimal reproduction: here are the two worker scripts I'm using.

import { getAssetFromKV, mapRequestToAsset } from '@cloudflare/kv-asset-handler'

addEventListener('fetch', event => {
  try {
    event.respondWith(handleEvent(event))
  } catch (e) {
    event.respondWith(new Response('Internal Error', { status: 500 }))
  }
})

async function handleEvent(event) {
  const cacheControl = { browserTTL: 60 * 60 * 6 };
  return await getAssetFromKV(event, { mapRequestToAsset, cacheControl });
}

The above worker script returns the binary data without content encoding header, so the browser does not automatically inflate the response.

So then I tried adding the header manually via

import { getAssetFromKV, mapRequestToAsset } from '@cloudflare/kv-asset-handler'

addEventListener('fetch', event => {
  try {
    event.respondWith(handleEvent(event))
  } catch (e) {
    event.respondWith(new Response('Internal Error', { status: 500 }))
  }
})

async function handleEvent(event) {
  const cacheControl = { browserTTL: 60 * 60 * 6 };
  const resp = await getAssetFromKV(event, { mapRequestToAsset, cacheControl });
  resp.headers.set("Content-Encoding", "gzip");
  return resp;
}

The response has the correct content-encoding but cloudflare compresses the response so now it is serving gzip within gzip.

Is there any way to get a middle ground where I can serve compressed data from within cloudflare KV with the correct header and without doubly compressing the response?

Can you try adding a `Content-Type: application/octet-stream` header (in addition to `Content-Encoding`)? That may prevent CF from gzipping it again. https://support.cloudflare.com/hc/en-us/articles/200168396-What-will-Cloudflare-compress- — Max Ivanov, Nov 14 '20 at 19:01
Good idea, I tried it and unfortunately it is still doubly compressed. It seems like adding the content encoding header is instructing cloudflare to compress the response even if it is unsupported content type. — Nick Babcock, Nov 14 '20 at 19:47

score 4 · Accepted Answer · answered Nov 15 '20 at 20:54

This is an unfortunate wart of the Service Workers spec -- it's designed for use in the browser, where it expects response bodies to be decompressed before the worker runs, and does not expect they will be transmitted over the network again. To make things consistent, Cloudflare Workers has to recompress the data according to the content-encoding header when transmitting. But this in turn means there's no way to serve data that is already compressed.

To solve this, we (Cloudflare) have added a non-standard Response option called encodeBody, which can be "auto" (the default) or "manual" (assumes the body is already compressed).

So you can write code like this:

let resp = await getAssetFromKV(event, { mapRequestToAsset, cacheControl });

// Make a new response with the same body but using manual encoding.
resp = new Response(resp.body, {
  status: resp.status,
  headers: resp.headers,
  encodeBody: "manual"
});

// Modify headers and return.
resp.headers.set("Content-Encoding", "gzip");
return resp;

Accessing Compressed Data for Cloudflare Workers Sites

Reference

1 Answers1