12

I am creating a mobile service based on Pyramid framework. Because it's mobile everything to reduce bandwidth usage is plus. I am considering gzipping all the traffic, even dynamic HTML pages.

What kind of hooks Pyramid framework provides for this? Or is there WSGI middleware for the task? I'd like to do this still on Python level, not Nginx/Apache, so I can better statistics how much gzip brings benefirts.

Mikko Ohtamaa
  • 69,174
  • 40
  • 208
  • 346

3 Answers3

24

First of all I should stress that you should do this on the web server level (nginx or apache). There are several reasons for this:

  1. Performance - If you do this in Python you are using one of your threads that could be handling requests to do cpu-intensive compression. This is way less efficient than allowing your optimized web server to handle it.

  2. Blocking - Most GZip middleware will block your responses, buffering the body so that it can compress the entire response. This is a huge problem if you are attempting to stream any response back to the client because it will get caught in middleware. This is actually a violation of PEP333, the WSGI specification.

With all of this in mind, it might make sense to do it in Python at least for debugging purposes during development.

Since you're already using Pyramid then you have Paste installed. Thus you can simply add the paste.gzipper.GzipMiddleware to your application's pipeline like so:

[filter:gzip]
use = egg:Paste#gzip
compress_level = 6

[pipeline:main]
pipeline =
    gzip
    app

Obviously if you don't want to change the compression level from the default of 6 you can simply add the egg:Paste#gzip to the pipeline in place of configuring the filter and giving it a custom name (gzip).

Michael Merickel
  • 22,500
  • 3
  • 47
  • 66
  • 2
    p.s. use RAW DEFLATE if possible (apache's mod_deflate sends GZIP and GZIP only). GZIP is an unnecessary wrapper around RAW DEFLATE. It is always larger and requires an extra checksum calculation. – David Murdoch Jul 11 '11 at 11:43
  • 2019 update request: nowadays, a Pyramid app will have paste.deploy installed, but not all the rest of Paste. Is there another easy way? or should one use a WSGI app server that has gzip feature (waitress doesn’t)? (This is for contexts where you deploy only app code, so can’t use nginx) – Éric Araujo Dec 31 '19 at 17:24
  • You can still install `pip install paste` fwiw. – Michael Merickel Jan 03 '20 at 05:57
5

You can still get per-request compression statistics with Apache. I created a deflate.log like so:

DeflateFilterNote Input instream
DeflateFilterNote Output outstream
DeflateFilterNote Ratio ratio
LogFormat '%a %v %{outstream}n/%{instream}n (%{ratio}n%%) "%r"' deflate
CustomLog /var/log/httpd/deflate.log deflate

Now I get log entries like:

ip domain -/- (-%) "GET /wp-content/uploads/2010/03/favicon.ico HTTP/1.1"
ip domain 10995/52406 (20%) "GET /2006/07/19/ HTTP/1.0"
ip domain 1873/7891 (23%) "POST /registration/regForm HTTP/1.1"

that I can analyze to my heart's content.

Kirk Strauser
  • 27,753
  • 5
  • 45
  • 62
1

Here are a couple of options at the WSGI level:

ars
  • 106,073
  • 21
  • 135
  • 131