On a machine running Sophos, why do all my browsers fail to receive server sent events (sse) from my Python apps in realtime?

Question

My ASGI app sends events fine to curl, and to my phone. However, even though the server is sending the events, and the headers look right, neither Firefox nor Chrome on my Windows machine receives the events until the connection is closed.

This happens whether I host the server in WSL, in Powershell terminal, or on a separate Linux box.

However, those same browsers work fine if I host the server on repl.it (please fork it and try it out).

I have tried fiddling with Windows firewall settings, to no avail.

Here is the application code:

import asyncio
import datetime


async def app(scope, receive, send):
    headers = [(b"content-type", b"text/html")]
    if scope["path"] == "/":
        body = (
            "<html>"
            "<body>"
            "</body>"
            "<script>"
            "  let eventSource = new EventSource('/sse');"
            "  eventSource.addEventListener('message', (e) => {"
            "    document.body.innerHTML += e.data + '<br>';"
            "  });"
            "</script>"
            "</html>"
        ).encode()

        await send({"type": "http.response.start", "status": 200, "headers": headers})
        await send({"type": "http.response.body", "body": body})
    elif scope["path"] == "/sse":
        headers = [
            (b"content-type", b"text/event-stream"),
            (b"cache-control", b"no-cache"),
            (b"connection", b"keep-alive"),
        ]

        async def body():
            ongoing = True
            while ongoing:
                try:
                    payload = datetime.datetime.now()
                    yield f"data: {payload}\n\n".encode()
                    await asyncio.sleep(10)
                except asyncio.CancelledError:
                    ongoing = False

        await send({"type": "http.response.start", "status": 200, "headers": headers})
        async for chunk in body():
            await send({"type": "http.response.body", "body": chunk, "more_body": True})
        await send({"type": "http.response.body", "body": b""})
    else:
        await send({"type": "http.response.start", "status": 404, "headers": headers})
        await send({"type": "http.response.body", "body": b""})

This can be run by naming the file above to asgi_sse.py, then pip install uvicorn, then using something like

uvicorn asgi_sse:app

(substitute daphne or hypercorn instead of uvicorn above to see how those servers handle the app.)

The headers:

$ curl -I http://localhost:8000/sse
HTTP/1.1 200 OK
date: Mon, 01 Jun 2020 09:51:41 GMT
server: uvicorn
content-type: text/event-stream
cache-control: no-cache
connection: keep-alive

And the response:

$ curl http://localhost:8000/sse
data: 2020-06-01 05:52:40.735403

data: 2020-06-01 05:52:50.736378

data: 2020-06-01 05:53:00.736812

Any insights are quite welcome!

score 3 · Answer 1 · answered Jul 15 '20 at 09:42

I ran into the the same issue recently but with a NodeJS + Express app. I ended up sending a 2 megabyte chunk if the connection is not secure.

if (!req.secure) {
  res.write(new Array(1024 * 1024).fill(0).toString());
}

This is to get server sent events working on a development environment without a secure connection.

The complete implementation:

app.use('/stream', (req, res) => {
  res.set({
    'Content-Type': 'text/event-stream;charset=utf-8',
    'Cache-Control': 'no-cache, no-transform',
    'Content-Encoding': 'none',
    'Connection': 'keep-alive'
  });
  res.flushHeaders();

  if (!req.secure) {
    res.write(new Array(1024 * 1024).fill(0).toString());
  }

  const sendEvent = (event, data) => {
    res.write(`event: ${String(event)}\n`);
    res.write(`data: ${data}`);
    res.write('\n\n');
    res.flushHeaders();
  };

  const intervalId = setInterval(() => {
    sendEvent('ping', new Date().toLocaleTimeString());
  }, 5000);

  req.on('close', () => {
    clearInterval(intervalId);
  });
});

Exactly! Same as what I did in Python code. Test if not HTTPS, then send chunk if cleartext. Thanks for this code snippet. Not exactly sure, but maybe it should be a comment, not an answer? I am sure it will help someone working with Javascript, though. — jdbow75, Jul 16 '20 at 12:09

jdbow75 · Accepted Answer · 2020-06-11T00:27:41.353

The Explanation

My company has Sophos Endpoint Security's web protection turned on. According to this entry in Sophos's community, the web protection buffers and scans text/event-stream content for malware. Hence the unexpected buffering.

The Solution

There are two workarounds I have found:

Send a 2 megabyte (more is fine; less is not) chunk of data before the first event. You do not need to send this with every event; just the first one.
Or use https (SSL/TLS). For local development, consider using mkcert for a convenient way to set this up.

The Python Mystery

The above issue was not simply an issue with my code, nor with Uvicorn, Hypercorn, or ASGI. In fact, I even tried an aiohttp implementation, with the same sad results. However, when I tried a Go example implementation of SSE, and another in Node.js it worked fine, no workarounds needed. The only difference I can see was that the Go implementation used a flush method after each event. I am uncertain why ASGI and aiohttp do not expose some sort of flush method, or, if they do, why I cannot find it. If they did, would it make these workarounds unnecessary? I am unsure.

Here is the updated code that makes it work with Sophos, and checks if serving over https or not:

async def app(scope, receive, send):
    headers = [(b"content-type", b"text/html")]
    if scope["path"] == "/":
        body = (
            "<!DOCTYPE html>"
            "<html>"
            "<body>"
            "</body>"
            "<script>"
            "  let eventSource = new EventSource('/sse');"
            "  eventSource.addEventListener('message', (e) => {"
            "    document.body.innerHTML += e.data + '<br>';"
            "  });"
            "</script>"
            "</html>"
        ).encode()

        await send({"type": "http.response.start", "status": 200, "headers": headers})
        await send({"type": "http.response.body", "body": body})
    elif scope["path"] == "/sse":
        headers = [
            (b"Content-Type", b"text/event-stream"),
            (b"Cache-Control", b"no-cache"),
            (b"Connection", b"keep-alive"),
        ]

        async def body():
            ongoing = True
            while ongoing:
                try:
                    payload = datetime.datetime.now()
                    yield f"data: {payload}\n\n".encode()
                    await asyncio.sleep(10)
                except asyncio.CancelledError:
                    ongoing = False

        await send({"type": "http.response.start", "status": 200, "headers": headers})
        if scope["scheme"] != "https": # Sophos will buffer, so send 2MB first
            two_meg_chunk = "." * 2048 ** 2
            await send(
                {
                    "type": "http.response.body",
                    "body": f": {two_meg_chunk}\n\n".encode(),
                    "more_body": True,
                }
            )
        async for chunk in body():
            await send({"type": "http.response.body", "body": chunk, "more_body": True})
        await send({"type": "http.response.body", "body": b""})
    else:
        await send({"type": "http.response.start", "status": 404, "headers": headers})
        await send({"type": "http.response.body", "body": b""})

On a machine running Sophos, why do all my browsers fail to receive server sent events (sse) from my Python apps in realtime?

2 Answers2

The Explanation

The Solution

The Python Mystery