Problems with SSE and chunked encoding + gzip

[joshiste]

It looks to me as the proxy has some issues with chunked encoding.
This is my config:

const proxy = require('http-proxy-middleware');

module.exports = function (app) {
    app.use(['/ui', '/oauth2'], proxy({
        target: 'http://localhost:8080',
        changeOrigin: true,
        xfwd: true
    }));
};

This is the request/response from the backend (captured via wireshark)

GET /ui/events?tenantKey=test HTTP/1.1
x-forwarded-host: localhost:3000
x-forwarded-proto: http
x-forwarded-port: 3000
x-forwarded-for: 127.0.0.1
cookie: SESSION=ZmMyYjg2YzUtMTJhMC00N2ZiLTkyNmYtMDcwMjBhZGM5NGNi; XSRF-TOKEN=42832139-13e2-4774-86c2-708c7fb6af40; JSESSIONID=2724B00DF6ACDB799973E37D4D5B1E0B
accept-language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
accept-encoding: gzip, deflate, br
referer: http://localhost:3000/experiments
sec-fetch-site: same-origin
sec-fetch-mode: cors
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36
x-tenant-key: test
accept: text/event-stream
cache-control: no-cache
pragma: no-cache
connection: close
host: localhost:8080

HTTP/1.1 200 
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Frame-Options: DENY
Content-Type: text/event-stream;charset=UTF-8
Transfer-Encoding: chunked
Date: Sat, 19 Oct 2019 21:04:36 GMT
Connection: close

7
:ping


5
data:
4
test
2



5
data:
4
test
2

and this is the request/response from the proxy:

GET /ui/events?tenantKey=test HTTP/1.1
Host: localhost:3000
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Accept: text/event-stream
X-Tenant-Key: test
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
Referer: http://localhost:3000/experiments
Accept-Encoding: gzip, deflate, br
Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
Cookie: SESSION=ZmMyYjg2YzUtMTJhMC00N2ZiLTkyNmYtMDcwMjBhZGM5NGNi; XSRF-TOKEN=42832139-13e2-4774-86c2-708c7fb6af40; JSESSIONID=2724B00DF6ACDB799973E37D4D5B1E0B

HTTP/1.1 200 OK
X-Powered-By: Express
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
cache-control: no-cache, no-store, max-age=0, must-revalidate
pragma: no-cache
expires: 0
x-frame-options: DENY
content-type: text/event-stream;charset=UTF-8
transfer-encoding: chunked
date: Sat, 19 Oct 2019 21:04:36 GMT
connection: close
Vary: Accept-Encoding
Content-Encoding: gzip

a
..........

The response looks somehow broken to me.

[joshiste]

I just figured out, if I disable the compression for the webpack dev server it works as expected...

[joshiste]

I guess this is the culprit:
https://github.com/expressjs/compression#server-sent-events

[Fractaliste]

I just figured out, if I disable the compression for the webpack dev server it works as expected...

How do you perform it?

[joshiste]

@Fractaliste I hacked this into my scripts:

   "start": "sed -i '' -e 's/compress: true/compress: false/' ./node_modules/react-scripts/config/webpackDevServer.config.js && react-scripts start",

[ShaMan123]

@joshiste Thanks man! You saved me a lot of time and frustration. I wouldn't have managed understanding this.

For those not using linux (sed command unavailable) and using yarn the following works using yarn patch protocol

1. yarn patch react-scripts
2. cd to the folder yarn created (named here as <path/to/folder>):

➤ YN0000: Package react-scripts@npm:4.0.3 got extracted with success!
➤ YN0000: You can now edit the following folder: <path/to/folder>
➤ YN0000: Once you are done run yarn patch-commit "<path/to/folder>" and Yarn will store a patchfile based on your changes.
➤ YN0000: Done in 0s 436ms

3. edit config/webpackDevServer.config.js

- compress: true
+ compress: false

4. run yarn patch-commit "<path/to/folder>" > ./react-scripts.diff from your app's directory. This will generate the diff file react-scripts.diff
5. edit package.json:

-    "react-scripts": "^4.0.3",
+    "react-scripts": "patch:react-scripts@^4.0.3#./react-scripts.diff",

6. run yarn
7. all set, run npm start

[michael-vasyliv]

here might be the fix, at least only this works for me when I do redirect to another gateway

...
onProxyRes(proxyRes, req, res) {
  if (req.headers.accept === 'text/event-stream') {
    res.writeHead(res.statusCode, {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-transform',
      Connection: 'keep-alive',
    });

    proxyRes.pipe(res);
  }
}
...

[mesmith]

@michael-vasyliv Your fix is close, but not quite enough. I found the trick here :

res.writeHead(res.statusCode, {
'Content-Type': 'text/event-stream',
'Connection': 'keep-alive',
'Cache-Control': 'no-cache, no-transform',
'X-Accel-Buffering': 'no',
'Access-Control-Allow-Origin': '*'
});

[tverilytt]

@michael-vasyliv Thanks for the fix! It works nice, but in my case resulted in duplicated messages to clients. I commented out this line:

proxyRes.pipe(res);

And that removed the duplicates 😄