Upgrade: websocket vs Upgrade: WebSocket

Today I’ve been trying to improve the security of our Comrex bricklinks by hiding their webpages behind a secure SSL proxy.

Until recently, comrexes were controlled via a flash client using a proprietary communication protocol which ran on the http port – they couldn’t be proxied. Recently however the demise of flash means they’ve been upgraded to a html/javascript/websocket system, which works fine. However serving this over http is pretty bad. There’s no proper authentication (comrexes just have a password, not individual users), and managing vpn connections for everyone that may need access isn’t great either.

Instead, hiding behind our SSL based proxy which authenticates based on a client certificate (other authentication modules are available), solves many problems, including AAA, and upgrades the communication to secure so people can’t sniff the traffic (at least aside from the proxy-comrex channel)

Being an old-school chap I still use apache2 for proxying. To hide a web-socket xen-orchestra server for example, I use the rewrite and proxy_wstunnel modules, and a standard bit of templating that’s included to do the certificate authentication (client.require sets SSLVerifyClient require)

<virtualhost *:443>
        Header always set Strict-Transport-Security "max-age=604800; includeSubDomains"
        Header always set X-Frame-Options SAMEORIGIN
        Header always set X-XSS-Protection "1; mode=block"
        Header always set X-Content-Type-Options nosniff
	Header always set Content-Security-Policy "upgrade-insecure-requests;"

	servername ${fqdn}

	errordocument 404 https://${fqdn}/

	documentroot /var/www/html/
	<location />
		require all granted

	define client.require

	define ssl.cert clean.cert

	# configure ssl, standard logging etc
	include /etc/ssl/fqdn.conf

	RewriteEngine On
	RewriteCond %{HTTP:Upgrade} =websocket [NC]
	RewriteRule /(.*)           ws://$1 [P,L]
	RewriteCond %{HTTP:Upgrade} !=websocket [NC]
	RewriteRule /(.*) $1 [P,L]

	RequestHeader set REMOTE_USER %{SSL_CLIENT_S_DN_Email}s
	RequestHeader set AUTHEMAIL %{SSL_CLIENT_S_DN_Email}s

This works fine, with the “upgrade-insecure-requests” header ensuring the browser (Firefox, other browsers are available) upgrades any ws requests to wss.

However it isn’t working on comrex.

tcpdump and wireshark come to the rescue as they so often do. Dumping the traffic between the proxy and the comrex (which is in the clear on port 80) shows various requests, and running on my desktop between the browser and the comrex shows a different request.

First on the desktop – which works.

GET / HTTP/1.1
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0
Accept: */*
Accept-Language: en-GB,en;q=0.5
Accept-Encoding: gzip, deflate
Sec-WebSocket-Version: 13
Sec-WebSocket-Protocol: ComrexIPC
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Key: xxxxxxxxxxxxxxxxxxx==
Connection: keep-alive, Upgrade
Sec-GPC: 1
Pragma: no-cache
Cache-Control: no-cache
Upgrade: websocket

And sure enough telnetting to the comrex on port 80 and pasting that in works, with the comrex responding

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: xxxxxxxxxxxxxxxxxxxxx=
Sec-WebSocket-Protocol: ComrexIPC

Looking at the request from the proxy is far more involved, with extra cookie and authentication headers

GET / HTTP/1.1
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0
Accept: */*
Accept-Language: en-GB,en;q=0.5
Accept-Encoding: gzip, deflate, br
Sec-WebSocket-Version: 13
Sec-WebSocket-Protocol: ComrexIPC
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Key: xxxxxxxxxxx==
Cookie: (all sorts of random stuff from
Sec-GPC: 1
Pragma: no-cache
Cache-Control: no-cache
Upgrade: WebSocket
Connection: Upgrade

Sure enough pasting that in to the telnet session just hangs. Surely it can’t be hanging on a few extraneous headers?

vimdiff to the rescue, adding in each header into the working request until it breaks, and sure enough the extra headers were fine. The problem is the proxy issues this.

Upgrade: WebSocket

Apache sends this in CamelCase, firefox sends in lower case. Websockets are defined in rfc6455,

An |Upgrade| header field containing the value "websocket", treated as an ASCII case-insensitive value.

So it looks like the comrex should accept “WebSocket” as well as websocket, but that doesn’t really help me. Time and time again, “should just” is the leading symptom of computer problems.

Apache2 on my ubuntu proxy is delivered via apt, and the proxy_wstunnel module is in the base apache2-bin package.

$ grep proxy_wstunnel /var/lib/dpkg/info/*list

apt source apache2-bin gets the right version of mod_proxy_wstunnel.c, and here’s the line.

const char *upgrade_method = *worker->s->upgrade ? worker->s->upgrade : "WebSocket";
if (ap_cstr_casecmp(upgrade_method, "NONE") == 0) {
    buf = apr_pstrdup(p, "Upgrade: WebSocket" CRLF "Connection: Upgrade" CRLF CRLF);
} else if (ap_cstr_casecmp(upgrade_method, "ANY") == 0) {
    const char *upgrade;
    upgrade = apr_table_get(r->headers_in, "Upgrade");
    buf = apr_pstrcat(p, "Upgrade: ", upgrade, CRLF "Connection: Upgrade" CRLF CRLF, NULL);
} else {
    buf = apr_pstrcat(p, "Upgrade: ", upgrade_method, CRLF "Connection: Upgrade" CRLF CRLF, NULL);

This function is in static int proxy_wstunnel_request

So there’s hope this can be overwritten without recompiling the module. Interestingly enough on older apache 2.4.18 (used on the now end-of-life Ubuntu 16.04), only the first line was there, but it seems there’s hope on more recent versions.

upgrade_method can be set to NONE, ANY, or any string. ANY should pass firefox’s string through direct and thus work, as should setting it to “websocket”.

const char *upgrade_method = *worker->s->upgrade ? worker->s->upgrade : "WebSocket";

The manual says

In fact the module can be used to upgrade to other protocols, you can set the upgrade parameter in the ProxyPass directive to allow the module to accept other protocol. NONE means you bypass the check for the header but still upgrade to WebSocket. ANY means that Upgrade will read in the request headers and use in the response Upgrade

However that doesn’t seem to work in my setup.

To check I’m on the right path though I compile and build apache, and change the upgrade_method line to be lowercase

const char *upgrade_method = *worker->s->upgrade ? worker->s->upgrade : "websocket";

Building apache can be done with a quick

sudo apt build-dep apache2-bin
sudo apt source apache2-bin
cd apa*
./configure --enable-layout=Debian
sed -i "s/BUILD_DATETIME/\"`date +%FT%T`\"/" ./server/buildmark.c 

I’m no C expert, and my debian packages are embarassingly non-standard. I ran into a tiresome /home/me/ap/apache2-2.4.18/server/buildmark.c:20:36: static const char server_built[] = BUILD_DATETIME; error, so replacing that with the current date (the inline sed) made the problem go away.

Once built, copy the .so into /usr/lib/apache2/modules/ and restart apache, and lo and behold it works.

Of course this isn’t a nice solution at all. From the looks of it there should be a way in apache to override the “WebSocket” to “websocket” using the “upgrade=ANY” or “upgrade=websocket” lines, but for whatever reason that’s not working. Maybe nginx would work out of the box too.

However the amount of work to be done is infinite, and the amount of time left is finite, so I’ll leave it there and working and ask comrex about changing their code to accept case-insensitive websocket commands as in the RFC.

Leave a Reply

Your email address will not be published. Required fields are marked *