Today I’ve been trying to improve the security of our Comrex bricklinks by hiding their webpages behind a secure SSL proxy.
Until recently, comrexes were controlled via a flash client using a proprietary communication protocol which ran on the http port – they couldn’t be proxied. Recently however the demise of flash means they’ve been upgraded to a html/javascript/websocket system, which works fine. However serving this over http is pretty bad. There’s no proper authentication (comrexes just have a password, not individual users), and managing vpn connections for everyone that may need access isn’t great either.
Instead, hiding behind our SSL based proxy which authenticates based on a client certificate (other authentication modules are available), solves many problems, including AAA, and upgrades the communication to secure so people can’t sniff the traffic (at least aside from the proxy-comrex channel)
Being an old-school chap I still use apache2 for proxying. To hide a web-socket xen-orchestra server for example, I use the rewrite and proxy_wstunnel modules, and a standard bit of templating that’s included to do the certificate authentication (client.require sets SSLVerifyClient require)
<virtualhost *:443>
Header always set Strict-Transport-Security "max-age=604800; includeSubDomains"
Header always set X-Frame-Options SAMEORIGIN
Header always set X-XSS-Protection "1; mode=block"
Header always set X-Content-Type-Options nosniff
Header always set Content-Security-Policy "upgrade-insecure-requests;"
servername ${fqdn}
errordocument 404 https://${fqdn}/
documentroot /var/www/html/
<location />
require all granted
</location>
define client.require
define ssl.cert clean.cert
# configure ssl, standard logging etc
include /etc/ssl/fqdn.conf
RewriteEngine On
RewriteCond %{HTTP:Upgrade} =websocket [NC]
RewriteRule /(.*) ws://172.16.1.2:80/$1 [P,L]
RewriteCond %{HTTP:Upgrade} !=websocket [NC]
RewriteRule /(.*) http://172.16.1.2:80/$1 [P,L]
RequestHeader set REMOTE_USER %{SSL_CLIENT_S_DN_Email}s
RequestHeader set AUTHEMAIL %{SSL_CLIENT_S_DN_Email}s
</virtualhost>
This works fine, with the “upgrade-insecure-requests” header ensuring the browser (Firefox, other browsers are available) upgrades any ws requests to wss.
However it isn’t working on comrex.
tcpdump and wireshark come to the rescue as they so often do. Dumping the traffic between the proxy and the comrex (which is in the clear on port 80) shows various requests, and running on my desktop between the browser and the comrex shows a different request.
First on the desktop – which works.
GET / HTTP/1.1
Host: 192.168.0.114
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0
Accept: */*
Accept-Language: en-GB,en;q=0.5
Accept-Encoding: gzip, deflate
Sec-WebSocket-Version: 13
Origin: http://192.168.0.114
Sec-WebSocket-Protocol: ComrexIPC
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Key: xxxxxxxxxxxxxxxxxxx==
Connection: keep-alive, Upgrade
Sec-GPC: 1
Pragma: no-cache
Cache-Control: no-cache
Upgrade: websocketAnd sure enough telnetting to the comrex on port 80 and pasting that in works, with the comrex responding
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: xxxxxxxxxxxxxxxxxxxxx=
Sec-WebSocket-Protocol: ComrexIPCLooking at the request from the proxy is far more involved, with extra cookie and authentication headers
GET / HTTP/1.1
Host: mycomrex.mydomain.co.uk
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0
Accept: */*
Accept-Language: en-GB,en;q=0.5
Accept-Encoding: gzip, deflate, br
Sec-WebSocket-Version: 13
Origin: https://mycomrex.mydomain.co.uk
Sec-WebSocket-Protocol: ComrexIPC
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Key: xxxxxxxxxxx==
Cookie: (all sorts of random stuff from mydomain.co.uk)
Sec-GPC: 1
Pragma: no-cache
Cache-Control: no-cache
SSL_CLIENT_VERIFY: SUCCESS
REMOTE_USER: me@mydomain.co.uk
AUTHEMAIL: me@mydomain.co.uk
AUTH_FULL_NAME: Paul Weaver
AUTH_FULLNAME: Paul Weaver
AUTH_EMAIL: me@mydomain.co.uk
X-Forwarded-For: 172.16.12.9
X-Forwarded-Host: mycomrex.mydomain.co.uk
X-Forwarded-Server: mycomrex.mydomain.co.uk
Upgrade: WebSocket
Connection: UpgradeSure enough pasting that in to the telnet session just hangs. Surely it can’t be hanging on a few extraneous headers?
vimdiff to the rescue, adding in each header into the working request until it breaks, and sure enough the extra headers were fine. The problem is the proxy issues this.
Upgrade: WebSocketApache sends this in CamelCase, firefox sends in lower case. Websockets are defined in rfc6455,
An |Upgrade| header field containing the value "websocket", treated as an ASCII case-insensitive value.So it looks like the comrex should accept “WebSocket” as well as websocket, but that doesn’t really help me. Time and time again, “should just” is the leading symptom of computer problems.
Apache2 on my ubuntu proxy is delivered via apt, and the proxy_wstunnel module is in the base apache2-bin package.
$ grep proxy_wstunnel /var/lib/dpkg/info/*list
/var/lib/dpkg/info/apache2-bin.list:/usr/lib/apache2/modules/mod_proxy_wstunnel.so
/var/lib/dpkg/info/apache2.list:/etc/apache2/mods-available/proxy_wstunnel.loadapt source apache2-bin gets the right version of mod_proxy_wstunnel.c, and here’s the line.
const char *upgrade_method = *worker->s->upgrade ? worker->s->upgrade : "WebSocket";
....
if (ap_cstr_casecmp(upgrade_method, "NONE") == 0) {
buf = apr_pstrdup(p, "Upgrade: WebSocket" CRLF "Connection: Upgrade" CRLF CRLF);
} else if (ap_cstr_casecmp(upgrade_method, "ANY") == 0) {
const char *upgrade;
upgrade = apr_table_get(r->headers_in, "Upgrade");
buf = apr_pstrcat(p, "Upgrade: ", upgrade, CRLF "Connection: Upgrade" CRLF CRLF, NULL);
} else {
buf = apr_pstrcat(p, "Upgrade: ", upgrade_method, CRLF "Connection: Upgrade" CRLF CRLF, NULL);
}This function is in static int proxy_wstunnel_request
So there’s hope this can be overwritten without recompiling the module. Interestingly enough on older apache 2.4.18 (used on the now end-of-life Ubuntu 16.04), only the first line was there, but it seems there’s hope on more recent versions.
upgrade_method can be set to NONE, ANY, or any string. ANY should pass firefox’s string through direct and thus work, as should setting it to “websocket”.
const char *upgrade_method = *worker->s->upgrade ? worker->s->upgrade : "WebSocket";The manual says
In fact the module can be used to upgrade to other protocols, you can set the upgrade parameter in the ProxyPass directive to allow the module to accept other protocol. NONE means you bypass the check for the header but still upgrade to WebSocket. ANY means that Upgrade will read in the request headers and use in the response Upgrade
https://httpd.apache.org/docs/2.4/mod/mod_proxy_wstunnel.html
However that doesn’t seem to work in my setup.
To check I’m on the right path though I compile and build apache, and change the upgrade_method line to be lowercase
const char *upgrade_method = *worker->s->upgrade ? worker->s->upgrade : "websocket";Building apache can be done with a quick
sudo apt build-dep apache2-bin
sudo apt source apache2-bin
cd apa*
./configure --enable-layout=Debian
sed -i "s/BUILD_DATETIME/\"`date +%FT%T`\"/" ./server/buildmark.c
makeI’m no C expert, and my debian packages are embarassingly non-standard. I ran into a tiresome /home/me/ap/apache2-2.4.18/server/buildmark.c:20:36: static const char server_built[] = BUILD_DATETIME; error, so replacing that with the current date (the inline sed) made the problem go away.
Once built, copy the .so into /usr/lib/apache2/modules/mod_proxy_wstunnel.so and restart apache, and lo and behold it works.
Of course this isn’t a nice solution at all. From the looks of it there should be a way in apache to override the “WebSocket” to “websocket” using the “upgrade=ANY” or “upgrade=websocket” lines, but for whatever reason that’s not working. Maybe nginx would work out of the box too.
However the amount of work to be done is infinite, and the amount of time left is finite, so I’ll leave it there and working and ask comrex about changing their code to accept case-insensitive websocket commands as in the RFC.