This is an idea that SSH can be tunneled with nginx on port 443, without disturbing normal HTTPS traffic. I’ve used it for over a year, and this is useful to bypass firewalls on hotel/airport wifi, and other places which may lock down traffic to only HTTPS.
This is possible because the nginx stream ssl module can tunnel anything over SSL/TLS, and the nginx ssl_preread module, can filter based on early TLS handshake variables such as SNI (server name) and ALPN (protocol name).
Then, in the SSH client config, we can use the ProxyCommand parameter with openssl s_client to “untunnel” the SSH connection.
SNI filtering
This was the basic nginx config I started with:
stream {
map $ssl_preread_server_name $proxy {
ssh.example.com unix:/sock/nginx-ssh.sock;
default unix:/sock/nginx-https.sock;
}
server { # Server 1 - the filter
listen 443;
proxy_timeout 60m;
proxy_pass $proxy;
ssl_preread on;
proxy_protocol on;
}
server { # Server 2 - the SSH forwarder
listen unix:/sock/nginx-ssh.sock ssl proxy_protocol;
ssl_certificate "/etc/letsencrypt/live/ssh.example.com/fullchain.pem";
ssl_certificate_key "/etc/letsencrypt/live/ssh.example.com/privkey.pem";
proxy_ssl off;
proxy_pass localhost:22;
proxy_timeout 60m;
}
}
http { # Server 3 - the HTTPS server
server {
listen unix:/sock/nginx-https.sock ssl proxy_protocol;
ssl_certificate "/etc/letsencrypt/live/www.example.com/fullchain.pem";
ssl_certificate_key "/etc/letsencrypt/live/www.example.com/privkey.pem";
# More normal http server stuff ..
}
}
This is pretty much the example in the documentation, but using unix sockets instead of regular TCP, and with this SSH-config it works well:
Host ssh.example.com
ProxyCommand openssl s_client -connect ssh.example.com:443 -verify_return_error -quiet -verify_quiet
The way it works is by starting several “servers”, listening on several ports, and since everything here is on the same machine, I use unix sockets, but it works just as well with IP addresses and ports.
In reverse order:
- Server 3: the HTTPS server
- A regular HTTPS Server, listening on unix socket unix:/sock/nginx-https.sock, with SSL and proxy protocol enabled.
- Since the proxy protocol is enabled, the client IP will be taken from the proxy protocol instead of the client connecting the unix socket.
- This is also where the certificate is configured
- Server 2: the SSH Forwarder server
- A streaming server, this listens on unix socket unix:/sock/nginx-ssh.sock, with SSL and proxy protocol enabled
- This server has a different certificate
- The server will take SSL an encrypted stream, decrypt it and forward it to localhost:22
- Server 1: the filter server
- A streaming server, listening on port 443.
- Does not have any certficates, and instead forwards the connections to other servers without decrypting
- Adds proxy protocol so the https server can get the correct client IP addresses
Notes:
To get s_client working properly with SSH, I had to add -verify_return_error to actually verify the certificate, and -quiet -verify_quiet so it doesn’t output unnecessary information.
I also had to extend the proxy_timeout in the nginx config. This is 10m by default, and it will close idle SSH sessions after only 10 minutes, so I extended it to an hour.
ALPN filtering
The main problem with this is that it filters on domain names. So if i want to have the same name for SSH and HTTPS (for example git.example.com), that will collide and not work. If i route the git subdomain to SSH, then the browser will get SSH as well, and that’s not good.
So I decided to filter on ALPN instead, and it can be done just by changing the map:
map $ssl_preread_alpn_protocols $proxy {
~\bssh\b unix:/sock/nginx-ssh.sock;
default unix:/sock/nginx-https.sock;
}
and adding -alpn ssh to the openssl s_client in the ssh_config:
Host ssh.example.com
ProxyCommand openssl s_client -connect %h:443 -alpn ssh -verify_return_error -quiet -verify_quiet
Note the %h in the proxycommand. It will be replaced by the hostname in ssh, so it’s possible to use this line for any host.
Turns out, I don’t have to give up on SNI filtering to get this working.
I can use two maps, one for ALPN, and one for server_name, and have a long list of unix sockets:
stream {
map $ssl_preread_alpn_protocols $alpn {
~\bssh\b ssh.sock;
default https.sock;
}
map $ssl_preread_server_name $servername {
git.example.com unix:/sock/git;
ssh.example.com unix:/sock/nginx;
default unix:/sock/nginx;
}
...
server {
listen 443;
...
proxy_pass $servername-$alpn;
}
server { # ssh forwarder
listen unix:/sock/nginx-ssh.sock ssl proxy_protocol;
# cert and route to localhost:22
}
server { # ssh forwarder
listen unix:/sock/git-ssh.sock ssl proxy_protocol;
# cert and route to git-server:22
}
}
...
This quickly gets out of hand with more domain names.
A better alternative is to use server_name in the stream server.
To get that working through the proxy, I turn on proxy_ssl_server_name for the server listening on 443, then I can use server_name in the SSH forwarder blocks:
stream {
map $ssl_preread_alpn_protocols $proxy {
~\bssh\b unix:/sock/nginx-ssh.sock;
default unix:/sock/nginx-https.sock;
}
server {
listen 443;
proxy_timeout 60m;
proxy_pass $proxy;
proxy_ssl_server_name on;
ssl_preread on;
proxy_protocol on;
}
server {
listen unix:/sock/nginx-ssh.sock ssl proxy_protocol default_server;
server_name example.com;
ssl_certificate "/etc/letsencrypt/live/www.example.com/fullchain.pem";
ssl_certificate_key "/etc/letsencrypt/live/www.example.com/privkey.pem";
proxy_ssl off;
proxy_pass localhost:22;
proxy_timeout 60m;
}
server {
listen unix:/sock/nginx-ssh.sock ssl proxy_protocol;
server_name git.example.com;
ssl_certificate "/etc/letsencrypt/live/git.example.com/fullchain.pem";
ssl_certificate_key "/etc/letsencrypt/live/git.example.com/privkey.pem";
proxy_ssl off;
proxy_pass git-server:1022;
proxy_timeout 60m;
}
}
http {
server {
listen unix:/sock/nginx-https.sock ssl proxy_protocol;
ssl_certificate "/etc/letsencrypt/live/www.example.com/fullchain.pem";
ssl_certificate_key "/etc/letsencrypt/live/www.example.com/privkey.pem";
# More normal http server stuff ...
}
}
Now I can forward to any number of SSH servers by listening only on one unix socket, and every SSH server can have their own SSL configuration.
This can be extended with mTLS and logging, but this post is already getting long, so I’ll continue that another day.