Tuesday, February 28, 2012

SSH/HTTP(S) multiplexing with sshttp

Sebastian Krahmer (@steaIth, c-skills) made and released a nice SSH/HTTP(S) multiplexer: sshttp. Such a program is needed when you want to share your HTTP (or HTTPS) port with SSH to be able to use SSH when behind a network that only allows outbound connections to HTTP (or HTTPS) and does not bother to do protocol inspection. Learn more by reading the readme.

I was previously using sslh but sshttp has a killer feature: it uses Linux IP_TRANSPARENT feature with netfilter trickery (marking + specific routing table) to pass the original IP to destination (sshd or httpd) which means your logs still show original IP (and not 127.0.0.1). Since it's a great program, not necessarily easy to set up (not yet? packaged), in this post I'm sharing my setup.

How does it work?

With HTTP the client speaks first (GET/POST/etc. request) whereas with SSH the server speaks first (banner). Similarly, HTTPS uses the SSL/TLS protocol where the client speaks first (ClientHello). Using this property and by waiting a little time, the multiplexer is able to determine whether the traffic should be sent to SSH server or HTTP/HTTPS server.

Setup

I have a main router/firewall in front of the LAN where the server with ssh/http/https is. I keep my webserver running on default ports (80/443), ssh as well (22). I want multiplexing on both http and https, so I'll have two daemons running on 2280 and 2244. Since sshttp requires an INPUT DROP on the ports where traffic is redirected, and I want to keep normal ssh access on port 22, I open a two ports for ssh: 228 for sshttp and 224 for sshttps.
[HTTP]
           _____               ______
 port 80  |     |  port 2280  |      |
--------> | NAT | ----------> | serv | sshttp listening on 2280
          |_____|             |______|            ||
                                            /- multiplex -\
                                           /               \
                                    http server        ssh server
                                  listening on 80   listening on 228
 
[HTTPS]
           _____               ______
port 443  |     |  port 2244  |      |
--------> | NAT | ----------> | serv | sshttps listening on 2244
          |_____|             |______|            ||
                                            /- multiplex -\
                                           /               \
                                   https server        ssh server
                                 listening on 443   listening on 224
 

Installation

Compile from source on github:
$ cd /usr/local/src
$ git clone https://github.com/stealth/sshttp
$ cd sshttp
$ make
$ cp sshttpd /usr/local/sbin/sshttp
$ cp sshttpd /usr/local/sbin/sshttps
Note: different file names and no hardlink or start-stop-daemon is unhappy...

Add dedicated user accounts for the two daemons:
$ adduser --system --ingroup nogroup \
  --home /var/run/sshttp --no-create-home sshttp
$ adduser --system --ingroup nogroup \
  --home /var/run/sshttps --no-create-home sshttps

Debian-like init scripts /etc/init.d/sshttp and /etc/init.d/sshttps:
$ cd /etc/init.d
$ wget https://raw.github.com/StalkR/misc/master/sshttp/etc/init.d/sshttp
$ wget https://raw.github.com/StalkR/misc/master/sshttp/etc/init.d/sshttps
Install them with the new dependency-based init:
$ insserv sshttp
$ insserv sshttps
Or old-school:
$ update-rc.d sshttp defaults
$ update-rc.d sshttps defaults

Configuration

Default configuration for both /etc/default/sshttp /etc/default/sshttps:
$ cd /etc/default
$ wget https://raw.github.com/StalkR/misc/master/sshttp/etc/default/sshttp
$ wget https://raw.github.com/StalkR/misc/master/sshttp/etc/default/sshttps
It's simple:
$ cat /etc/default/sshttp
SSH_PORT=228
HTTP_PORT=80
LISTEN_PORT=2280
USER=sshttp
CHROOT=/var/run/sshttp
MARK=1
TABLE=80
$ cat /etc/default/sshttps
SSH_PORT=224
HTTP_PORT=443
LISTEN_PORT=2244
USER=sshttps
CHROOT=/var/run/sshttps
MARK=2
TABLE=443

Next is to configure iptables, marking and routing. I use ferm (for Easy Rule Making) as a frontend to iptables and I highly recommend it. As long as you are familiar with iptables, it gives you the power to do a complex set of rules easily and ships with an easy init script to safely start/stop/reload the firewall. Much better in so many ways than those manual iptables.sh scripts.

So here are the sshttp rules to integrate with your /etc/init.d/ferm.conf:
domain ip {
  # sshttp
  table mangle {
    chain OUTPUT { proto tcp outerface eth0 sport (80 228) jump SSHTTP; }
    chain PREROUTING { proto tcp sport (80 228) mod socket jump SSHTTP; }
    chain SSHTTP { MARK set-mark 0x1; ACCEPT; } # 1st bit
  }
  # sshttps
  table mangle {
    chain OUTPUT { proto tcp outerface eth0 sport (443 224) jump SSHTTPS; }
    chain PREROUTING { proto tcp sport (443 224) mod socket jump SSHTTPS; }
    chain SSHTTPS { MARK set-mark 0x2; ACCEPT; } # 2nd bit
  }
}
 

Finally, open the sshttp and sshttps dedicated ssh ports by adding in /etc/ssh/sshd_config:
Port 228 # sshttp
Port 224 # sshttps

Run

Reload firewall, ssh and start multiplexers:
$ invoke-rc.d ferm reload
$ invoke-rc.d ssh reload
$ invoke-rc.d sshttp start
$ invoke-rc.d sshttps start

If I did not forget anything, you should be ready to use your http/https ports with ssh and still have your webserver running and serving smoothly. Enjoy!

21 comments:

  1. Cool writeup.Some comments:
    Its probably enough to multiplex either http/ssh or
    https/ssh if its the same machine, except you expect one service to
    be DoS'ed. Second, it is important to block direct access to
    real SSH and HTTP port, otherwise outside folks can still access
    it directly. Take care this cannot happen if you run sshd on port 222.
    (Your NAT might protect you here, but if you dont have a NAT box...)
    sshttp works perfectly with -S 22 -H 8080 and both ports being blocked on
    INPUT. sshttp can still access these services as the traffic is going via
    loopback. So the only outside visible port is 80.
    sshttp would also work on the NAT (or firewall) box to multiplex
    ssh/http for a whole subnet (not just locally) but that'd require more
    complex netfilter setup and additional care to only expose
    the dedicated machines to outside.

    ReplyDelete
  2. Hello,
    How does this impact on performance of normal http traffic?

    ReplyDelete
  3. According to the readme: "sshttpd has small footprint and was optimized for speed so it also runs on heavily loaded web servers."

    ReplyDelete
  4. i use shorewall, please help with using shorewall with sshttps

    ReplyDelete
  5. Hi,

    I'm testing your tool I'm trying to figure out why it doesn't work.
    I tried it locally:

    # /usr/local/sbin/sshttp -S 222 -H 80 -L 2280 -U sshttp -R /var/run/sshttp
    sshttpd: Using HTTP_PORT=80 SSH_PORT=222 and local port=2280. Going background. Using caps/chroot.

    # netstat -luntp | grep 2280
    tcp 0 0 0.0.0.0:2280 0.0.0.0:* LISTEN 16352/sshttp

    # netstat -luntp | grep 222
    tcp 0 0 0.0.0.0:222 0.0.0.0:* LISTEN 15074/sshd

    So when I try to connect via ssh I receive the message below:
    # ssh localhost -p 2280
    ssh_exchange_identification: Connection closed by remote host

    Any suggestion?

    Thanks,
    Mike

    ReplyDelete
  6. Try from another host and make sure you set up the netfilter and routing rules.

    ReplyDelete
  7. Same here.

    $ sudo service sshttps start
    sshttpd: Using HTTP_PORT=443 SSH_PORT=2222 and local port=4443. Going background. Using caps/chroot.

    $ sudo netstat -luntp | grep -E '(2222|443|4443)'
    tcp 0 0 0.0.0.0:2222 0.0.0.0:* LISTEN 390/sshd
    tcp 0 0 0.0.0.0:4443 0.0.0.0:* LISTEN 3088/sshttps
    tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 552/apache2


    Correct me if i'm wrong, but at this stage, I should be able to "ssh localhost" on port 2222 AND 4443. I can connect to port 2222 only.


    $ ssh localhost -p 2222
    user@host's password:

    $ ssh localhost -p 4443
    ssh_exchange_identification: Connection closed by remote host

    ReplyDelete
  8. Yes, connecting directly to sshd or httpd from localhost works.

    Connecting to sshttp(s) from localhost does not work.. because of the way the whole thing works. Try from another host, if your setup is correct it should work.


    On another note, I noted that from some ISPs, connecting to SSH via sshttp on port 80 does not work. The only reason I see is that the ISP uses a transparent proxy to observe all your http traffic :) So, think of sshttp as a tool to reveal ISPs spying on you! (For these same ISPs, sshttp on 443 works fine, apparently they don't inspect this port.)

    ReplyDelete
  9. Does IP_TRANSPARENT not involve the proxy running as root? Meaning, one vulnerability in sshttp and you've lost your box?

    ReplyDelete
  10. Yes and no. Processes can drop their privileges once they don't need them anymore and that's what sshttp does: https://github.com/stealth/sshttp/blob/master/main.cc#L144

    ReplyDelete
  11. Thanks for the guide, works well but kills Virtual Hosts in apache for me, everything just gets redirected to the default site. Any ideas on resolving this?

    ReplyDelete
  12. Sorry I don't know why, and strange it shouldn't affect virtual hosts as they are recognized with "Host:" http header.

    ReplyDelete
  13. Anyone know where I can find a CygWin version (like sslh has) or a full ported Win32/Win64 version?

    ReplyDelete
  14. Great article,
    I used sslh, but was unhappy because of the 127.0.0.1 redirection to apache.
    But I have a problem with ferm:
    I set everything like told, but when starting ferm I get this error:

    service ferm start
    Starting Firewall: fermiptables-restore: line 15 failed
    Failed to run /sbin/iptables-restore

    Firewall rules rolled back.
    failed!

    I only use sshttps, and I set services and ports like this:
    (apache on 80 and 443 , sshd on 22 and 1444)

    netstat -ntapu |grep ":22\|:443\|:80"
    tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 10413/sshd
    tcp 0 0 127.0.0.1:443 0.0.0.0:* LISTEN 12046/apache2
    tcp 0 0 0.0.0.0:224 0.0.0.0:* LISTEN 10413/sshd
    tcp 0 0 0.0.0.0:2244 0.0.0.0:* LISTEN 10870/sshttps
    tcp6 0 0 :::80 :::* LISTEN 12046/apache2
    tcp6 0 0 :::22 :::* LISTEN 10413/sshd
    tcp6 0 0 :::224 :::* LISTEN 10413/sshd

    and this is what i put in ferm.conf:

    51 domain ip {
    52 # sshttp
    53 #table mangle {
    54 # chain OUTPUT { proto tcp outerface eth0 sport (80 228) jump SSHTTP; }
    55 # chain PREROUTING { proto tcp sport (80 228) mod socket jump SSHTTP; }
    56 # chain SSHTTP { MARK set-mark 0x1; ACCEPT; } # 1st bit
    57 #}
    58 # sshttps
    59 table mangle {
    60 chain OUTPUT { proto tcp outerface eth0 sport (443 224) jump SSHTTPS; }
    61 chain PREROUTING { proto tcp sport (443 224) mod socket jump SSHTTPS; }
    62 chain SSHTTPS { MARK set-mark 0x2; ACCEPT; } # 2nd bit
    63 }
    64 }

    so, can you tell me where's my mistake?

    thanks

    alain

    ReplyDelete
    Replies
    1. Hi Alain, try loading ferm with:
      # ferm --slow --lines /etc/ferm/ferm.conf

      With --slow, instead of using "iptables-restore" to load all rules at once (making it hard to know which rule failed), it will use "iptables" and load rules one by one (which you will see with --lines). That way, you'll see which rule fails and better understand the issue.

      It could be because a netfilter module (mangle or socket) isn't available/loaded.

      Delete
    2. Hi StalKR,

      Thanks a lot for your answer.
      Here's the output with your command:

      ferm --slow --lines /etc/ferm/ferm.conf
      /sbin/iptables -t mangle -P FORWARD ACCEPT
      /sbin/iptables -t mangle -P INPUT ACCEPT
      /sbin/iptables -t mangle -P OUTPUT ACCEPT
      /sbin/iptables -t mangle -P PREROUTING ACCEPT
      /sbin/iptables -t mangle -P POSTROUTING ACCEPT
      /sbin/iptables -t mangle -F
      /sbin/iptables -t mangle -X
      /sbin/iptables -t mangle -N SSHTTPS
      /sbin/iptables -t mangle -A OUTPUT --protocol tcp --out-interface eth0 --sport 443 --jump SSHTTPS
      /sbin/iptables -t mangle -A OUTPUT --protocol tcp --out-interface eth0 --sport 224 --jump SSHTTPS
      /sbin/iptables -t mangle -A PREROUTING --protocol tcp --sport 443 --match socket --jump SSHTTPS
      iptables: No chain/target/match by that name.

      Firewall rules rolled back.
      ### and then:
      iptables -L
      Chain INPUT (policy ACCEPT)
      target prot opt source destination

      Chain FORWARD (policy ACCEPT)
      target prot opt source destination

      Chain OUTPUT (policy ACCEPT)
      target prot opt source destination


      But a tried to comment/uncomment lines in my ferm.conf, and the line causing the error is:
      chain PREROUTING { proto tcp sport (443 224) mod socket jump SSHTTPS; }

      so, it seems that a king of 'module' for 'socket' is not loaded?
      I'm quite novice with iptables ...

      I googled about that, but found nothing interesting ...

      Can you help me?

      Thanks again,
      Alain

      Delete
    3. mod socket is provided by the netfilter kernel module xt_socket http://cateee.net/lkddb/web-lkddb/NETFILTER_XT_MATCH_SOCKET.html
      Try to load it with: modprobe xt_socket
      If it works, re-try loading ferm rules and you should be good.

      If it doesn't work, what is your kernel? (uname -r)
      Maybe you need a more recent one and/or re-compile with this enabled.

      Delete
    4. indeed it does not work:
      modprobe xt_socket
      FATAL: Could not load /lib/modules/3.2.13-grsec-xxxx-grs-ipv6-64/modules.dep: No such file or directory
      uname -r
      3.2.13-grsec-xxxx-grs-ipv6-64
      (i use a dedicated server and the kernel is tuned by the provider)
      well, does this mean i have to recompile my kernel?

      Delete
    5. Yes. And for optimal performance base your kernel config on theirs (/boot/config-3.2.13-grsec-xxxx-grs-ipv6-64).
      Alternatively you can use a generic distro kernel (e.g. debian/ubuntu maintain some), or ask the provider to include this module in their next release.

      Delete
    6. Thank you very much for your answers :)

      Delete