smpCTF challenge #3 write-up

This week-end was smpCTF, again I played with Nibbles and we ended 1st!

Challenge #3 was just horrible, right? ;) Worse than defcon packet100! But we finally got it, after PPP.

We were given the following instructions and hints:

• Generate a file which has a SHA-1 hash of: 008ce55c7d1b602dc4c4c3ad52a5d064e6d1ef12
• Hint: DRM-0, Linux-1
• _DO NOT BRUTE FORCE_ it's not required...
• Hidden hint (HTML comment): t3=(*((unsigned int *)(key+2)))^(*((unsigned int *)(sec+0x56)));

Understand the hint as: Linux won against DRM (1-0). Using this and the code snippet from the hidden hint quickly leads you to http://decss.zoy.org, a funny page releasing (in 42 different ways) the DeCSS source code (decss.c) to break the CSS DRM, thus allowing Linux platforms to play DVDs.

Sadly, SHA-1 sum of decss.c does not match, would have been too easy. From then, we worked really hard to get the hash. Staff mentioned that we had to consider a part of the file. Which one? I don't know, so I created a small python script trying all possible substrings:

$ python -c 'from hashlib import sha1
from sys import exit
f=open("decss.c","r").read()
for s in xrange(len(f)):
  for l in xrange(1,len(f)-s+1):
    if sha1(f[s:s+l]).hexdigest()=="008ce55c7d1b602dc4c4c3ad52a5d064e6d1ef12":
      print "Found it! offset", s, "length", l
      exit(0)'

It takes about 20 minutes to compute on my computer. Sadly, no match. So we tried many things on the file, for instance converting it to DOS format to have \n (LF) replaced by \r\n (CR+LF). One can do it with unix2dos tool:

$ unix2dos decss.c
unix2dos: converting file decss.c to DOS format ...

Running the python script again to check all the substrings, still no match. In addition to the previous operation, we replace \t (tabulation) with spaces (8, 4, 2). One can do this with UNIX expand, and sponge (from moreutils):

$ expand -t 2 decss.c |sponge decss.c

Or - I prefer - with sed:

$ sed -i 's/\t/  /g' decss.c

Again, we run the python script (on file with CRLF+double space):

$ python -c 'from hashlib import sha1
from sys import exit
f=open("decss.c","r").read()
for s in xrange(len(f)):
  for l in xrange(1,len(f)-s+1):
    if sha1(f[s:s+l]).hexdigest()=="008ce55c7d1b602dc4c4c3ad52a5d064e6d1ef12":
      print "Found it! offset", s, "length", l
      exit(0)'
Found it! offset 9149 length 572

In about 10 minutes, it finds a match! What is it?

$ head -c$[9149+572] decss.c |tail -c572 |sha1sum
008ce55c7d1b602dc4c4c3ad52a5d064e6d1ef12  -
$ head -c$[9149+572] decss.c |tail -c572

void CSSdescramble(unsigned char *sec,unsigned char *key) {
  unsigned int t1,t2,t3,t4,t5,t6;
  unsigned char *end=sec+0x800;
  t1=key[0]^sec[0x54]|0x100;
  t2=key[1]^sec[0x55];
  t3=(*((unsigned int *)(key+2)))^(*((unsigned int *)(sec+0x56)));
  t4=t3&7;
  t3=t3*2+8-t4;
  sec+=0x80;
  t5=0;
  while(sec!=end) {
    t4=CSSt2[t2]^CSSt3[t1];
    t2=t1>>1;
    t1=((t1&1)<<8)^t4;
    t4=CSSt5[t4];
    t6=(((((((t3>>3)^t3)>>1)^t3)>>8)^t3)>>5)&0xff;
    t3=(t3<<8)|t6;
    t6=CSSt4[t6];
    t5+=t6+t4;
    *sec++=CSSt1[*sec]^(t5&0xff);
    t5>>=8;
  }
}

It's CSSdescramble function (from which the line of code of the hidden hint was extracted) that can also be seen on Wikipedia.

Submit this (with CRLF & double space as tabs) and the flag is yours:

Challenge ID: 20434ef4
Flag: DiDyouEXPECTthatAtALl?