Practical Reverse Engineering p.17 #4

Question number 4 on page 17 of Practical Reverse Engineering is as follows:

In all of the calling conventions explained, the return value is stored in a 32-bit register (EAX). What happens when the return value does not fit in a 32-bit register? Write a program to experiment and evaluate your answer. Does the mechanism change from compiler to compiler?

Here is a C function which returns a value that is a 128-bit struct:

struct toolong {
    int a;
    int b;
    int c;
    int d;
};

struct toolong get_toolong()
{
    struct toolong t;

    t.a = 0xDABBAD00;
    t.b = 0xDEADBEEF;
    t.c = 0xBADF00D;
    t.d = 0xBA5EBA11;

    return t;
}

Here is what happens when it is compiled with: gcc -O3 -m32

; GCC 4.8.2

get_toolong:
    mov    eax, dword ptr [esp+0x4]
    mov    dword ptr [eax], 0xdabbad00
    mov    dword ptr [eax+0x4], 0xdeadbeef
    mov    dword ptr [eax+0x8], 0xbadf00d
    mov    dword ptr [eax+0xc], 0xba5eba11
    ret    0x4

It is obvious that the return value of EAX is actually a pointer. The returned struct is stored on the stack.

The compiler for Microsoft Visual Studio 2013 produced essentially the same assembly, but with a full function frame. It was compiled with: /GS- /Gd /Gy /Ox

; MSVC 17.00.60610.1

get_toolong:
    push    ebp
    mov     ebp, esp
    mov     eax, [ebp + 8]
    mov     dword ptr [eax], 0DABBAD00h
    mov     dword ptr [eax + 4], 0DEADBEEFh
    mov     dword ptr [eax + 8], 0BADF00Dh
    mov     dword ptr [eax + 0Ch], 0BA5EBA11h
    pop     ebp
    retn

Notice that the GCC compiler stores the struct at [ESP + 4] and MSVC at [EBP + 8]. This means space needs to be pre-allocated by the caller for each.