Function Calling Conventions

Definition

In computer science, a calling convention defines how subroutines receive parameters from calllers and return results. Different calling conventions vary in:

• Where parameters and return values are stored
• The order of parameter passing
• Who manages stack cleanup before or after the call
• Which registers the called function can use directly

Three Common Calling Conventions

The main differences between calling conventions involve:

1. Whether parameters are passed via stack or registers
2. Whether the caller or callee cleans up the parameter space

Let's examine these conventions using an online compiler (https://godbolt.org) with the 'x86 msvc v19.latest' option.

Specifying Calling Conventions

You can specify a calling convention by adding a keyword before the function declaration:

int calculate(int first, int second) {
    int result = first + second;
    return result;
}

int main() {
    int x = 1, y = 2;
    int z = calculate(x, y);
    return 0;
}

__cdecl Convention

In the __cdecl convention:

• Parameters are passed on the stack
• The caller cleans up the stack space

Assembly analysis shows the main function pushes parameters onto the stack before calling the function, and adjusts the stack pointer afterward.

__stdcall Convention

In the __stdcall convention:

• Parameters are passed on the stack
• The callee cleans up the stack space

The __stdcall function internally handles stack cleanup, equivalent to an 'add esp, X' instruction where X is the total parameter size.

__fastcall Convention

int __fastcall compute(int a, int b, int c) {
    int total = a + b + c;
    return total;
}

int main() {
    int val1 = 1, val2 = 2, val3 = 3;
    int result = compute(val1, val2, val3);
    return 0;
}

In the __fastcall convention:

• The first parameters are passed in registers (typically ecx and edx for x86)
• Additional parameters are passed on the stack
• The callee cleans up the stack space

Assembly analysis shows that the main function passes the first two parameters in registers while pushing the remaining parameters onto the stack. The called function then creates its own stack space to store register values if needed.

Practice Exercise

Analyze how the following code works with the fastcall convention:

int calculate(int a, int b, int c, int d) {
    return a*a + b*b + 2*a*b + c + d;
}

int main() {
    int answer = calculate(3, 4, 5, 6);
    return 0;
}

Solution approach:

1. With fastcall, the first two parameters (3, 4) are passed in registers, while the remaining parameters (5, 6) are pushed onto the stack
2. When the call instruction executes, the return address is pushed onto the stack
3. The called function allocates space for local variables and intermediate results
4. Before returning, the function cleans up its allocated stack space

Technical Summary

In assembly language, function calling involves two primary steps:

1. Parameter passing
2. Function execution

Parameter passing typically uses the push instruction to place parameters on the stack, followed by the call instruction to transfer execution to the function's code. The push instruction marks the beginning of the function call process, as it initiates parameter passing. The call instruction then jumps to the function's code, completing the call transition.

Only after the call instruction executes does the program truly enter the function's code. Before the call instruction, the program remains in the caller's context, having already begun parameter passing.