Reverse Engineering ELF Initialization and Exit Handlers for XOR Decryption

Challenge Overview

The binary implements a flag verification mechanism using chained XOR operations. Understanding the execution order is critical: the program applies XOR transformations both before and after the main function executes, leveraging ELF initialization sections and exit handlers.

ELF Initialization Function Table

In ELF binaries, the Initialization Function Table stores addresses of functions that execute automatically before main(). This mechanism handles startup tasks such as global variable initialization and hardware setup. The startup code iterates through entries in this table, calling each function in sequence.

In this binary, function pre_init_transform() is registered in the initialization table:

unsigned __int64 pre_init_transform()
{
    signed int idx;
    unsigned __int64 canary;

    canary = __readfsqword(0x28u);
    for ( idx = 0; idx <= 33; ++idx )
        encrypted_buffer[idx] ^= 2 * idx + 65;
    return __readfsqword(0x28u) ^ canary;
}

This function XORs each byte of encrypted_buffer with a computed key: (2 * index + 65). This occurs before main() starts.

Exit Handler Registration

The main function calls register_exit_handler(), which internally uses _cxa_atexit. This function registers a callback to execute during program termination. The registered callback is check_result():

unsigned __int64 check_result()
{
    unsigned __int64 canary;

    canary = __readfsqword(0x28u);
    if ( !strcmp(encrypted_buffer, target_buffer) )
        puts("Congratulations!");
    else
        puts("Wrong!");
    return __readfsqword(0x28u) ^ canary;
}

When exit() is called or main() returns, this handler compares the processed buffer against the target and displays the result.

Main Function Logic

The main function reads user input and applies a second XOR operation:

__int64 __fastcall main(__int64 a1, char **a2, char **a3)
{
    signed int idx;
    char user_input[40];
    unsigned __int64 canary;

    canary = __readfsqword(0x28u);
    register_exit_handler(check_result, a2, a3);
    fgets(user_input, 35, stdin);
    for ( idx = 0; idx <= 33; ++idx )
        encrypted_buffer[idx] ^= user_input[idx];
    return 0LL;
}

Decryption Process

The complete execution flow performs two XOR operations:

  1. Pre-main phase: encrypted_buffer[i] ^= (2 * i + 65)
  2. Main phase: encrypted_buffer[i] ^= user_input[i]
  3. Exit phase: Compare encrypted_buffer with target_buffer

To retrieve the flag, XOR the original encrypted string with both keys. The target buffer contains the expected result after both transformations:

encrypted_str = 'qasxcytgsasxcvrefghnrfghnjedfgbhn'
target_bytes = [
    0x56, 0x4E, 0x57, 0x58, 0x51, 0x51, 0x09, 0x46,
    0x17, 0x46, 0x54, 0x5A, 0x59, 0x59, 0x1F, 0x48,
    0x32, 0x5B, 0x6B, 0x7C, 0x75, 0x6E, 0x7E, 0x6E,
    0x2F, 0x77, 0x4F, 0x7A, 0x71, 0x43, 0x2B, 0x26,
    0x89, 0xFE, 0x00
]

flag = ''
for i in range(33):
    first_key = 2 * i + 65
    second_key = target_bytes[i]
    plaintext = ord(encrypted_str[i]) ^ first_key ^ second_key
    flag += chr(plaintext)

print(flag)

Key Takeaways

  • ELF binaries can execute code before main() through initialization function tables, commonly used for constructor-like operations.
  • _cxa_atexit registers handlers that execute during program termination, enabling destructor-style cleanup in C++ environments.
  • When analyzing crackme challenges, examine all execution phases including pre-main initialization and exit callbacks.

Tags: reverse-engineering ELF-binary XOR-encryption CTF binary-analysis

Posted on Thu, 04 Jun 2026 17:21:17 +0000 by jpbox