Introduction to nginx Memory Pool

nginx implements its own memory managemant system through the ngx_pool_t structure, which appears throughout the codebase. This article examines the allocation strategy and internal mechanics of nginx's memory pool, which handles small memory blocks, large memory blocks, and resource cleanup operations.

Core Structure Definitions

The memory pool architecture consists of three primary structures that work together to manage memory efficiently:

Large Memory Block Structure

typedef struct ngx_pool_large_s  ngx_pool_large_t;
struct ngx_pool_large_s {
   ngx_pool_large_t     *next;         // Link to next large block
   void                 *data;         // Pointer to allocated memory
};

Small Memory Block Structure

typedef struct {
   u_char               *start;        // Beginning of allocatable memory
   u_char               *end;          // End of allocatable memory
   ngx_pool_t           *next;         // Link to next pool manager
   ngx_uint_t            fail_count;   // Allocation failure counter
} ngx_pool_data_t;

Main Pool Management Structure

typedef struct ngx_pool_s            ngx_pool_t;
struct ngx_pool_s {
   ngx_pool_data_t       small;        // Small block management
   size_t                max_size;     // Threshold for large vs small allocations
   ngx_pool_t           *active;       // Current starting pool for allocations
   ngx_chain_t          *chain;        // Buffer chain reference
   ngx_pool_large_t     *large;        // Head of large block list
   ngx_pool_cleanup_t   *cleanup;      // Resource cleanup handlers
   ngx_log_t            *log;          // Logging interface
};

The ngx_pool_t serves as the primary management interface for the entire memory pool system. While multiple pool instances may exist during runtime, users always interact with the original instance returned during creation. These instances connect through the small.next pointer, with active indicating where new allocations should begin searching. The ngx_pool_data_t structure occupies the beginning of ngx_pool_t, enabling type conversion for flexible member access.

Memory Alignment Utilities

nginx provides two alignment macros to ensure proper memory alignment throughout the pool:

#define ngx_align_ptr_value(ptr, align)  \
   (((ptr) + (align - 1)) & ~(align - 1))

#define ngx_align_pointer(ptr, align)                                 \
   (u_char *) (((uintptr_t) (ptr) + ((uintptr_t) align - 1)) & ~((uintptr_t) align - 1))

These macros ensure that all memory allocations meet the required alignment boundaries, reducing cache line crossings and improving performance.

Creating a Memory Pool

When creating a memory pool, nginx establishes the initial allocation boundaries and sets up the management structures. The maximum allocatable size from a single pool is capped at 4095 bytes:

ngx_uint_t  system_page_size = getpagesize();  // Returns 4096 on Linux
#define POOL_ALIGNMENT       16
#define MAX_SMALL_ALLOCATION (system_page_size - 1)  // 4095

ngx_pool_t *
ngx_create_pool(size_t pool_size, ngx_log_t *logger)
{
   ngx_pool_t  *pool;

   // Allocate pool with 16-byte alignment
   pool = ngx_memalign(POOL_ALIGNMENT, pool_size, logger);
   if (pool == NULL) {
       return NULL;
   }

   pool->small.start = (u_char *) pool + sizeof(ngx_pool_t);
   pool->small.end = (u_char *) pool + pool_size;
   pool->small.next = NULL;
   pool->small.fail_count = 0;

   // Calculate available space for small allocations
   size_t available = pool_size - sizeof(ngx_pool_t);
   pool->max_size = (available < MAX_SMALL_ALLOCATION) ? 
                    available : MAX_SMALL_ALLOCATION;

   pool->active = pool;
   pool->chain = NULL;
   pool->large = NULL;
   pool->cleanup = NULL;
   pool->log = logger;

   return pool;
}

After pool creation, the structure establishes a single memory region with properly initialized start and end pointers, ready for allocations.

Memory Allocation Strategy

The allocation function ngx_palloc() determines whether to use the small or large allocation path based on the requested size:

void *
ngx_palloc(ngx_pool_t *pool, size_t size)
{
#if !(NGX_DEBUG_PALLOC)
   if (size <= pool->max_size) {
       return ngx_allocate_small(pool, size, 1);
   }
#endif

   return ngx_allocate_large(pool, size);
}

Small Block Allocation

The active pointer marks where the allocation search begins. This allows the system to skip pools that have accumulated too many allocation failures:

static ngx_inline void *
ngx_allocate_small(ngx_pool_t *pool, size_t size, ngx_uint_t align)
{
   u_char      *memory;
   ngx_pool_t  *current;

   current = pool->active;

   for (;;) {
       memory = current->small.start;

       if (align) {
           memory = ngx_align_pointer(memory, NGX_ALIGNMENT);
       }

       if ((size_t)(current->small.end - memory) >= size) {
           current->small.start = memory + size;
           return memory;
       }

       if (current->small.next == NULL) {
           break;
       }

       current = current->small.next;
   }

   return ngx_create_block(pool, size);
}

When no existing pool has sufficient space, a new block is created. The allocation logic increments the failure counter for pools that couldn't satisfy the request. If a pool accumulates more than 4 failures, its skipped in future allocations:

#define NGX_ALIGNMENT   sizeof(unsigned long)  // Typically 8 bytes

static void *
ngx_create_block(ngx_pool_t *pool, size_t size)
{
   u_char      *memory;
   size_t       block_size;
   ngx_pool_t  *existing, *new_block;

   block_size = (size_t)(pool->small.end - (u_char *)pool);

   memory = ngx_memalign(POOL_ALIGNMENT, block_size, pool->log);
   if (memory == NULL) {
       return NULL;
   }

   new_block = (ngx_pool_t *) memory;

   new_block->small.end = memory + block_size;
   new_block->small.next = NULL;
   new_block->small.fail_count = 0;

   memory += sizeof(ngx_pool_data_t);
   memory = ngx_align_pointer(memory, NGX_ALIGNMENT);
   new_block->small.start = memory + size;

   // Update active pointer and check failure counts
   for (existing = pool->active; existing->small.next; 
        existing = existing->small.next) {
       if (existing->small.fail_count++ > 4) {
           pool->active = existing->small.next;
       }
   }

   existing->small.next = new_block;

   return memory;
}

Large Block Allocation

Large allocations bypass the small block system and use a separate linked list managed through ngx_pool_large_t structures. To prevent excessive traversal, the search for reusable nodes is limited to 3 iterations. New nodes are inserted at the list head using front-insertion:

static void *
ngx_allocate_large(ngx_pool_t *pool, size_t size)
{
   void              *memory;
   ngx_uint_t         iterations;
   ngx_pool_large_t  *node;

   memory = ngx_alloc(size, pool->log);
   if (memory == NULL) {
       return NULL;
   }

   iterations = 0;

   for (node = pool->large; node; node = node->next) {
       if (node->data == NULL) {
           node->data = memory;
           return memory;
       }

       if (iterations++ > 3) {
           break;
       }
   }

   // No reusable large block nodes found
   node = ngx_allocate_small(pool, sizeof(ngx_pool_large_t), 1);
   if (node == NULL) {
       ngx_free(memory);
       return NULL;
   }

   node->data = memory;
   node->next = pool->large;
   pool->large = node;

   return memory;
}

Pool Architecture Summary

The complete memory pool system exhibits several key characteristics:

• All pool instances connect through small.next pointers, forming a chain
• The active pointer typically references a pool several positions down the chain after initial allocations
• All ngx_pool_large_t nodes originate from small block allocations
• Large block nodes attach to the first pool's large list, regardless of which pool created them
• When large block memory is freed, the ngx_pool_large_t node remains allocated for potential reuse

Key Design Considerations

The nginx memory pool implementation prioritizes simplicity and performance through several mechanisms:

Connection Strategies: Small blocks use tail-insertion to maintain allocation order, while large blocks use front-insertion for immediate access to recently allocated nodes.

Failure Threshold: After a pool fails to satisfy 4 consecutive allocation requests, it is skipped during future searches, preventing repeated unsuccessful attempts.

Large Block Optimization: The 3-iteration limit on large block node traversal prevents excessive链表 traversal while still allowing reasonable node reuse.

Memory Alignment: Consistent alignment throughout the pool reduces cache line fragmentation and improves memory access patterns.

While this design introduces some memory overhead through retained failure counters and unused large block nodes, it achieves an effective balance between implementation simplicity, allocation efficiency, and memory utilization.