How Reverse Engineering Binaries Actually Works
Try the interactive lab for this articleTake the quiz (6 questions)Reverse engineering compiled binary executables is the systematic process of extracting high-level operational logic, control flow structures, and data schemas from raw machine instructions without access to source code. Whether analyzing malicious software payloads, performing vulnerability research on closed-source embedded firmware, or auditing third-party native libraries, reverse engineering requires a precise understanding of executable file formats, instruction set architecture (ISA) semantics, Intermediate Representation (IR) transformations, dynamic execution states, and compiler optimization patterns.
At the hardware level, a compiled executable is a sequence of bytes containing machine code opcodes, static data tables, import relocation records, and operating system loader directives. Operating system loaders parse these binary containers to map code and data segments into virtual memory pages, resolve dynamic library references, and initialize execution at a defined instruction entry point. Reverse engineering systematically reverses this compilation process using static disassemblers, intermediate representation decompilers, dynamic debuggers, and symbolic execution engines.
Executable File Structures: ELF vs PE Headers and Binary Layout
Operating system loaders read structured binary containers to construct a process virtual address space. On Linux and Unix systems, the standard format is the Executable and Linkable Format (ELF). On Windows systems, it is the Portable Executable (PE) format. Both containers define how executable code sections, initialized data, import tables, and relocation entries are laid out on disk and mapped into virtual memory.
The Executable and Linkable Format (ELF)
An ELF binary consists of an ELF header, a Program Header Table describing memory segments, a Section Header Table describing link-time sections, and binary payload data.
+-------------------------------------------------------+
| ELF Header (64 bytes) |
| Magic: \x7f ELF, Architecture, Entry Point (e_entry) |
+-------------------------------------------------------+
| Program Header Table (Phdr) |
| PT_LOAD: .text, .rodata (Read / Execute) |
| PT_LOAD: .data, .bss (Read / Write) |
| PT_DYNAMIC: Dynamic linking structures |
+-------------------------------------------------------+
| Section Header Table (Shdr) |
| .text : Executable instructions |
| .rodata : Constant strings and jump tables |
| .data : Initialized global variables |
| .bss : Uninitialized zero-filled memory |
| .symtab : Symbol table (Function names and offsets) |
| .got : Global Offset Table |
| .plt : Procedure Linkage Table |
+-------------------------------------------------------+The 64-bit ELF header (Elf64_Ehdr) resides at offset 0 of the binary file and dictates how the remaining headers and payload data are parsed:
typedef struct {
unsigned char e_ident[16]; /* Magic number \x7fELF, class, data encoding, version */
uint16_t e_type; /* Object file type (ET_EXEC=2, ET_DYN=3, ET_REL=1) */
uint16_t e_machine; /* Target architecture (EM_X86_64 = 0x3E) */
uint32_t e_version; /* EV_CURRENT (1) */
uint64_t e_entry; /* Virtual address of execution entry point */
uint64_t e_phoff; /* Program header table file offset */
uint64_t e_shoff; /* Section header table file offset */
uint32_t e_flags; /* Processor-specific flags */
uint16_t e_ehsize; /* ELF header size in bytes (64 bytes) */
uint16_t e_phentsize; /* Size of one program header table entry */
uint16_t e_phnum; /* Number of program header entries */
uint16_t e_shentsize; /* Size of one section header table entry */
uint16_t e_shnum; /* Number of section header entries */
uint16_t e_shstrndx; /* Section header index for name string table */
} Elf64_Ehdr;The operating system loader processes Program Header entries (Elf64_Phdr) to instantiate process virtual memory segments:
typedef struct {
uint32_t p_type; /* Segment type (PT_LOAD=1, PT_DYNAMIC=2, PT_INTERP=3) */
uint32_t p_flags; /* Segment flags (PF_X=1, PF_W=2, PF_R=4) */
uint64_t p_offset; /* File offset of segment */
uint64_t p_vaddr; /* Virtual address in memory */
uint64_t p_paddr; /* Physical address (unused on modern systems) */
uint64_t p_filesz; /* Segment byte count in file */
uint64_t p_memsz; /* Segment byte count in memory */
uint64_t p_align; /* Memory alignment boundary */
} Elf64_Phdr;Segments with flags PF_X | PF_R (value 0x5) map executable code (.text) and read-only constants (.rodata). Segments with PF_W | PF_R (value 0x6) map read-write data (.data and .bss). When p_memsz exceeds p_filesz, the loader zero-fills the remaining memory region to allocate .bss uninitialized variables.
ELF Section Headers and Symbol Resolution
While Program Headers dictate runtime memory layout for the kernel loader, Section Headers (Elf64_Shdr) provide logical boundary mappings essential for static analysis and linking.
typedef struct {
uint32_t sh_name; /* Section name offset in .shstrtab */
uint32_t sh_type; /* Section type (SHT_PROGBITS=1, SHT_SYMTAB=2, SHT_STRTAB=3) */
uint64_t sh_flags; /* Section attribute flags (SHF_WRITE=1, SHF_ALLOC=2, SHF_EXECINSTR=4) */
uint64_t sh_addr; /* Virtual address of section in memory */
uint64_t sh_offset; /* File offset of section */
uint64_t sh_size; /* Section size in bytes */
uint32_t sh_link; /* Section header index link */
uint32_t sh_info; /* Additional section information */
uint64_t sh_addralign; /* Address alignment boundary */
uint64_t sh_entsize; /* Entry size if section holds fixed-size array */
} Elf64_Shdr;Key ELF sections analyzed during reverse engineering include:
.text(SHT_PROGBITS, SHF_ALLOC | SHF_EXECINSTR): Raw compiled executable machine instructions..rodata(SHT_PROGBITS, SHF_ALLOC): Read-only constants, string literals, and switch jump tables..data(SHT_PROGBITS, SHF_ALLOC | SHF_WRITE): Initialized global and static variables..bss(SHT_NOBITS, SHF_ALLOC | SHF_WRITE): Uninitialized global variables allocated dynamically by the loader..symtab(SHT_SYMTAB): Link-time symbol table mapping function names and global variables to virtual addresses..dynsym(SHT_DYNSYM): Dynamic symbol table required for resolving shared library functions at runtime.
Symbol table entries (Elf64_Sym) contain symbol names, virtual memory addresses, and data types:
typedef struct {
uint32_t st_name; /* Symbol name offset in .strtab / .dynstr */
unsigned char st_info; /* Type (STT_FUNC=2, STT_OBJECT=1) and Binding (STB_GLOBAL=1) */
unsigned char st_other; /* Visibility flags */
uint16_t st_shndx; /* Associated section header index */
uint64_t st_value; /* Symbol virtual address or value */
uint64_t st_size; /* Symbol byte size */
} Elf64_Sym;When binaries are stripped during release compilation, the .symtab section and string table .strtab are removed. Reverse engineers must then rely on dynamic symbols in .dynsym and PLT/GOT stubs to identify function boundaries.
The Portable Executable (PE/PE32+) Format
Windows binaries use the Portable Executable format. A 64-bit PE file (PE32+) begins with a legacy DOS header (IMAGE_DOS_HEADER) for backwards compatibility. The initial word of the DOS header contains the magic bytes 0x5A4D ("MZ"). The field at offset 0x3C (e_lfanew) stores the file offset pointing to the main IMAGE_NT_HEADERS64 structure.
typedef struct _IMAGE_DOS_HEADER {
WORD e_magic; /* Magic number (0x5A4D = "MZ") */
WORD e_cblp; /* Bytes on last page of file */
WORD e_cp; /* Pages in file */
/* ... intermediate DOS stub fields ... */
LONG e_lfanew; /* File offset to NT header (IMAGE_NT_HEADERS) */
} IMAGE_DOS_HEADER;
typedef struct _IMAGE_NT_HEADERS64 {
DWORD Signature; /* PE\0\0 (0x00004550) */
IMAGE_FILE_HEADER FileHeader; /* Architecture, section count, timestamp */
IMAGE_OPTIONAL_HEADER64 OptionalHeader; /* Entry point, ImageBase, DataDirectories */
} IMAGE_NT_HEADERS64;The IMAGE_FILE_HEADER identifies target system parameters:
typedef struct _IMAGE_FILE_HEADER {
WORD Machine; /* Target ISA (0x8664 = x64, 0x014C = x86) */
WORD NumberOfSections; /* Number of section header entries */
DWORD TimeDateStamp; /* UNIX timestamp of binary compilation */
DWORD PointerToSymbolTable; /* COFF symbol table offset */
DWORD NumberOfSymbols; /* Number of COFF symbols */
WORD SizeOfOptionalHeader; /* Size of IMAGE_OPTIONAL_HEADER64 */
WORD Characteristics; /* Flags (IMAGE_FILE_EXECUTABLE_IMAGE = 0x0002) */
} IMAGE_FILE_HEADER;The IMAGE_OPTIONAL_HEADER64 dictates runtime loader configuration:
typedef struct _IMAGE_OPTIONAL_HEADER64 {
WORD Magic; /* 0x020B for PE32+ (64-bit), 0x010B for PE32 */
BYTE MajorLinkerVersion;
BYTE MinorLinkerVersion;
DWORD SizeOfCode;
DWORD AddressOfEntryPoint; /* RVA of entry point (mainCRTStartup) */
DWORD BaseOfCode; /* RVA of code section */
ULONGLONG ImageBase; /* Preferred load address (0x140000000) */
DWORD SectionAlignment; /* Memory page alignment (typically 0x1000) */
DWORD FileAlignment; /* Disk file alignment (typically 0x200) */
/* ... OS version and subsystem fields ... */
DWORD NumberOfRvaAndSizes; /* Number of DataDirectory entries (16) */
IMAGE_DATA_DIRECTORY DataDirectory[16]; /* Array of key structure pointers */
} IMAGE_OPTIONAL_HEADER64;PE Data Directories and Import Address Table (IAT)
The DataDirectory array maps vital internal PE data structures by Relative Virtual Address (RVA). Key indexes include:
- Index 0 (
IMAGE_DIRECTORY_ENTRY_EXPORT): Export Directory containing exported function RVAs. - Index 1 (
IMAGE_DIRECTORY_ENTRY_IMPORT): Import Directory (IMAGE_IMPORT_DESCRIPTORarray). - Index 2 (
IMAGE_DIRECTORY_ENTRY_RESOURCE): Resource Directory (icons, manifests, embedded binaries). - Index 3 (
IMAGE_DIRECTORY_ENTRY_EXCEPTION): Exception Directory (.pdatacontaining stack unwind structures for x64 SEH). - Index 12 (
IMAGE_DIRECTORY_ENTRY_IAT): Import Address Table (IAT) virtual address range.
The Windows loader processes IMAGE_IMPORT_DESCRIPTOR entries to load dynamic libraries (kernel32.dll, ntdll.dll) and resolve function pointers:
typedef struct _IMAGE_IMPORT_DESCRIPTOR {
DWORD OriginalFirstThunk; /* RVA to Import Lookup Table (INT) */
DWORD TimeDateStamp; /* 0 if not bound */
DWORD ForwarderChain; /* Forwarder chain index */
DWORD Name; /* RVA to DLL name ASCII string */
DWORD FirstThunk; /* RVA to Import Address Table (IAT) */
} IMAGE_IMPORT_DESCRIPTOR;[ Disk PE Binary ] [ Memory Virtual Address Space ]
+----------------------------+ +----------------------------+
| IMAGE_IMPORT_DESCRIPTOR | | Import Address Table (IAT) |
| Name: "kernel32.dll" | | [FirstThunk RVA] |
| OriginalFirstThunk -> INT | +----------------------------+
| FirstThunk -> IAT | |
+----------------------------+ v (Loader overwrites)
| +----------------------------+
v | Resolved Address: |
+----------------------------+ | 0x7FFF8A123040 |
| Import Lookup Table (INT) | | (kernel32!CreateFileW) |
| [0] RVA -> "CreateFileW" | +----------------------------+
| [1] RVA -> "ReadFile" |
+----------------------------+When the PE executable is loaded:
- The loader reads
Nameto load the target DLL into memory via internal kernel calls. - The loader iterates through the Import Lookup Table (
OriginalFirstThunk), extracting function name strings (IMAGE_IMPORT_BY_NAME) or ordinal numbers. - The loader queries the loaded DLL export table to retrieve function virtual addresses.
- The loader writes resolved absolute function addresses into the corresponding slot in the Import Address Table (
FirstThunk). - Application code invokes external API functions via indirect call instructions targeting the IAT slot:
call QWORD PTR [rip + iat_CreateFileW_offset].
Dynamic Symbol Resolution: GOT and PLT Mechanics
Executables dynamically link against shared libraries (such as libc.so on Linux) to invoke system APIs. Because shared library functions reside at arbitrary virtual addresses determined at runtime, Position Independent Executables (PIE) utilize the Global Offset Table (GOT) and Procedure Linkage Table (PLT).
Call Site: call printf@plt
|
v
+------------------------------------------+
| PLT Entry (.plt) |
| jmp QWORD PTR [rip + GOT_entry_offset] |--+ (Initial jump points to stub)
+------------------------------------------+ |
^ |
| (Resolved address) v
+------------------------------------------+ |
| GOT Entry (.got.plt) |<-+
| Holds address of libc!printf |
+------------------------------------------+
|
v (After dynamic linker resolution)
+------------------------------------------+
| libc.so virtual memory page |
| Execution of printf code |
+------------------------------------------+Lazy dynamic binding operates via the following execution steps:
- Initial Call: Execution reaches
call printf@plt. The PLT code executes an indirect jump loading the address stored inside.got.plt. - Unresolved Stub: Prior to resolution,
.got.pltcontains the address of the PLT resolution stub immediately following the jump instruction. - Linker Invocation: The PLT stub pushes the symbol relocation index onto the stack and branches to the dynamic linker resolver (
ld-linux.so). - Relocation Patching: The dynamic linker parses the
Elf64_Relaentry (typeR_X86_64_JUMP_SLOT), locatesprintfinsidelibc.so, and overwrites the.got.pltslot with the absolute virtual address ofprintf. - Direct Jump: Subsequent calls to
printf@pltread the resolved address from.got.pltdirectly, executing without dynamic linker overhead.
When analyzing stripped binaries, auditing .got.plt entries and Elf64_Rela relocation tables allows reverse engineers to identify external API functions despite missing debug symbols.
Disassembly and Decompilation Mechanics
Disassembly parses machine code bytes into human-readable assembly instructions. Decompilation lifts assembly instructions into an intermediate representation to reconstruct structured C/C++ source constructs.
Instruction Decoding: Machine Code Structure (x86-64 Encoding)
Translating variable-length x86-64 machine instructions (ranging from 1 to 15 bytes per instruction) into assembly requires parsing complex instruction byte layouts:
+-------------------+-----------------+---------------+---------------+---------------+------------------+-----------------+
| Legacy Prefixes | REX Prefix | Opcode | ModR/M Byte | SIB Byte | Displacement | Immediate |
| (0 - 4 Bytes) | (0 - 1 Byte) | (1 - 3 Bytes) | (0 - 1 Byte) | (0 - 1 Byte) | (0 - 4 Bytes) | (0 - 4 Bytes) |
| e.g. 0x66, 0xF0 | 0x40 - 0x4F | e.g. 0x8B | Mod|Reg|R/M | Scale|Idx|Base| e.g. 0x10000000 | e.g. 0x00000005 |
+-------------------+-----------------+---------------+---------------+---------------+------------------+-----------------+Instruction field breakdowns:
- Legacy Prefixes: Optional byte flags modifying operand size (
0x66), address size (0x67), lock bus operations (0xF0), or string repetition (0xF2,0xF3). - REX Prefix: Mandatory byte prefix (values
0x40through0x4F) introduced in 64-bit mode to access 64-bit registers (RAX-R15). REX bit flags follow the pattern0100WRXB:- Bit 3 (
W): 64-bit operand size flag when set to 1. - Bit 2 (
R): Extension to the ModR/MRegfield (selects registersR8-R15). - Bit 1 (
X): Extension to the SIBIndexfield. - Bit 0 (
B): Extension to the ModR/MR/Mfield or SIBBasefield.
- Bit 3 (
- Opcode: 1 to 3 bytes defining the CPU operation (for example,
0x8BforMOV,0x01forADD). - ModR/M Byte: Dictates operand addressing modes:
- Bits [7:6] (
Mod): Addressing mode (00 = indirect[reg], 01 = displacement[reg + disp8], 10 = displacement[reg + disp32], 11 = direct register operand). - Bits [5:3] (
Reg/Opcode): Source/Destination register or opcode extension. - Bits [2:0] (
R/M): Base register or addressing combination. WhenModis 00 andR/Mis 101, x86-64 uses RIP-relative addressing ([RIP + disp32]).
- Bits [7:6] (
- SIB Byte: Scale-Index-Base byte used for array indexing (
[Base + Index * Scale]):- Bits [7:6] (
Scale): Index multiplier scale factor (00=1, 01=2, 10=4, 11=8). - Bits [5:3] (
Index): Index register selection (RAX-R15). - Bits [2:0] (
Base): Base memory address register.
- Bits [7:6] (
- Displacement and Immediate: Signed integer values specifying memory offsets or numeric constants.
Consider decoding the byte sequence 48 8B 44 89 10:
48: REX prefix (01001000-> REX.W=1 for 64-bit operation).8B: Opcode (MOVregister from memory).44: ModR/M byte (00 001 100-> Mod=01 for disp8, Reg=001 forRAX, R/M=100 indicating SIB byte follows).89: SIB byte (10 001 001-> Scale=10 [factor 4], Index=001 [RCX], Base=001 [RCX]).10: Displacement byte (0x10).- Decoded Assembly Result:
mov rax, [rcx + rcx*4 + 0x10].
Disassembly Algorithms: Linear Sweep vs Recursive Traversal
Parsing binary byte streams into assembly instructions requires specialized traversal strategies:
-
Linear Sweep: Sequentially decodes instructions byte by byte from the start of
.textto the end.- Disadvantage: Compilers inline constant data tables (such as switch statement jump tables or literal constants) into code sections. Linear sweep misinterprets data bytes as instruction opcodes, causing instruction boundary misalignment (cascade disassembler errors).
-
Recursive Traversal: Begins execution decoding at entry points (
e_entry, exported symbols) and recursively follows all control flow branch targets (jmp,call,jcc).- Advantage: When encountering a conditional branch (
je 0x401080), recursive traversal queues0x401080and the fall-through instruction address while bypassing inline data blocks or NOP padding. Ghidra, IDA Pro, and radare2 use recursive traversal.
- Advantage: When encountering a conditional branch (
Machine Bytes: E8 2B 00 00 00 48 89 C7 EB 05 90 90 90 90 90 48 83 C4 08
Linear Sweep: Decodes sequentially without checking branch targets.
Recursive CFG: Follows E8 (CALL) -> Follows EB (JMP offset 0x05) -> Skips 90 NOP bytes -> Decodes 48 83 C4 08.Control Flow Graph (CFG) Construction & SSA Transformation
To analyze decompiled control flow, disassemblers partition instructions into basic blocks to form a Control Flow Graph $G = (V, E)$. A basic block is a sequence of consecutive instructions with exactly one entry instruction (the leader) and one exit instruction (a branch, call, or return).
Leader Identification Algorithm:
- The first instruction in the code section is a leader.
- Any instruction that is the target of a conditional or unconditional branch (
jmp,jcc,call) is a leader. - Any instruction immediately following a branch or return instruction is a leader.
+------------------------------------------+
| Basic Block 1 (Leader: 0x401000) |
| mov rdi, [rsp + 0x10] |
| test rdi, rdi |
| jz 0x401030 (Conditional Branch) |
+------------------------------------------+
/ \
/ \
v v
+------------------------+ +------------------------+
| Block 2 (Fall-through) | | Block 3 (Branch Target)|
| mov rax, [rdi + 0x08] | | xor eax, eax |
| jmp 0x401040 | | ret |
+------------------------+ +------------------------+Once the CFG is built, the decompiler converts the intermediate representation into Static Single Assignment (SSA) form. In SSA form, every variable is defined exactly once. Where distinct control paths converge, the decompiler places Phi ($\phi$) functions to merge variable versions.
Dominance Frontier and Phi Node Placement:
- Node $D$ dominates node $N$ ($D ext{ dom } N$) if every path from the graph entry to $N$ must pass through $D$.
- Node $D$ strictly dominates $N$ if $D ext{ dom } N$ and $D eq N$.
- The Dominance Frontier $DF(X)$ of a node $X$ is the set of all nodes $Y$ such that $X$ dominates a predecessor of $Y$, but does not strictly dominate $Y$ itself.
- Cytron's Algorithm places $\phi$-nodes at the Dominance Frontiers of basic blocks containing variable assignments, establishing unified variable data flow for decompiler type propagation.
Ghidra P-Code and Intermediate Representation (IR)
Directly lifting x86-64 assembly to C code is complex due to architecture-specific implicit side effects (flag registers, stack manipulation). Decompilers map native assembly into an architecture-agnostic Intermediate Representation (IR).
Ghidra uses the Sledge specification language to translate native opcodes into P-Code micro-operations. P-Code operates on abstract data units called Varnodes, defined by a triple (AddressSpace, Offset, Size):
Native Assembly:
add rax, rbx
P-Code Translation (Micro-operations):
1. $u0 = INT_ADD rax, rbx (Varnode addition)
2. zf = INT_EQUAL $u0, 0 (Zero flag evaluation)
3. sf = INT_LESS $u0, 0 (Sign flag evaluation)
4. cf = INT_CARRY rax, rbx (Carry flag evaluation)
5. rax = COPY $u0 (Commit result to rax varnode)P-Code opcodes explicitly expose register side effects:
- Arithmetic:
INT_ADD,INT_SUB,INT_MULT,INT_DIV,INT_2COMP. - Logical:
INT_AND,INT_OR,INT_XOR,INT_LEFT,INT_RIGHT. - Comparison:
INT_EQUAL,INT_NOTEQUAL,INT_LESS,INT_CARRY. - Control Flow:
BRANCH,CBRANCH,CALL,RETURN. - Memory:
LOAD(read from space),STORE(write to space).
The Decompiler Pipeline: From P-Code to C Abstract Syntax Tree
The decompiler transforms P-Code micro-operations into structured C AST output through sequential optimization phases:
+------------------+ +-------------------+ +-----------------------+
| Machine Opcodes | --> | Recursive Decoder | --> | P-Code IR Generation |
+------------------+ +-------------------+ +-----------------------+
|
v
+------------------+ +-------------------+ +-----------------------+
| High-Level C AST | <-- | Struct & Control | <-- | Static Single |
| (Decompiled Code)| | Flow Recovery | | Assignment (SSA Form) |
+------------------+ +-------------------+ +-----------------------+- SSA Form Transformation: Varnodes are assigned unique iteration indexes. Phi functions are placed at dominance frontiers.
- Dead Code Elimination and Constant Folding: Flag evaluations (
zf,sf,cf) that are never read by subsequent conditional branches are pruned. Expressions containing literal values undergo algebraic folding. - Expression Structuring and Type Propagation: Data access sizes and array stride patterns are analyzed to infer variable types (
char*,int,struct). Raw conditional branch nodes are synthesized into structuredif-else,while,for, andswitchstatements using T1-T2 interval graph reduction algorithms.
Dynamic Debugging with GDB
Static analysis reveals binary code structures, but dynamic debugging exposes runtime memory allocations, register modifications, and execution flow. On Linux, GDB (GNU Debugger) interfaces with the kernel ptrace system call to control target process execution.
Software vs Hardware Breakpoints and Watchpoints
Debuggers use software and hardware execution traps to intercept execution:
Software Breakpoints
Setting a software breakpoint (break *0x401050) causes GDB to modify the target process memory:
- GDB reads and backs up the original instruction byte at virtual address
0x401050(for example,0x55corresponding topush rbp). - GDB overwrites
0x401050with the 1-byte opcode0xCC(INT 3). - Execution resumes (
PTRACE_CONT). - When the CPU encounters
0xCC, it raises Vector 3 (SIGTRAP), pausing execution and transferring control back to GDB. - To step over or resume, GDB restores the original byte
0x55, decrements the instruction pointer (RIP = RIP - 1), single-steps the original instruction (PTRACE_SINGLESTEP), re-inserts0xCC, and continues execution.
Hardware Breakpoints and Watchpoints
Software breakpoints cannot monitor data reads or writes because placing 0xCC in data memory corrupts variables. Hardware watchpoints utilize x86-64 debug registers (DR0 through DR7).
DR0: Virtual Address 1 (0x7FFFF7A12040)
DR1: Virtual Address 2 (0x000000000000)
DR2: Virtual Address 3 (0x000000000000)
DR3: Virtual Address 4 (0x000000000000)
DR6: Debug Status Register (Bits B0-B3 indicate which DR0-DR3 register triggered)
DR7: Debug Control Register
+--------------------+-------------------+-------------------+-------------------+
| DR3 Condition/Len | DR2 Condition/Len | DR1 Condition/Len | DR0 Condition/Len | ...
| [31:28] | [27:24] | [23:20] | [19:16] |
+--------------------+-------------------+-------------------+-------------------+Control fields in DR7 configure monitoring rules:
- Condition Bits (
R/W0throughR/W3):00= Execute breakpoint,01= Data write watchpoint,11= Data read/write watchpoint. - Length Bits (
LEN0throughLEN3):00= 1 byte,01= 2 bytes,10= 8 bytes,11= 4 bytes. - Local/Global Enable Flags (
L0-L3,G0-G3): Activate hardware monitoring for the current thread or system-wide.
Hardware watchpoints incur zero execution latency because condition evaluation is performed directly inside the CPU memory execution pipeline.
Linux Process Memory Layout and Kernel Interfaces
GDB interacts with the Linux kernel via /proc/<pid>/maps to inspect virtual memory permissions:
00400000-00401000 r--p 00000000 08:01 1048576 /home/user/target
00401000-00403000 r-xp 00001000 08:01 1048576 /home/user/target (.text)
00403000-00404000 r--p 00003000 08:01 1048576 /home/user/target (.rodata)
00404000-00405000 rw-p 00004000 08:01 1048576 /home/user/target (.data)
7ffff7d90000-7ffff7db2000 r-xp 00000000 08:01 2097152 /lib/x86_64-linux-gnu/libc.so.6
7ffffffde000-7ffffffff000 rw-p 00000000 00:00 0 [stack]System call interface operations:
PTRACE_ATTACH: Attaches to target PID, sendingSIGSTOP.PTRACE_PEEKTEXT/PTRACE_PEEKDATA: Reads memory words from target virtual addresses.PTRACE_POKETEXT/PTRACE_POKEDATA: Overwrites memory words in target virtual addresses (used for patching and software breakpoints).PTRACE_GETREGS/PTRACE_SETREGS: Reads or modifies register sets (user_regs_struct).
Practical GDB Investigation Session
The following GDB session demonstrates inspecting x86-64 register states, dissecting stack frames, configuring hardware watchpoints, dumping memory ranges, and overriding runtime branch decisions:
$ gdb ./target_binary
(gdb) set disassembly-flavor intel
(gdb) entrybreak
Downloading symbols for ./target_binary...
Breakpoint 1 at 0x401080
(gdb) run
Starting program: /home/user/target_binary
Hit Breakpoint 1, 0x0000000000401080 in _start ()
(gdb) disassemble 0x401080, +32
Dump of assembler code from 0x401080 to 0x4010a0:
0x0000000000401080 <+_start+0>: 31 rbp, rbp
0x0000000000401083 <+_start+3>: 49 89 d1 mov r11, rdx
0x0000000000401086 <+_start+6>: 48 8b 3c 24 mov rdi, QWORD PTR [rsp]
0x000000000040108a <+_start+10>: 48 8d 74 24 08 lea rsi, [rsp+0x8]
0x000000000040108f <+_start+15>: e8 2c 00 00 00 call 0x4010c0 <main>
(gdb) break *0x4010c0
(gdb) continue
Continuing.
Breakpoint 2, 0x00000000004010c0 in main ()
(gdb) info registers rax rbx rcx rdx rsi rdi rbp rsp rip
rax 0x0 0
rbx 0x0 0
rcx 0x7ffff7f9b000 140737353723904
rdx 0x7fffffffe398 140737488347032
rsi 0x7fffffffe388 140737488346984
rdi 0x1 1
rbp 0x0 0x0
rsp 0x7fffffffe378 0x7fffffffe378
rip 0x004010c0 0x4010c0 <main>
(gdb) x/8gx $rsp
0x7fffffffe378: 0x00007ffff7df0083 0x0000000000000001
0x7fffffffe388: 0x7fffffffe5c5 0x0000000000000000
0x7fffffffe398: 0x7fffffffe5eb 0x7fffffffe600
0x7fffffffe3a8: 0x7fffffffe61a 0x0000000000000000
(gdb) watch *(uint64_t*)0x7fffffffe388
Hardware watchpoint 3: *(uint64_t*)0x7fffffffe388
(gdb) continue
Continuing.
Hardware watchpoint 3: *(uint64_t*)0x7fffffffe388
Old value = 140737488346985
New value = 0
0x0000000000401115 in main ()
(gdb) dump memory payload.bin 0x404000 0x405000
(gdb) set $rax = 1
(gdb) set $rip = 0x401130
(gdb) stepi
0x0000000000401130 in main ()In this dynamic debugging trace, x/8gx $rsp displays 64-bit stack quadwords, watch sets a CPU hardware watchpoint on target stack coordinates, dump memory exports initialized heap/data regions to disk, set $rax = 1 overrides return values, and altering $rip redirects program execution past security check functions.
Reconstructing High-Level Control Flow Constructs
Compilers translate high-level constructs (loops, switch structures, struct field accesses, C++ classes) into distinct assembly patterns. Recognizing these patterns allows reverse engineers to reconstruct original source mechanics.
Loops and Branching Mechanics
Compilers generate structural idioms for do-while, while, for, and vectorized loops.
Do-While Loop Structure
A do-while loop executes the body first, followed by a conditional check at the bottom jumping back to the top if true:
// C Source Code
int i = 0;
do {
buffer[i] ^= 0x5A;
i++;
} while (i < 100);; Compiled Assembly (x86-64)
xor eax, eax ; i = 0
.loop_head:
movzx ecx, byte ptr [rdi + rax] ; load buffer[i]
xor ecx, 0x5A ; buffer[i] ^= 0x5A
mov byte ptr [rdi + rax], cl; store byte back
inc rax ; i++
cmp rax, 100 ; compare i to 100
jl .loop_head ; branch back if i < 100Standard While / For Loop Structure
Optimizing compilers transform standard while (i < 100) loops by executing loop inversion: placing an unconditional jump at the top pointing to the conditional evaluation at the bottom. This ensures the loop body executes only one conditional branch per iteration instead of two jumps.
; Compiled Assembly (x86-64 Loop Inversion Pattern)
xor eax, eax ; i = 0
jmp .loop_check ; branch to evaluation at bottom
.loop_body:
mov byte ptr [rdi + rax], 0 ; loop body work
inc rax ; i++
.loop_check:
cmp rax, 100 ; evaluate condition
jl .loop_body ; conditional branch up to loop bodyVectorized Loop Mechanics (AVX2 / SSE)
When processing large arrays, compilers emit SIMD (Single Instruction, Multiple Data) instructions to process multiple elements per clock cycle:
; Vectorized Loop Processing 32 Bytes per Iteration via AVX2
vmovdqa ymm0, ymmword ptr [rip + .xor_pattern] ; Load 256-bit byte vector (32x 0x5A)
xor eax, eax
.vector_loop:
vmovdqu ymm1, ymmword ptr [rdi + rax] ; Load 32 bytes from buffer[i]
vpxor ymm2, ymm1, ymm0 ; XOR 32 bytes simultaneously
vmovdqu ymmword ptr [rdi + rax], ymm2 ; Store 32 bytes back
add rax, 32 ; Advance index by 32
cmp rax, 1024
jl .vector_loopFollowing the vectorized loop, compilers generate a scalar cleanup loop to process remaining array bytes that do not fit 32-byte SIMD alignments.
Switch Statements: Decision Trees vs Jump Tables
Compilers select between binary search decision trees and Jump Tables based on switch case density.
When cases are sparse (for example, case 1, case 100, case 50000), compilers emit a binary search tree of conditional comparisons (cmp, je, jg). When cases are dense (for example, cases 0, 1, 2, 3, 4), compilers construct a Jump Table:
// C Source Code
switch (cmd) {
case 0: process_init(); break;
case 1: process_read(); break;
case 2: process_write(); break;
case 3: process_close(); break;
default: process_error(); break;
}; Compiled x86-64 Jump Table Pattern
cmp edi, 3 ; Check upper bound of dense cases
ja .default_case ; Branch to default if cmd > 3
lea rdx, [rip + .jump_table]; Load base address of jump table
mov eax, edi
movsxd rax, dword ptr [rdx + rax*4] ; Load 32-bit relative offset from table
add rax, rdx ; Compute absolute virtual address target
jmp rax ; Indirect jump to case handler
.jump_table:
.long .case_0 - .jump_table
.long .case_1 - .jump_table
.long .case_2 - .jump_table
.long .case_3 - .jump_tableReverse engineers reconstruct original switch statements by extracting the upper bound check (cmp edi, 3) and reading the array of 32-bit relative target offsets stored inside .rodata.
Structure Offsets, Array Indexing, and Memory Alignment
C/C++ compilers align struct fields according to primitive data sizes to maintain hardware bus performance:
- 1-byte
charaligns to 1-byte boundaries. - 2-byte
shortaligns to 2-byte boundaries. - 4-byte
intaligns to 4-byte boundaries. - 8-byte pointers (
void*) anduint64_talign to 8-byte boundaries.
struct DeviceConfig {
char enabled; /* Offset 0x00 (1 byte) */
/* 3 bytes padding inserted by compiler */
int timeout_ms; /* Offset 0x04 (4 bytes) */
char* device_name; /* Offset 0x08 (8 bytes) */
short port; /* Offset 0x10 (2 bytes) */
/* 6 bytes padding to align struct size to 8-byte multiple (0x18 total bytes) */
};In assembly, structural accesses map directly to byte offsets:
mov rdi, [rbx + 0x08] ; Accessing device_name field (offset 0x08)
mov eax, [rbx + 0x04] ; Accessing timeout_ms field (offset 0x04)
movzx ecx, byte ptr [rbx]; Accessing enabled field (offset 0x00)Array index calculations use scaled indexing modes:
- 1D Array Access (
array[i]): Address =Base + Index * ElementSize. Assembly:mov eax, [rdi + rsi*4](for 4-byteintelements). - 2D Array Access (
matrix[row][col]): Address =Base + (Row * NumCols + Col) * ElementSize. Assembly:imul rax, rsi, 16 ; Row * NumCols (16 columns) add rax, rdx ; + Col mov ecx, [rdi + rax*4] ; Load 4-byte element at matrix[row][col]
In Ghidra, creating a custom Data Type Manager struct matching these offsets transforms raw pointer arithmetic ([rbx + 0x08]) into named high-level symbol references (rbx->device_name).
C++ Object-Oriented Structures and Virtual Method Tables (Vtables)
C++ classes containing virtual functions place a hidden pointer called the vptr as their first member field (offset 0x00). The vptr points to a static array of function pointers in .rodata known as the Virtual Method Table (vtable).
Object Instance in RAM Vtable in Read-Only Memory (.rodata)
+-------------------------------+ +-------------------------------+
| vptr (8 bytes) |------> | &Device::read_telemetry() |
+-------------------------------+ +-------------------------------+
| int sensor_id (4 bytes) | | &Device::write_configuration()|
| char status (1 byte) | +-------------------------------+
+-------------------------------+ | &Device::reset_hardware() |
+-------------------------------+Executing a virtual function invocation (obj->read_telemetry()) produces an indirect call sequence:
mov rdi, QWORD PTR [rbp - 0x18] ; Load object instance address ('this' pointer)
mov rax, QWORD PTR [rdi] ; Step 1: Dereference vptr at offset 0x00
mov rax, QWORD PTR [rax] ; Step 2: Fetch 1st virtual function pointer (offset 0x00)
call rax ; Step 3: Indirect call to virtual methodIf invoking the second virtual function (obj->write_configuration()), the compiler accesses offset 0x08 within the vtable: mov rax, QWORD PTR [rax + 0x08].
In multiple inheritance classes, compilers insert non-zero vptr offset adjustments and thunk functions to re-align the this pointer before executing derived class methods:
; Thunk Function Adjusting 'this' Pointer for Second Base Class
.thunk_Device_reset:
sub rdi, 16 ; Adjust 'this' pointer back to primary object layout
jmp Device::reset_hardware ; Branch to virtual methodSymbols demangle C++ internal naming schemes (Itanium ABI on Linux, MSVC ABI on Windows) to reveal class hierarchy trees:
$ c++filt _ZN6Device14read_telemetryEv
Device::read_telemetry()Anti-Reversing and Obfuscation Mechanics
Malware payloads and protected commercial software incorporate anti-reversing defenses to hinder static decompilation and break dynamic debuggers.
Opaque Predicates and Mixed Boolean-Arithmetic (MBA)
An opaque predicate is a conditional control expression whose output result is known at compile time to the software author, but requires complex symbolic analysis for static disassemblers to resolve.
// Mathematical Opaque Predicate Example
// For any integer x, the expression (x * x + x) is ALWAYS an even number.
// Therefore, ((x * x + x) % 2) GUARANTEES an output evaluation of 0.
int x = rand();
if ((x * x + x) % 2 == 0) {
// Real Execution Branch (Executed 100% of the time)
decrypt_payload();
} else {
// Junk Dead-Code Branch (Never executed at runtime)
// Contains byte sequences designed to desynchronize disassemblers.
__asm__(" .byte 0xE8, 0xFF, 0xFF, 0xFF ");
}Multi-threaded opaque predicates leverage shared atomic variables or POSIX mutex state invariants:
pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
// In variant thread execution context:
pthread_mutex_lock(&lock);
// Value of mutex internal state is invariant while held
if (lock.__data.__lock > 100) {
// Dead-code branch inserted for decompiler confusion
invalid_instruction_sequence();
}
pthread_mutex_unlock(&lock);Obfuscators combine opaque predicates with Mixed Boolean-Arithmetic (MBA) expressions to hide simple assignment and identity operations. An MBA expression combines standard arithmetic operators (+, -, *) with bitwise logic operators (&, |, ^, ~).
For example, the simple identity x + y can be rewritten as an equivalent non-linear MBA expression:
$$x + y = (x \oplus y) + 2(x \wedge y)$$
Or the equivalence assignment x = y converted to:
$$x = (y ee
eg y) imes y + (y \wedge
eg y)$$
Static decompilers present complex mathematical expressions for simple variable assignments. Defeating MBA obfuscation requires utilizing SMT solvers (such as Z3) to simplify algebraic representations.
Control Flow Flattening (CFF)
Control Flow Flattening eliminates structured control hierarchies (if, while, for) and flattens basic blocks into a single horizontal plane underneath a central switch dispatcher statement inside an infinite loop.
Standard Control Flow Graph (CFG) Flattened Control Flow Graph (CFF)
+--------+ +---------------+
| BlockA | | State Init |
+--------+ +---------------+
/ \ |
v v v
+--------+ +--------+ +--------------+
| BlockB | | BlockC | | Dispatcher |<---+
+--------+ +--------+ | (Switch-Case)| |
\ / +--------------+ |
v v / | \ |
+--------+ v v v |
| BlockD | BlockA BlockB BlockC |
+--------+ \ | / |
+-----+---+-------+Every flattened basic block updates a state variable before jumping back to the main dispatcher loop:
// Flattened State Machine Representation
int state = 0x1A4B;
while (state != 0x9999) {
switch (state) {
case 0x1A4B:
execute_block_A();
state = condition ? 0x2C8D : 0x3F11;
break;
case 0x2C8D:
execute_block_B();
state = 0x88AE;
break;
case 0x3F11:
execute_block_C();
state = 0x88AE;
break;
case 0x88AE:
execute_block_D();
state = 0x9999; // Exit loop state
break;
}
}Unflattening control flow graphs requires utilizing symbolic execution frameworks (such as Angr or Triton) to track state variable modifications, map original basic block transitions, and patch dispatcher jump instructions back into direct conditional branch instructions.
Anti-Debugging Checks
Executables query kernel state, process structures, and timing counters to detect dynamic debugging environments.
Linux API and Procfs Checks
- Ptrace Self-Attach: Linux allows only one debugger process to attach to a target via
ptrace. Protected binaries execute a self-attach call to detect existing debuggers:
if (ptrace(PTRACE_TRACEME, 0, 1, 0) < 0) {
// Debugger process detected! Terminate execution.
exit(1);
}-
Procfs Status Inspection: Reading
/proc/self/statusto inspect theTracerPidfield. IfTracerPidis non-zero, GDB or an analytical process is attached. -
Procfs Wchan Inspection: Inspecting
/proc/self/wchanchecks if the main thread is waiting insideptrace_stoporsys_ptrace.
Windows PEB and API Checks
On Windows systems, the Process Environment Block (PEB) contains explicit debugging status flags accessed via thread environment blocks (gs:[0x60] on x64):
// Direct PEB BeingDebugged Flag Inspection
unsigned char* peb = (unsigned char*)__readgsqword(0x60);
unsigned char beingDebugged = peb[2]; // Offset 0x02: BeingDebugged byte
if (beingDebugged) {
ExitProcess(0);
}
// Inspecting NtGlobalFlag (Offset 0xBC on x64 PEB)
// Debugged processes set flags FLG_HEAP_ENABLE_TAIL_CHECK (0x10),
// FLG_HEAP_ENABLE_FREE_CHECK (0x20), and FLG_HEAP_VALIDATE_PARAMETERS (0x40).
DWORD ntGlobalFlag = *(DWORD*)(peb + 0xBC);
if (ntGlobalFlag & 0x70) {
ExitProcess(0);
}Windows API anti-debugging queries include:
IsDebuggerPresent(): Wraps the PEBBeingDebuggedcheck.CheckRemoteDebuggerPresent(): CallsNtQueryInformationProcesswithProcessDebugPort(value 0x07).NtQueryInformationProcess()withProcessDebugObjectHandle(value 0x1E): Checks for an active kernel debug object.
RDTSC Execution Timing Checks
Binaries measure elapsed CPU cycles between execution blocks using the rdtsc (Read Time-Stamp Counter) instruction. When single-stepping inside a debugger, execution latency expands from hundreds of cycles to millions:
rdtsc ; Read start cycle count into EDX:EAX
mov rbx, rax ; Store lower 32-bit timestamp
; --- Monitored Code Region ---
mov rcx, [rsp + 0x10]
xor rcx, 0xDEADBEEF
; -----------------------------
rdtsc ; Read second cycle count
sub rax, rbx ; Compute cycle delta
cmp rax, 0xFFFFF ; Threshold evaluation
ja .debugger_detected ; Branch to exit if delay indicates debuggingReverse engineers bypass timing checks by patching conditional jumps (ja to nop) or configuring GDB to trap and mock RDTSC instruction output.
Binary Packing and Unpacking Execution Flow
Executable packers (such as UPX, Themida, or custom malware crypters) compress or encrypt payload code sections (.text, .data) on disk. The outer file contains minimal import symbols and starts execution inside a specialized unpacking stub.
Packed Binary Execution Timeline:
[ Disk Binary File ] [ Memory Allocation Page ] [ Payload Execution ]
+--------------------+ +------------------------+ +-------------------+
| Unpack Stub Code | ------> | Memory Protection RWX | | Decrypted Payload |
+--------------------+ +------------------------+ +-------------------+
| Encrypted Payload | | ^
| Section Bytes | --(Decrypted)--------+ |
+--------------------+ (Jump to OEP)
|
Unpacker Allocates Memory (VirtualProtect) |
Decrypts Payload Bytes into Memory --------------+Unpacking follows a deterministic execution timeline:
- Packer Entry Point: Operating system loader transfers execution to the packer stub's entry point rather than the real application code.
- Memory Allocation: The stub invokes memory allocation APIs (
mprotecton Linux orVirtualProtecton Windows) to configure target memory pages as Read-Write-Execute (RWX). - Payload Decryption: The stub iterates through compressed/encrypted payload sections, decrypting original code bytes into the allocated virtual memory pages.
- Import Table Reconstruction: The stub manually resolves dynamic API pointers by parsing loaded DLL structures (
kernel32.dll,libc.so) using dynamic calls toLoadLibraryAandGetProcAddress(or custom hash resolution loops), populating the reconstructed Import Address Table. - Tail Jump to OEP: The stub executes an indirect jump (
jmp raxorpush oep; ret) transferring execution control to the Original Entry Point (OEP).
To unpack and reconstruct binaries using dynamic debuggers:
- Identify Memory Pages: Monitor memory protection calls (
VirtualProtect/mprotect) to locate allocatedRWXpayload pages. - Set OEP Breakpoint: Place a hardware execution breakpoint on the initial address of the decrypted
.textmemory page. - Catch Execution: Resume process execution; the debugger halts at the OEP when the packer stub executes its tail jump.
- Dump Process Memory: Use GDB memory dump commands (
dump memory payload.bin <start> <end>) or Scylla to export unencrypted memory sections to disk. - Fix Headers and IAT: Reconstruct PE/ELF section headers and fix Import Address Table RVAs to yield a functional binary executable ready for static decompilation in Ghidra.