How Fine-Tuning and Alignment Modify LLM Weights

Autoregressive language models undergo a distinct multi-stage post-training pipeline before deployment in production environments. Pre-training yields a base model whose weight parameters $\Theta_0$ parameterize a probability distribution over token sequences by minimizing causal language modeling loss across trillions of unstructured tokens. Base models generate text by predicting the most probable continuation of a prompt, but they do not inherently follow instructions, adhere to safety boundaries, or maintain consistent conversational roles.

Transforming a base model into an instruction-following assistant requires post-training: Supervised Fine-Tuning (SFT), Parameter-Efficient Fine-Tuning (PEFT) such as Low-Rank Adaptation (LoRA), and preference alignment algorithms like Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO). Each phase modifies parameter matrices across attention blocks and feed-forward networks under distinct mathematical constraints.

Understanding post-training requires analyzing parameter space transformations, tensor gradient mechanics, memory allocations, and alignment loss dynamics.

Mathematical Principles of Autoregressive Weight Parameterization

An autoregressive language model parameterizes the conditional probability distribution of text sequences. Given a vocabulary $\mathcal{V}$ of size $V = |\mathcal{V}|$ and a sequence of tokens $x = (x_1, x_2, \dots, x_N)$, the joint probability $P_{\Theta}(x)$ is factorized via the chain rule of probability:

$$P_{\Theta}(x) = \prod_{i=1}^N P_{\Theta}(x_i \mid x_1, x_2, \dots, x_{i-1}) = \prod_{i=1}^N P_{\Theta}(x_i \mid x_{<i})$$

Pre-training optimizes the initial parameter set $\Theta_0 \in \mathbb{R}^d$ across an uncurated dataset $\mathcal{D}_{\text{pre}}$ by minimizing the empirical negative log-likelihood risk:

$$\mathcal{L}{\text{pre}}(\Theta) = -\frac{1}{|\mathcal{D}{\text{pre}}|} \sum_{x \in \mathcal{D}{\text{pre}}} \sum{i=1}^{|x|} \log P_{\Theta}(x_i \mid x_{<i})$$

Transformer Layer Tensor Breakdown

Modern decoder-only architectures (such as Llama, Mistral, and Qwen) consist of $L$ stacked transformer block layers. Each layer $l \in {1, \dots, L}$ contains a Multi-Head Self-Attention (MHSA) module and a Feed-Forward Network (FFN) or SwiGLU Gated Linear Unit module, bounded by normalization steps (such as RMSNorm).

Transformer Layer Block Tensor Dimensions:
 
Input Activation Tensor: X_l in R^(b x s x d_model)
  │
  ├──► RMSNorm(X_l) ──► Multi-Head Self-Attention (MHSA)
  │                      ├── Query: W_q in R^(d_model x d_model)
  │                      ├── Key:   W_k in R^(d_model x d_model)
  │                      ├── Value: W_v in R^(d_model x d_model)
  │                      └── Out:   W_o in R^(d_model x d_model)
  │                      │
  │                      ▼
  │                   A_l = Attn(Q, K, V) * W_o
  │                      │
  ├── (Residual Connection: X_l + A_l) ──┐
  │                                      ▼
  └──► RMSNorm(X_l + A_l) ──► SwiGLU Gated Feed-Forward Network (FFN)
                              ├── Gate Projection: W_gate in R^(d_ffn x d_model)
                              ├── Up Projection:   W_up   in R^(d_ffn x d_model)
                              └── Down Projection: W_down in R^(d_model x d_ffn)
                              │
                              ▼
                           F_l = (SiLU(X * W_gate) (X * W_up)) * W_down
                              │
  ┌───────────────────────────┘
  ▼
Output Activation Tensor: X_(l+1) = X_l + A_l + F_l

For a hidden dimension $d_{\text{model}}$ and an intermediate FFN dimension $d_{\text{ffn}}$ (typically set to $d_{\text{ffn}} = \frac{8}{3} d_{\text{model}}$ in SwiGLU setups), the trainable weight tensor shapes per layer are defined as follows:

1. Self-Attention Projection Matrices:

   • Query Projection: $W_q \in \mathbb{R}^{d_{\text{model}} \times d_{\text{model}}}$
   • Key Projection: $W_k \in \mathbb{R}^{d_{\text{model}} \times d_{\text{model}}}$
   • Value Projection: $W_v \in \mathbb{R}^{d_{\text{model}} \times d_{\text{model}}}$
   • Output Projection: $W_o \in \mathbb{R}^{d_{\text{model}} \times d_{\text{model}}}$

2. SwiGLU FFN Projection Matrices:

   • Gate Projection: $W_{\text{gate}} \in \mathbb{R}^{d_{\text{ffn}} \times d_{\text{model}}}$
   • Up Projection: $W_{\text{up}} \in \mathbb{R}^{d_{\text{ffn}} \times d_{\text{model}}}$
   • Down Projection: $W_{\text{down}} \in \mathbb{R}^{d_{\text{model}} \times d_{\text{ffn}}}$

The total parameter count for a transformer layer $l$ is dominated by these seven dense linear projection operators:

$$N_{\text{layer}} = 4 \cdot d_{\text{model}}^2 + 3 \cdot d_{\text{model}} \cdot d_{\text{ffn}}$$

For a 7B parameter model with $L = 32$, $d_{\text{model}} = 4096$, and $d_{\text{ffn}} = 11008$, each individual layer contains approximately $203.4 \times 10^6$ parameters.

Gradient Propagation Mechanics

During the backward pass of training, partial derivatives of the scalar loss $\mathcal{L}$ flow backward through activations to update weight matrices. For any linear projection $Y = X W^T$ where $X \in \mathbb{R}^{B \times S \times d_{\text{in}}}$, $W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$, and output gradient $\delta = \frac{\partial \mathcal{L}}{\partial Y} \in \mathbb{R}^{B \times S \times d_{\text{out}}}$, the gradient with respect to weight parameter tensor $W$ is calculated as:

$$\frac{\partial \mathcal{L}}{\partial W} = \sum_{b=1}^B \sum_{s=1}^S \delta_{b, s, :} \otimes X_{b, s, :}^T \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$$

Un-tuned base models parameterize raw sequence co-occurrence statistics. When prompted with an instruction, base models often output plausible multi-turn continuations or generate alternative prompt completions rather than executing the requested command. Supervised fine-tuning reshapes the weight space to enforce target behavior formats.

Supervised Fine-Tuning (SFT) Dynamics

Supervised Fine-Tuning adapts a pre-trained base model to target specific task formats, system roles, and instruction-response patterns. SFT processes formatted datasets consisting of input prompt sequences $x = (x_1, x_2, \dots, x_m)$ and target response sequences $y = (y_1, y_2, \dots, y_n)$.

Full Sequence Format:
[BOS] <system_prompt> [EOS] <user_prompt> [EOS] <assistant_response> [EOS]
|<-------------- Prompt Tokens (Masked) ------------->|<-- Target Tokens -->|

Masked Cross-Entropy Loss Formulation

Unlike pre-training, where causal language modeling loss is computed over every token in the sequence, standard SFT masks prompt tokens $x$ so that loss gradients are calculated exclusively over target response tokens $y$.

Given a sequence of length $N = m + n$, the autoregressive probability of the response sequence $y$ conditioned on prompt $x$ is:

$$P_\theta(y \mid x) = \prod_{t=1}^n P_\theta(y_t \mid x, y_{<t})$$

The SFT loss $\mathcal{L}_{\text{SFT}}(\theta)$ is the average negative log-likelihood of the response tokens:

$$\mathcal{L}{\text{SFT}}(\theta) = -\frac{1}{n} \sum{t=1}^n \log P_\theta(y_t \mid x, y_1, \dots, y_{t-1})$$

In terms of logits $z_t \in \mathbb{R}^V$ produced by the final transformer linear projection layer (where $V$ is the vocabulary size), the probability distribution $P_\theta(y_t \mid \dots)$ is computed using the softmax function:

$$P_\theta(y_t = k \mid x, y_{<t}) = \frac{\exp(z_{t, k})}{\sum_{j=1}^V \exp(z_{t, j})}$$

Gradient Derivation for Response Tokens

To derive the exact parameter update signal, consider the derivative of the cross-entropy loss $\mathcal{L}_{\text{SFT}}$ at token step $t$ with respect to logit index $k$:

$$\frac{\partial \mathcal{L}t}{\partial z{t, k}} = \frac{\partial}{\partial z_{t, k}} \left( -\log \frac{\exp(z_{t, y_t})}{\sum_{j=1}^V \exp(z_{t, j})} \right) = -\frac{\partial z_{t, y_t}}{\partial z_{t, k}} + \frac{\partial}{\partial z_{t, k}} \log \sum_{j=1}^V \exp(z_{t, j})$$

Applying exponent derivative rules gives:

$$\frac{\partial}{\partial z_{t, k}} \log \sum_{j=1}^V \exp(z_{t, j}) = \frac{\exp(z_{t, k})}{\sum_{j=1}^V \exp(z_{t, j})} = P_\theta(y_t = k \mid x, y_{<t})$$

Evaluating the target label indicator derivative gives:

$$\frac{\partial z_{t, y_t}}{\partial z_{t, k}} = \mathbb{I}(y_t = k)$$

Thus, the gradient of the masked SFT loss with respect to logit $z_{t, k}$ resolves to:

$$\frac{\partial \mathcal{L}{\text{SFT}}}{\partial z{t, k}} = P_\theta(y_t = k \mid x, y_{<t}) - \mathbb{I}(y_t = k)$$

where $\mathbb{I}(\cdot)$ is the indicator function. The error vector $e_t = P_\theta(\cdot \mid x, y_{<t}) - \mathbf{y}t \in \mathbb{R}^V$ acts as the incoming gradient signal to the unembedding projection layer $W{\text{head}} \in \mathbb{R}^{V \times d_{\text{model}}}$. For prompt tokens ($i \le m$), $e_i$ is set to $\mathbf{0}$, blocking gradient backpropagation into upstream transformer layers for input prompt positions.

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader
 
def compute_masked_sft_loss(
    logits: torch.Tensor,       # Shape: [batch_size, seq_len, vocab_size]
    labels: torch.Tensor,       # Shape: [batch_size, seq_len]
    ignore_index: int = -100
) -> torch.Tensor:
    """
    Computes cross-entropy loss over target response tokens while ignoring prompt tokens.
    Prompt tokens in the labels tensor are masked with ignore_index (-100).
    """
    # Shift logits and labels for causal autoregressive prediction
    shift_logits = logits[:, :-1, :].contiguous()
    shift_labels = labels[:, 1:].contiguous()
    
    # Flatten tensors for cross-entropy calculation
    vocab_size = shift_logits.size(-1)
    flat_logits = shift_logits.view(-1, vocab_size)
    flat_labels = shift_labels.view(-1)
    
    # Calculate masked loss
    loss = F.cross_entropy(
        flat_logits, 
        flat_labels, 
        ignore_index=ignore_index, 
        reduction='mean'
    )
    return loss
 
class MaskedSFTTrainer:
    def __init__(
        self, 
        model: nn.Module, 
        optimizer: torch.optim.Optimizer, 
        ignore_index: int = -100,
        grad_accum_steps: int = 1
    ):
        self.model = model
        self.optimizer = optimizer
        self.ignore_index = ignore_index
        self.grad_accum_steps = grad_accum_steps
 
    def train_step(self, input_ids: torch.Tensor, labels: torch.Tensor, step: int) -> float:
        self.model.train()
        outputs = self.model(input_ids=input_ids)
        logits = outputs.logits if hasattr(outputs, 'logits') else outputs
        
        loss = compute_masked_sft_loss(logits, labels, self.ignore_index)
        scaled_loss = loss / self.grad_accum_steps
        scaled_loss.backward()
        
        if (step + 1) % self.grad_accum_steps == 0:
            torch.nn.utils.clip_grad_norm_(self.model.parameters(), max_norm=1.0)
            self.optimizer.step()
            self.optimizer.zero_grad()
            
        return loss.item()

Parameter Weight Updates and Optimization Footprint

During full-parameter SFT, every trainable parameter $W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$ in the neural network is updated according to the gradient of the loss function using optimizers like AdamW:

$$m_t = \beta_1 m_{t-1} + (1 - \beta_1) \nabla_W \mathcal{L}_{\text{SFT}}(W^{(t)})$$

$$v_t = \beta_2 v_{t-1} + (1 - \beta_2) \left( \nabla_W \mathcal{L}_{\text{SFT}}(W^{(t)}) \right)^2$$

$$\hat{m}_t = \frac{m_t}{1 - \beta_1^t}, \quad \hat{v}_t = \frac{v_t}{1 - \beta_2^t}$$

$$W^{(t+1)} = W^{(t)} - \eta \left( \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} + \lambda W^{(t)} \right)$$

Executing full-parameter SFT using AdamW requires substantial GPU VRAM. Memory consumption per parameter splits into four distinct allocation categories:

1. Parameters ($\theta$): Stored in 16-bit precision (FP16 or BF16), consuming 2 bytes per parameter.
2. Gradients ($\nabla_\theta \mathcal{L}$): Stored in 16-bit precision (FP16 or BF16), consuming 2 bytes per parameter.
3. Master Weights: Stored in 32-bit single-precision (FP32) to prevent numerical underflow during weight update additions, consuming 4 bytes per parameter.
4. AdamW Optimizer States:
   • First moment vector $m_t$ (FP32): 4 bytes per parameter.
   • Second moment vector $v_t$ (FP32): 4 bytes per parameter.

Total static memory for model states equals:

$$\text{Memory}_{\text{states}} = 2 + 2 + 4 + 4 + 4 = 16 \text{ bytes per parameter}$$

For a model with 7 billion parameters ($7 \times 10^9$), static state memory requirements reach:

$$\text{Memory}_{7\text{B}} = 7 \times 10^9 \times 16 \text{ bytes} = 112 \text{ GB VRAM}$$

This static footprint excludes activation memory generated during forward passes, sequence KV caches, and workspace buffers. Activation memory grows linearly with batch size, sequence length $N$, and hidden layer dimensionality $d_{\text{model}}$.

To execute full SFT on large models across clusters, distributed strategies like Fully Sharded Data Parallel (FSDP) or DeepSpeed ZeRO are required:

• ZeRO-Stage 1: Shares optimizer states across $N_{\text{GPUs}}$ nodes (reducing memory to $4 + 12/N_{\text{GPUs}}$ bytes/param).
• ZeRO-Stage 2: Shares optimizer states and gradients across nodes (reducing memory to $2 + 14/N_{\text{GPUs}}$ bytes/param).
• ZeRO-Stage 3 / FSDP Full Shard: Shares optimizer states, gradients, and model parameters across nodes (reducing memory to $16 / N_{\text{GPUs}}$ bytes/param).

Parameter-Efficient Fine-Tuning with LoRA

Full-parameter fine-tuning becomes computationally expensive when scaling models beyond tens of billions of parameters. Low-Rank Adaptation (LoRA) freezes the pre-trained model weights $W_0 \in \mathbb{R}^{d \times k}$ and injects trainable rank decomposition matrices into each transformer layer.

Standard Linear Layer Forward Pass:
h = W_0 * x
 
LoRA Modified Forward Pass:
h = W_0 * x + (alpha / r) * (B * A) * x
    |___Frozen___|            |__Trainable__|

Intrinsic Rank Hypothesis and Mathematical Decomposition

The foundational premise of LoRA is the intrinsic rank hypothesis (Aghajanyan et al., Hu et al.), which posits that weight updates $\Delta W$ during task adaptation occupy a subspace with a significantly lower intrinsic rank $r$ than the full dimension of the weight matrix $\min(d, k)$.

For a dense linear weight matrix $W_0 \in \mathbb{R}^{d \times k}$, LoRA parameterizes the weight update $\Delta W$ by decomposing it into two low-rank matrices $A \in \mathbb{R}^{r \times k}$ and $B \in \mathbb{R}^{d \times r}$, where rank $r \ll \min(d, k)$:

$$W = W_0 + \Delta W = W_0 + \frac{\alpha}{r} B A$$

The hyperparameter $r$ controls matrix inner dimension rank, while $\alpha$ serves as a constant scaling factor. The forward pass computation for input vector $x \in \mathbb{R}^k$ becomes:

$$h = W_0 x + \Delta W x = W_0 x + \frac{\alpha}{r} B (A x)$$

FLOP and Computational Complexity Derivation

Evaluating $B(Ax)$ instead of $(BA)x$ changes computational complexity. Computing $BA$ explicitly requires $O(d \cdot k \cdot r)$ operations and generates a dense matrix of size $d \times k$. Multiplying $x$ sequentially by $A$ and then $B$ requires computing:

1. $y_A = A x \in \mathbb{R}^r$: requires $2 \cdot r \cdot k$ floating-point operations (FLOPs).
2. $y_B = B y_A \in \mathbb{R}^d$: requires $2 \cdot d \cdot r$ FLOPs.

Total FLOPs per token for the LoRA adapter path equals:

$$\text{FLOPs}_{\text{LoRA}} = 2 \cdot r \cdot k + 2 \cdot d \cdot r = 2 r (k + d)$$

Comparing this to a full matrix multiplication $W_0 x$, which takes $\text{FLOPs}_{\text{Dense}} = 2 \cdot d \cdot k$:

$$\frac{\text{FLOPs}{\text{LoRA}}}{\text{FLOPs}{\text{Dense}}} = \frac{2 r (k + d)}{2 d k} = \frac{r(k + d)}{d k} = r \left( \frac{1}{d} + \frac{1}{k} \right)$$

For $d = k = 4096$ and $r = 16$:

$$\frac{\text{FLOPs}{\text{LoRA}}}{\text{FLOPs}{\text{Dense}}} = 16 \left( \frac{1}{4096} + \frac{1}{4096} \right) = 16 \cdot \frac{2}{4096} = \frac{32}{4096} = 0.0078125$$

The low-rank adapter path reduces forward FLOP overhead for weight updates to under 0.8% of the dense projection cost.

Initialization Dynamics and Scaling Invariants

To ensure that $\Delta W = 0$ at the start of fine-tuning ($t = 0$), matrix weights are initialized asymmetrically:

• Matrix $A$ is drawn from a Gaussian distribution: $A \sim \mathcal{N}\left(0, \sigma^2\right)$ where $\sigma^2 = \frac{1}{r}$ (or Kaiming uniform distribution).
• Matrix $B$ is initialized entirely to zeros: $B = 0$.

Because $B = 0$ at step zero:

$$\Delta W(0) = \frac{\alpha}{r} B(0) A(0) = \frac{\alpha}{r} (0) A(0) = 0$$

This guarantees $h = W_0 x + 0 = W_0 x$ prior to training, preserving base model behavior before gradient updates occur.

The scaling ratio $\frac{\alpha}{r}$ stabilizes hyperparameter tuning. When adjusting rank $r$, scaling $\alpha$ proportionally maintains gradient magnitude scale without requiring re-tuning of learning rate $\eta$. Specifically, if rank $r$ is doubled from 16 to 32, setting $\alpha = 32$ keeps $\frac{\alpha}{r} = 1.0$, holding adapter gradient scales invariant.

import math
import torch
import torch.nn as nn
 
class LoRALinear(nn.Module):
    def __init__(
        self, 
        in_features: int, 
        out_features: int, 
        r: int = 8, 
        alpha: float = 16.0,
        dropout: float = 0.05
    ):
        super().__init__()
        # Base weight matrix is frozen
        self.linear = nn.Linear(in_features, out_features, bias=False)
        self.linear.weight.requires_grad = False
        
        self.r = r
        self.alpha = alpha
        self.scaling = alpha / r
        
        if r > 0:
            # Low-rank decomposition matrices
            self.lora_A = nn.Parameter(torch.zeros(r, in_features))
            self.lora_B = nn.Parameter(torch.zeros(out_features, r))
            self.dropout = nn.Dropout(p=dropout) if dropout > 0.0 else nn.Identity()
            
            # Initialization: A ~ Kaiming Uniform, B = 0
            nn.init.kaiming_uniform_(self.lora_A, a=math.sqrt(5))
            nn.init.zeros_(self.lora_B)
            
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # Base forward pass
        result = self.linear(x)
        
        if self.r > 0:
            # Low-rank forward computation: B @ (A @ dropout(x))
            dropout_x = self.dropout(x)
            lora_out = F.linear(dropout_x, self.lora_A)
            lora_out = F.linear(lora_out, self.lora_B)
            result = result + lora_out * self.scaling
            
        return result
 
    def merge_weights(self) -> None:
        """
        Merges LoRA adapter weights directly into W_0 for zero-latency deployment.
        W_merged = W_0 + (alpha / r) * (B @ A)
        """
        if self.r > 0:
            delta_w = (self.lora_B @ self.lora_A) * self.scaling
            self.linear.weight.data += delta_w
            self.r = 0 # Disable adapter path post-merge

Targeted Weight Matrices and VRAM Comparisons

In Transformer architectures, LoRA adapters can target different sub-layer weight projections:

• Attention Only ($W_q, W_v$): Minimal parameter insertion ($0.05%$ of base parameters). Effective for basic instruction formatting.
• Attention All ($W_q, W_k, W_v, W_o$): Covers attention representation transformations ($0.1%$ of base parameters).
• All Linear Layers ($W_q, W_k, W_v, W_o, W_{\text{gate}}, W_{\text{up}}, W_{\text{down}}$): Modifies both attention and feed-forward feature transformations ($0.25%$ to $0.5%$ of base parameters).

Targeting all linear layers with lower rank ($r=8$ or $r=16$) yields higher parameter expressivity and faster task convergence than targeting attention projections alone with high rank ($r=64$).

QLoRA: 4-Bit NormalFloat Quantization and Double Quantization

Quantized Low-Rank Adaptation (QLoRA) reduces memory usage by quantizing base model weight matrix $W_0$ into a 4-bit data format called NormalFloat4 (NF4), while keeping LoRA adapter parameters $A$ and $B$ in 16-bit floating point precision.

QLoRA Block Quantization & Forward Scheme:
 
Base Model Weights W_0 (FP16/BF16)
  │
  ▼  [Quantize to 4-bit NF4 Quantiles]
W_NF4 (4-bit representation)  +  Double Quantized Scales c_1, c_2
  │
  ▼  [On-the-Fly Dequantization during Forward Pass]
W_dequant = Dequantize(W_NF4, c_1, c_2) in BF16
  │
  ├──► Forward: h = (W_dequant * x) + (alpha / r) * (B * A * x)
  │
  └──► Backward: Gradients computed ONLY for trainable matrices A and B (BF16)

NormalFloat4 (NF4) Quantile Construction

NF4 constructs an information-theoretically optimal quantile distribution for weights normally distributed as $W_0 \sim \mathcal{N}(0, \sigma^2)$. Standard 4-bit integers (INT4) allocate uniform step sizes between minimum and maximum values, leading to poor resolution near zero where normal distributions concentrate mass.

NF4 sets 16 quantized bin values $q_i$ ($i=0, \dots, 15$) such that each bin receives an equal expected number of parameters. Given the theoretical Gaussian cumulative distribution function $q_i = Q_X(p_i)$:

$$q = [-1.0, -0.6961, -0.5251, -0.3949, -0.2844, -0.1848, -0.0910, 0.0, 0.0796, 0.1609, 0.2471, 0.3393, 0.4407, 0.5581, 0.6999, 1.0]$$

These non-uniform bin boundaries minimize information-theoretic quantization error for Gaussian weight tensors.

Double Quantization (DQ) Mechanics

Quantization computes scale constants $c_1$ across local parameter blocks (typically block size $B_{\text{block}} = 64$). Quantization scale constant $c_1^{\text{FP32}}$ consumes 32 bits per 64 parameters, adding $\frac{32}{64} = 0.5$ bits per parameter to memory overhead.

Double Quantization treats primary quantization scales $c_1^{\text{FP32}}$ as inputs to a secondary 8-bit FP8 quantization step with block size $B_{\text{scale}} = 256$:

1. Primary scale constants $c_1$ are quantized into 8-bit FP8 numbers ($c_1^{\text{FP8}}$), consuming $8$ bits per scale constant.
2. Secondary scale constants $c_2^{\text{FP32}}$ are stored at 32-bit precision per 256 primary scale values.

Calculating memory footprint per parameter under Double Quantization:

$$\text{Bits}{\text{DQ}} = \frac{32}{B{\text{block}} \cdot B_{\text{scale}}} + \frac{8}{B_{\text{block}}} = \frac{32}{64 \cdot 256} + \frac{8}{64} = \frac{32}{16384} + 0.125 = 0.00195 + 0.125 = 0.12715 \text{ bits/param}$$

Double Quantization saves $0.5 - 0.12715 = 0.37285$ bits per parameter. For a 70B parameter model, DQ frees approximately 3.26 GB of VRAM.

Paged Optimizers and CUDA Unified Memory

To handle memory spikes during backward pass gradient execution, QLoRA uses Paged Optimizers. Built on CUDA Unified Memory, Paged Optimizers allocate page-locked host memory (CPU RAM) as physical overflow for optimizer states.

During backward passes, parameter update allocations are paged out to CPU system memory when GPU VRAM utilization approaches capacity. Once gradient execution moves to preceding layers, optimizer pages are paged back asynchronously over PCIe lanes. This prevents out-of-memory errors during long context window processing.

import torch
import torch.nn as nn
 
class QLoRADequantizer(torch.autograd.Function):
    @staticmethod
    def forward(
        ctx, 
        qweight_nf4: torch.Tensor, # Shape: [out_features, in_features // 2] (packed uint8)
        absmax_c1: torch.Tensor,   # Shape: [num_blocks] (FP8)
        absmax_c2: torch.Tensor,   # Shape: [num_scale_blocks] (FP32)
        shape: torch.Size,
        nf4_code: torch.Tensor     # Shape: [16] (FP32 lookup table)
    ) -> torch.Tensor:
        # Step 1: Dequantize primary scales c1 = c1_fp8 * c2_fp32
        # Step 2: Unpack 4-bit indices from uint8 bytes
        # Step 3: Map indices to NF4 codebook values and scale by c1
        
        # Dequantized output tensor in BF16
        dequantized_weight = torch.empty(shape, dtype=torch.bfloat16, device=qweight_nf4.device)
        return dequantized_weight
 
    @staticmethod
    def backward(ctx, grad_output):
        # Base weight matrix W_0 is frozen; no gradients backpropagate to NF4 tensors
        return None, None, None, None, None

Alignment Algorithms: RLHF vs DPO

Following SFT, models undergo alignment to optimize output safety, truthfulness, and style. Alignment minimizes harmful outputs and aligns model responses with preferred human evaluation outcomes.

RLHF Pipeline:
[SFT Model] ---> [Train Reward Model] ---> [PPO Policy Optimization (Actor-Critic)]
                                                 |
                                         (Requires 4 Models in VRAM)
 
DPO Pipeline:
[SFT Model] ---> [Direct Preference Optimization Loss]
                        |
               (Requires 2 Models in VRAM)

Reinforcement Learning from Human Feedback (RLHF)

The classical RLHF pipeline consists of three sequential phases:

Phase 1: Supervised Fine-Tuning

A base model is fine-tuned on target dataset pairs to produce baseline policy $\pi^{\text{SFT}}$.

Phase 2: Reward Model Training

A dataset of prompt $x$ and paired completions $(y_w, y_l)$ is collected, where human evaluators mark $y_w$ as preferred and $y_l$ as dispreferred. A reward model $r_\psi(x, y) \in \mathbb{R}$ is trained under the Bradley-Terry preference probability framework:

$$P(y_w \succ y_l \mid x) = \sigma\left(r_\psi(x, y_w) - r_\psi(x, y_l)\right)$$

The negative log-likelihood loss for reward model parameters $\psi$ is:

$$\mathcal{L}{\text{RM}}(\psi) = -\mathbb{E}{(x, y_w, y_l) \sim D} \left[ \log \sigma \left( r_\psi(x, y_w) - r_\psi(x, y_l) \right) \right]$$

Phase 3: PPO Policy Optimization

The aligned policy $\pi_\theta$ is optimized against reward model output using Proximal Policy Optimization (PPO). The objective function contains a Kullback-Leibler (KL) divergence penalty that prevents policy $\pi_\theta$ from drifting too far from baseline policy $\pi_{\text{SFT}}$:

$$\max_\theta \mathbb{E}{(x, y) \sim \pi\theta} \left[ r_\psi(x, y) - \beta D_{\text{KL}}\left(\pi_\theta(y \mid x) \parallel \pi_{\text{SFT}}(y \mid x)\right) \right]$$

The token-level KL divergence penalty is defined as:

$$D_{\text{KL}}\left(\pi_\theta(y \mid x) \parallel \pi_{\text{SFT}}(y \mid x)\right) = \sum_{t=1}^n \log \frac{\pi_\theta(y_t \mid x, y_{<t})}{\pi_{\text{SFT}}(y_t \mid x, y_{<t})}$$

Running PPO in production requires storing four large language models concurrently in GPU VRAM: the active Actor policy $\pi_\theta$, the Critic value model $V_\phi$, the reference policy $\pi_{\text{SFT}}$, and the Reward model $r_\psi$. This setup creates significant system infrastructure complexity.

PPO VRAM Model Allocation Architecture:
 
GPU Memory Space
├── Actor Model pi_theta (Trainable)    [BF16 Weights + Gradients + Optimizer States]
├── Critic Model V_phi   (Trainable)    [BF16 Weights + Gradients + Optimizer States]
├── Reference Policy pi_ref (Frozen)     [BF16 Weights Only]
└── Reward Model r_psi      (Frozen)     [BF16 Weights Only]
 
Total Footprint for 70B Models > 1.4 Terabytes VRAM across GPU Nodes

Direct Preference Optimization (DPO)

Direct Preference Optimization (DPO) eliminates the need to train explicit reward models or sample from policy loops during training by algebraically reparametrizing the reward function.

Analytical Derivation of the Implicit Reward Function

The constrained optimization objective for policy alignment is:

$$\max_{\pi} \mathbb{E}{x \sim D, y \sim \pi(y \mid x)} \left[ r(x, y) \right] - \beta D{\text{KL}}\left( \pi(y \mid x) \parallel \pi_{\text{ref}}(y \mid x) \right)$$

Expanding the KL divergence summation yields:

$$\max_{\pi} \sum_{y} \pi(y \mid x) r(x, y) - \beta \sum_{y} \pi(y \mid x) \log \frac{\pi(y \mid x)}{\pi_{\text{ref}}(y \mid x)}$$

$$= \max_{\pi} \sum_{y} \pi(y \mid x) \left[ r(x, y) - \beta \log \frac{\pi(y \mid x)}{\pi_{\text{ref}}(y \mid x)} \right]$$

$$= \max_{\pi} \beta \sum_{y} \pi(y \mid x) \left[ \frac{r(x, y)}{\beta} - \log \frac{\pi(y \mid x)}{\pi_{\text{ref}}(y \mid x)} \right]$$

$$= \max_{\pi} \beta \sum_{y} \pi(y \mid x) \log \left[ \frac{\pi_{\text{ref}}(y \mid x) \exp\left(\frac{1}{\beta} r(x, y)\right)}{\pi(y \mid x)} \right]$$

Define an unnormalized partition function $Z(x)$:

$$Z(x) = \sum_{y} \pi_{\text{ref}}(y \mid x) \exp\left( \frac{1}{\beta} r(x, y) \right)$$

Now define a valid normalized probability distribution $\pi^*$:

$$\pi^*(y \mid x) = \frac{1}{Z(x)} \pi_{\text{ref}}(y \mid x) \exp\left( \frac{1}{\beta} r(x, y) \right)$$

Substituting $\pi^*(y \mid x)$ back into the optimization expression transforms the formulation into a single KL divergence minimization problem:

$$\max_{\pi} \beta \sum_{y} \pi(y \mid x) \log \left[ \frac{\pi^*(y \mid x) Z(x)}{\pi(y \mid x)} \right]$$

$$= \max_{\pi} \beta \left( \sum_{y} \pi(y \mid x) \log \frac{\pi^*(y \mid x)}{\pi(y \mid x)} + \sum_{y} \pi(y \mid x) \log Z(x) \right)$$

$$= \min_{\pi} \beta D_{\text{KL}}\left( \pi(y \mid x) \parallel \pi^*(y \mid x) \right) + \beta \log Z(x)$$

Because $D_{\text{KL}} \ge 0$, the global minimum occurs when $\pi(y \mid x) = \pi^*(y \mid x)$. Rearranging terms yields the exact expression for optimal implicit reward:

$$\pi^*(y \mid x) = \frac{1}{Z(x)} \pi_{\text{ref}}(y \mid x) \exp\left(\frac{1}{\beta} r(x, y)\right)$$

$$\frac{\pi^*(y \mid x)}{\pi_{\text{ref}}(y \mid x)} = \frac{1}{Z(x)} \exp\left(\frac{1}{\beta} r(x, y)\right)$$

Taking the natural logarithm of both sides:

$$\log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)} = \frac{1}{\beta} r(x, y) - \log Z(x)$$

Rearranging for explicit ground-truth reward $r(x, y)$:

$$r(x, y) = \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x)$$

The DPO Loss Function

Substituting this implicit reward expression into the Bradley-Terry preference probability formula cancels out the partition function term $Z(x)$:

$$r(x, y_w) - r(x, y_l) = \left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} + \beta \log Z(x) \right) - \left( \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} + \beta \log Z(x) \right)$$

$$r(x, y_w) - r(x, y_l) = \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}$$

The final Direct Preference Optimization loss function $\mathcal{L}{\text{DPO}}(\theta; \pi{\text{ref}})$ becomes:

$$\mathcal{L}{\text{DPO}}(\theta; \pi{\text{ref}}) = -\mathbb{E}{(x, y_w, y_l) \sim D} \left[ \log \sigma \left( \beta \log \frac{\pi\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right]$$

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple
 
def compute_dpo_loss(
    policy_chosen_logps: torch.Tensor,     # Shape: [batch_size]
    policy_rejected_logps: torch.Tensor,   # Shape: [batch_size]
    reference_chosen_logps: torch.Tensor,  # Shape: [batch_size]
    reference_rejected_logps: torch.Tensor, # Shape: [batch_size]
    beta: float = 0.1
) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
    """
    Computes Direct Preference Optimization (DPO) loss.
    """
    # Log-ratios of policy probabilities relative to reference policy
    policy_logratios = policy_chosen_logps - policy_rejected_logps
    reference_logratios = reference_chosen_logps - reference_rejected_logps
    
    # Implicit reward difference: r_hat(x, y_w) - r_hat(x, y_l)
    logits = policy_logratios - reference_logratios
    
    # DPO Loss calculation: -log(sigmoid(beta * logits))
    losses = -F.logsigmoid(beta * logits)
    
    # Implicit rewards for logging and monitoring
    chosen_rewards = beta * (policy_chosen_logps - reference_chosen_logps).detach()
    rejected_rewards = beta * (policy_rejected_logps - reference_rejected_logps).detach()
    
    return losses.mean(), chosen_rewards, rejected_rewards

Gradient Analysis of DPO

Differentiable updates to policy parameterization $\theta$ are derived by taking the gradient of $\mathcal{L}_{\text{DPO}}$:

$$\nabla_\theta \mathcal{L}{\text{DPO}}(\theta) = -\beta \mathbb{E}{(x, y_w, y_l)} \left[ \underbrace{\sigma\left( \hat{r}\theta(x, y_l) - \hat{r}\theta(x, y_w) \right)}{\text{Implicit Error Weight } \sigma(\hat{r}l - \hat{r}w)} \cdot \left( \underbrace{\nabla\theta \log \pi\theta(y_w \mid x)}{\text{Increase Preferred}} - \underbrace{\nabla_\theta \log \pi_\theta(y_l \mid x)}_{\text{Decrease Dispreferred}} \right) \right]$$

where $\hat{r}\theta(x, y) = \beta \log \frac{\pi\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}$.

The gradient dynamics reveal two key structural features:

1. Selective Updating: The update direction increases log-likelihood of preferred completion $y_w$ while simultaneously decreasing log-likelihood of dispreferred completion $y_l$.
2. Adaptive Weighting: The step scaling weight $\sigma(\hat{r}_l - \hat{r}_w)$ increases when the policy model incorrectly assigns a higher implicit reward to dispreferred completion $y_l$ than to preferred completion $y_w$. When the policy model already correctly ranks $y_w \succ y_l$, the term approaches zero, preventing over-fitting.

Catastrophic Forgetting and Distribution Shift

Fine-tuning a base model on domain-specific datasets can degrade its performance on unlearned general tasks. This loss of capability is known as catastrophic forgetting.

Pre-trained Parameter Distribution (Broad Reasoning Spectrum):
[ High Logit Entropy ] ---> Wide capability over general domains
 
Domain Fine-Tuned Parameter Distribution (Concentrated Mode Collapse):
[ Low Logit Entropy ]  ---> Overconfidence on target task, degradation on unlearned tasks

Singular Value Decomposition (SVD) Matrix Shift

Analyzing structural updates across dense weight matrices illustrates parameter drift during post-training. Applying Singular Value Decomposition to a base model projection weight matrix $W_0 \in \mathbb{R}^{d \times k}$ yields:

$$W_0 = U \Sigma V^T = \sum_{i=1}^{\min(d, k)} \sigma_i u_i v_i^T$$

where $\sigma_1 \ge \sigma_2 \ge \dots \ge \sigma_r$ represent ordered singular values, and $u_i, v_i$ denote left and right singular vectors encoding dominant spatial directions in latent activation space.

When fine-tuning modifies weights to $W_{\text{tuned}} = W_0 + \Delta W$, the update perturbation matrix $\Delta W$ reshapes the singular value spectrum:

$$\Delta W = U_{\Delta} \Sigma_{\Delta} V_{\Delta}^T$$

If top singular vectors of $\Delta W$ align orthogonal to top singular vectors of $W_0$, fine-tuning distorts feature directions learned during pre-training. This shift can disrupt zero-shot generalization capabilities across unrelated downstream tasks.

Entropy Collapse in Output Logits

During pre-training, causal language models maintain higher entropy distributions across output vocabulary logits $z$:

$$H(P_\theta(\cdot \mid x)) = -\sum_{k=1}^V P_\theta(k \mid x) \log P_\theta(k \mid x)$$

SFT and preference alignment often reduce entropy by penalizing alternative valid token continuations. This results in logit entropy collapse.

import torch
 
def calculate_logit_entropy(logits: torch.Tensor) -> torch.Tensor:
    """
    Computes average Shannon entropy across vocabulary predictions.
    Logits shape: [batch_size, seq_len, vocab_size]
    """
    probs = torch.softmax(logits, dim=-1)
    log_probs = torch.log_softmax(logits, dim=-1)
    
    # Entropy H(P) = -sum(P * log(P))
    entropy = -torch.sum(probs * log_probs, dim=-1)
    return entropy.mean()

As output distribution entropy decreases, model generations become increasingly deterministic. This overconfidence can lead to hallucinations when models evaluate out-of-distribution prompts.

Mitigation Strategies

1. Elastic Weight Consolidation (EWC)

EWC slows down learning on parameters that are critical to previously learned tasks. Parameter importance is estimated using the diagonal values of the Fisher Information Matrix $F$:

$$F_i = \mathbb{E}{x \sim D{\text{pre}}}\left[ \left( \frac{\partial \log P_\theta(x)}{\partial \theta_i} \right)^2 \right]$$

The composite loss function incorporates a quadratic penalty that restricts movement of sensitive parameters:

$$\mathcal{L}{\text{EWC}}(\theta) = \mathcal{L}{\text{SFT}}(\theta) + \sum_i \frac{\lambda}{2} F_i \left( \theta_i - \theta_{0, i} \right)^2$$

import torch
import torch.nn as nn
 
class EWCLoss(nn.Module):
    def __init__(self, model: nn.Module, fisher_matrix: dict, optpar_dict: dict, ewc_lambda: float = 400.0):
        super().__init__()
        self.model = model
        self.fisher = fisher_matrix    # Diagonal Fisher Information values per parameter
        self.optpar = optpar_dict      # Base model optimal parameters theta_0
        self.ewc_lambda = ewc_lambda
 
    def forward(self, sft_loss: torch.Tensor) -> torch.Tensor:
        ewc_penalty = 0.0
        for name, param in self.model.named_parameters():
            if name in self.fisher:
                fisher_val = self.fisher[name]
                opt_val = self.optpar[name]
                ewc_penalty += (fisher_val * (param - opt_val) ** 2).sum()
                
        total_loss = sft_loss + (self.ewc_lambda / 2.0) * ewc_penalty
        return total_loss

2. Pre-training Data Mixture Regularization

Adding a small proportion (5% to 15%) of raw pre-training text tokens directly into SFT instruction datasets helps anchor base model representations, preserving broad language capabilities.

$$\mathcal{L}{\text{joint}}(\theta) = \gamma \mathcal{L}{\text{SFT}}(\theta) + (1 - \gamma) \mathcal{L}_{\text{pre}}(\theta)$$

3. Low-Rank Layer Isolation

Restricting LoRA updates to higher layer attention projections (e.g., layers 24-32 in a 32-layer transformer) keeps early representation layers intact, maintaining base feature extraction mechanisms.

Evaluating Post-Trained Models

Evaluating post-trained models requires combining quantitative logit metrics, standardized task benchmark suites, and adversarial safety tests.

Evaluation Framework Matrix:
┌──────────────────────────┐    ┌──────────────────────────┐    ┌──────────────────────────┐
│   Intrinsic Metrics      │    │   Benchmark Datasets     │    │   Adversarial Safety     │
│ - Perplexity (PPL)       │    │ - MMLU (Knowledge)       │    │ - GCG Jailbreaks         │
│ - Logit Entropy          │    │ - HumanEval (Coding)     │    │ - System Prompt Extraction│
│ - KL Divergence Shift    │    │ - GSM8K (Reasoning)      │    │ - HarmBench Compliance   │
└──────────────────────────┘    └──────────────────────────┘    └──────────────────────────┘

Perplexity Metric Calculations

Perplexity measures how effectively a model predicts a held-out reference evaluation dataset $X = (x_1, x_2, \dots, x_N)$. It is defined as the exponentiated average negative log-likelihood per token:

$$\text{PPL}(X) = \exp\left( -\frac{1}{N} \sum_{i=1}^N \log P_\theta(x_i \mid x_1, \dots, x_{i-1}) \right)$$

Lower perplexity indicates that token predictions closely match the evaluation text distribution.

import torch
import math
 
def calculate_model_perplexity(model, tokenizer, text_sequence: str, device: str = "cuda") -> float:
    model.eval()
    encodings = tokenizer(text_sequence, return_tensors="pt")
    input_ids = encodings.input_ids.to(device)
    
    with torch.no_grad():
        outputs = model(input_ids, labels=input_ids)
        neg_log_likelihood = outputs.loss
        
    perplexity = math.exp(neg_log_likelihood.item())
    return perplexity

Standardized Automated Benchmarks

Post-training evaluation relies on standardized benchmark suites to evaluate specific model capabilities:

1. MMLU (Massive Multitask Language Understanding): Evaluates zero-shot and few-shot knowledge accuracy across 57 academic and professional subjects using multiple-choice prompts.
2. HumanEval: Evaluates Python code synthesis capabilities by measuring pass@k execution correctness across 164 functional programming problems.
3. GSM8K (Grade School Math 8K): Evaluates multi-step mathematical reasoning through chain-of-thought problem solving.
4. MT-Bench: Uses an LLM-as-a-Judge evaluation framework to score multi-turn conversational quality, instruction compliance, and coherence across complex user queries.

Adversarial Safety and Jailbreak Stress Testing

Safety alignment requires testing model resilience against adversarial attacks designed to bypass system prompt boundaries.

Greedy Coordinate Gradient (GCG) Attacks

GCG is an optimization-based attack that appends an adversarial suffix $S$ to a user prompt $P$ to force a model to start its response with an affirmative prefix (e.g., "Sure, here is how to").

The discrete optimization objective searches for token substitutions in suffix $S$ that maximize the probability of generating target response sequence $Y^*$:

$$\min_{S \in \mathcal{V}^{|S|}} -\sum_{t=1}^{|Y^|} \log P_\theta\left(y^t \mid P, S, y^*{<t}\right)$$

Evaluating alignment robustness requires tracking compliance rates across benchmark attack suites (such as HarmBench or AdvGLUE) to verify that post-training modifications successfully reject malicious prompts without causing over-refusal behavior on benign inputs.

Architectural Synthesis & Production Checklist

Post-training transitions base language models into aligned assistant systems through structured parameter space updates.

Post-Training Weight Modification Pipeline:
 
Base Weights (W_0)
  │
  ├── [SFT]: Full parameter updates via masked Cross-Entropy Loss
  │     └─► W_sft = W_0 + ΔW_sft  (Updates all parameter matrices)
  │
  ├── [LoRA]: Low-rank decomposition adapters
  │     └─► W_lora = W_0 + (alpha / r) * (B * A)  (Frozen base weights W_0)
  │
  └── [DPO]: Implicit reward alignment
        └─► Minimizes L_DPO(θ; π_ref) directly on preference pairs (y_w, y_l)

Fine-Tuning & Alignment Reference Matrix

Production Deployment Checklist

To ensure post-trained models execute cleanly without performance degradation or weight corruption:

1. Verify Tokenizer Masking: Ensure prompt tokens are assigned label -100 so loss gradients evaluate exclusively over target response positions.
2. Check Adapter Initialization: Confirm matrix $B = 0$ and $A \sim \mathcal{N}(0, 1/r)$ at step zero so $\Delta W = 0$ before training begins.
3. Set Scaling Factor $\alpha$: Set $\alpha = 1 \cdot r$ or $\alpha = 2 \cdot r$ to keep gradient scales invariant across low-rank tuning adjustments.
4. Merge Weights Prior to Serving: For zero-latency inference, run $W_{\text{merged}} = W_0 + \frac{\alpha}{r} (B A)$ to eliminate adapter lookup overhead.
5. Monitor Logit Entropy: Track Shannon entropy $H(P_\theta)$ during DPO alignment to prevent distribution collapse and mode collapse hallucinations.
6. Evaluate General Capability Benchmarks: Run MMLU and GSM8K post-alignment to measure and limit catastrophic forgetting.