Python LLM Fine-Tuning in Depth (2026): The Complete QLoRA Pipeline

AI与大数据AI agents in production(Updated Jul 20, 2026)

Quick preview

Full-param fine-tuning a 7B model needs 4× A100 — impractical for most teams. QLoRA combines 4-bit quantization, low-rank adaptation, and paged optimizer to let a single consumer GPU fine-tune large models.

This article provides a reproducible end-to-end pipeline: from NF4 quantization theory, to PEFT configuration, to TRL training, to vLLM deployment. Every step includes code, data, and gotcha warnings.

Deep principles: what LoRA and QLoRA actually do

LoRA: freeze the base, train only side-paths

LoRA doesn't touch pre-trained weights W. It inserts two trainable low-rank matrices A and B beside every attention layer:

h = W·x + (B·A)·x
    ^^^^   ^^^^^^^^^
    frozen  trainable (r << d, typically 8–64)

During backpropagation, only A and B have non-zero gradients — W's gradient stays zero (frozen). This is why LoRA is memory-efficient: optimizer states (AdamW momentum + variance) only need to track A and B, not the entire W.

Initialization trick: A is initialized with Kaiming/Xavier normal, B as all zeros. This means BA=0 at training start, so the model's initial output is identical to the original pre-trained model — guaranteeing training stability.

For a 7B model, trainable params can be as low as 0.1%–0.5%.

QLoRA: compress the base too

QLoRA extends LoRA by quantizing frozen base weights W to 4-bit NormalFloat (NF4).

Why NF4 beats INT4: INT4 is uniform quantization (equal spacing), assuming weights are uniformly distributed. In reality, pre-trained weights follow an approximately zero-mean normal distribution — concentrated near the mean, sparse in the tails. NF4 is the information-theoretically optimal non-uniform quantization scheme, allocating more quantization levels where probability density is high and fewer in the tails. Experiments show NF4 loses roughly 1/2 to 1/3 as much information as INT4 at equal bit-width.

Double Quantization: quantization requires storing a scaling factor (FP32) for each parameter block. When blocks are small, scaling factor storage becomes non-trivial. Double Quantization quantizes the scaling factors themselves (typically to FP8), saving an extra ~0.4 bits/param.

Paged Optimizer: when GPU VRAM nears capacity, transient allocations can trigger CUDA OOM. The Paged Optimizer splits optimizer states into fixed-size pages and pages them between CPU and GPU — analogous to an OS virtual memory system.

VRAM math

Llama 3.1-8B, ~16 GB BF16 weights:

Component Full-param QLoRA
Model weights 16 GB (BF16) ~3.8 GB (NF4)
Optimizer (AdamW) 32 GB (m+v) ~0.08 GB (LoRA only)
Gradients 16 GB ~0.08 GB
Activations (batch=4,seq=2048) ~12 GB ~12 GB
Total ~76 GB ❌ ~16 GB ✅

A 24 GB card is ample; even 16 GB cards (RTX 4060 Ti 16GB) can run with small batches.

Setup

pip install torch transformers accelerate peft bitsandbytes trl datasets tensorboard

Version gotchas: bitsandbytes must match CUDA version. trl 0.9+ uses SFTConfig. Windows users: use WSL2.

Load 4-bit quantized model

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True, bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=bnb_config, device_map="auto", torch_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")
tokenizer.pad_token = tokenizer.eos_token

Configure LoRA

from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

model = prepare_model_for_kbit_training(model)

lora_config = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    bias="none", task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable: ~42M / ~8B = 0.52%

Hyperparameter guide

Param Recommendation Rationale
r 8–32 r controls the rank. r=8 means expressing ΔW with 8 basis vectors — sufficient for most tasks. Returns diminish rapidly beyond 32.
lora_alpha r × 1–2 Effective learning rate ∝ alpha/r. With r=16, alpha=32 gives scaling 2.0.
lora_dropout 0.05 Regularization. Small data (<500) → 0.1. Large data → 0.
target_modules More is better Llama: q/k/v/o + gate/up/down. Check state_dict for other architectures.

Prepare dataset

from datasets import load_dataset

dataset = load_dataset("tatsu-lab/alpaca")

def format_instruction(example):
    prompt = f"### Instruction:\n{example['instruction']}\n\n### Response:\n{example['output']}"
    return {"text": prompt}

dataset = dataset.map(format_instruction)

💡 200–500 samples to validate → 2k–5k for visible improvement → 10k+ for competitive results. Quality >> quantity.

Training: TRL SFTTrainer

from trl import SFTTrainer, SFTConfig

training_args = SFTConfig(
    output_dir="./qlora-llama3-chat",
    per_device_train_batch_size=4, gradient_accumulation_steps=4,
    num_train_epochs=3, max_seq_length=2048,
    logging_steps=10, save_steps=200,
    learning_rate=2e-4, lr_scheduler_type="cosine", warmup_ratio=0.03,
    optim="paged_adamw_8bit", bf16=True, gradient_checkpointing=True,
    report_to="tensorboard",
)

trainer = SFTTrainer(model=model, args=training_args,
                     train_dataset=dataset["train"], tokenizer=tokenizer)
trainer.train()

Evaluation

from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B", torch_dtype=torch.bfloat16, device_map="auto"
)
base = PeftModel.from_pretrained(base, "./qlora-llama3-chat/checkpoint-600")
base = base.merge_and_unload()

def generate(prompt, max_new=256):
    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
    outputs = base.generate(**inputs, max_new_tokens=max_new, temperature=0.7)
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

Quantitative: GPT-4 as judge, score 1–5 on accuracy/completeness/helpfulness. Average > 4.0 is acceptable.

Save & deploy

trainer.save_model("./qlora-llama3-chat-final")  # ~60 MB, adapter only

model = PeftModel.from_pretrained(base, "./qlora-llama3-chat-final")
model = model.merge_and_unload()
model.save_pretrained("./merged-model")

vLLM

python -m vllm.entrypoints.openai.api_server \
  --model ./merged-model --dtype bfloat16 --max-model-len 4096 --port 8000

FAQ

Q1: QLoRA vs full fine-tune quality? — 96–99% on most domain-adaptation tasks. Only extreme tasks need full params.

Q2: r=8 vs r=64? — r=8 sufficient for medium datasets. Don't waste compute on r — ensure data quality first.

Q3: Multiple adapters? — model.set_adapter() switches in milliseconds without reloading the base.

Q4: Low loss, bad responses? — Overfitting to prompt template. Add diversity, reduce epochs, increase dropout.

Q5: CPU training? — Extremely slow. Use Colab free T4 / Kaggle free P100.

Conclusion

Recommended path: 200 samples validate → TensorBoard tune → manual test → add data (not r) → merge → vLLM deploy.

#LLM#微调#QLoRA#LoRA#PEFT#量化