Fine-Tuning with Unsloth

Overview

A comprehensive skill for fine-tuning LLMs using Unsloth — the fastest fine-tuning library that achieves 2-5x speedup with 70% less memory through custom CUDA kernels and mathematical optimizations. Unsloth supports all major architectures (Llama, Mistral, Gemma, Phi, Qwen) and is a drop-in replacement for HuggingFace training with automatic optimizations.

When to Use

• Fine-tuning on consumer GPUs (RTX 3090, 4090, T4, etc.)
• Need maximum training speed with minimal memory
• Want automatic optimizations without code changes
• Fine-tuning Llama, Mistral, Gemma, or other popular models
• Exporting to GGUF, vLLM, or other deployment formats
• Need free tier for Google Colab fine-tuning

Quick Start

Copy
# Install (auto-detects CUDA version)
pip install unsloth

# Or for specific CUDA
pip install "unsloth[cu121]"  # CUDA 12.1
pip install "unsloth[cu118]"  # CUDA 11.8

# Free Colab notebook available at github.com/unslothai/unsloth

Copy
from unsloth import FastLanguageModel

# Load model with Unsloth optimizations (automatic 2x speedup)
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Llama-3.1-8B-Instruct",
    max_seq_length=4096,
    load_in_4bit=True,     # QLoRA
    dtype=None,            # Auto-detect
)

# Add LoRA adapters (Unsloth-optimized)
model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    lora_alpha=16,
    lora_dropout=0,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                     "gate_proj", "up_proj", "down_proj"],
    use_gradient_checkpointing="unsloth",  # 30% less memory
)

Full Training Example

Copy
from unsloth import FastLanguageModel
from transformers import TrainingArguments
from trl import SFTTrainer
from datasets import load_dataset

# 1. Load model
model, tokenizer = FastLanguageModel.from_pretrained(
    "unsloth/Llama-3.1-8B-Instruct",
    max_seq_length=4096,
    load_in_4bit=True,
)

# 2. Add LoRA
model = FastLanguageModel.get_peft_model(
    model, r=16, lora_alpha=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                     "gate_proj", "up_proj", "down_proj"],
    use_gradient_checkpointing="unsloth",
)

# 3. Prepare dataset with chat template
dataset = load_dataset("my_dataset", split="train")

def format_chat(example):
    messages = [
        {"role": "user", "content": example["instruction"]},
        {"role": "assistant", "content": example["response"]},
    ]
    return {"text": tokenizer.apply_chat_template(messages, tokenize=False)}

dataset = dataset.map(format_chat)

# 4. Train
trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=4096,
    packing=True,
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        num_train_epochs=1,
        learning_rate=2e-4,
        bf16=True,
        logging_steps=10,
        output_dir="./output",
        optim="adamw_8bit",
        seed=42,
    ),
)

trainer.train()

# 5. Save and export
model.save_pretrained("./lora-adapter")

# Export to GGUF for llama.cpp
model.save_pretrained_gguf("./gguf-model", tokenizer, quantization_method="q4_k_m")

# Export merged model for vLLM
model.save_pretrained_merged("./merged", tokenizer, save_method="merged_16bit")

Unsloth vs Standard Training

Metric	Standard HF	Unsloth	Improvement
Training Speed	1x	2-5x	2-5x faster
Memory Usage	1x	0.3x	70% less
Llama-3 8B QLoRA	18GB	6GB	3x less
Llama-3 70B QLoRA	80GB+	48GB	Fits on 1 GPU
Accuracy	Baseline	Same	0% loss
Code Changes	-	Minimal	Drop-in replacement

Export Formats

Copy
# GGUF (for llama.cpp, Ollama)
model.save_pretrained_gguf("model", tokenizer, quantization_method="q4_k_m")
# Options: q4_0, q4_k_m, q5_k_m, q8_0, f16

# Merged 16-bit (for vLLM, TGI)
model.save_pretrained_merged("model", tokenizer, save_method="merged_16bit")

# Merged 4-bit (for bitsandbytes inference)
model.save_pretrained_merged("model", tokenizer, save_method="merged_4bit")

# Push to Hub
model.push_to_hub_merged("username/my-model", tokenizer)
model.push_to_hub_gguf("username/my-model-gguf", tokenizer, quantization_method="q4_k_m")

Best Practices

1. Use load_in_4bit=True — 4-bit QLoRA with Unsloth optimizations is the sweet spot
2. Use use_gradient_checkpointing="unsloth" — Unsloth's custom implementation is 30% more efficient
3. Enable packing — Packs short samples together for better GPU utilization
4. Use rank 16 — Good balance; increase to 32-64 for complex tasks
5. Use adamw_8bit optimizer — Saves ~30% optimizer memory
6. Set dropout to 0 — Unsloth works best without LoRA dropout
7. Export to GGUF for deployment — One command for llama.cpp/Ollama format
8. Use Unsloth model variants — Pre-optimized models from unsloth/ on HuggingFace
9. Train for 1-3 epochs — More epochs usually means overfitting
10. Start with Colab — Free T4 GPU is enough for 8B models with Unsloth

Troubleshooting

Installation fails

Copy
# Install with specific CUDA version
pip install "unsloth[cu121]"
# Check CUDA version
nvcc --version
python -c "import torch; print(torch.version.cuda)"

OOM even with Unsloth

Copy
# Reduce sequence length
max_seq_length=2048
# Reduce batch size
per_device_train_batch_size=1
# Enable gradient checkpointing
use_gradient_checkpointing="unsloth"

Model not supported

Copy
# Check supported models
from unsloth import FastLanguageModel
# Unsloth supports: Llama, Mistral, Gemma, Phi, Qwen, Yi, CodeLlama
# For unsupported models, use standard HuggingFace + PEFT