第 6 章 微调：文本分类

本章来源：本文翻译整理自 LLMs-from-scratch 仓库的 ch06/01_main-chapter-code/ch06.ipynb，原书为 Sebastian Raschka《Build a Large Language Model (From Scratch)》。

本章要做什么

在本章，我们把预训练的 GPT 模型微调成一个垃圾短信分类器。具体包括：微调的不同类别、准备数据集、创建数据加载器、用预训练权重初始化模型、添加分类头、计算分类损失和准确率、在有监督数据上微调模型、把 LLM 当垃圾短信分类器使用。

本 notebook 使用的包

from importlib.metadata import version

pkgs = ["matplotlib",  # Plotting library
        "numpy",       # PyTorch & TensorFlow dependency
        "tiktoken",    # Tokenizer
        "torch",       # Deep learning library
        "tensorflow",  # For OpenAI's pretrained weights
        "pandas"       # Dataset loading
       ]
for p in pkgs:
    print(f"{p} version: {version(p)}")

6.1 微调的不同类别

本节没有代码。

微调语言模型最常见的方式是指令微调（instruction-finetuning）和分类微调（classification finetuning）。指令微调是下一章的主题。

分类微调，也就是本章的主题，是一个如果你有机器学习背景可能已经熟悉的过程，它和训练卷积网络分类手写数字类似。在分类微调里，我们有一个特定数量的类标签（比如"spam"和"not spam"），模型可以输出。一个分类微调过的模型只能预测它在训练期间见过的类（比如"spam"或"not spam"），而一个指令微调过的模型通常可以执行许多任务。

我们可以把分类微调过的模型想成非常专门的模型；在实践中，创建一个专门模型比创建一个在许多不同任务上都表现良好的通才模型容易得多。

6.2 准备数据集

本节准备我们用于分类微调的数据集。我们用一个由垃圾和非垃圾文本短信组成的数据集来微调 LLM，让它分类它们。

首先，我们下载并解压数据集：

import requests
import zipfile
import os
from pathlib import Path

url = "https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip"
zip_path = "sms_spam_collection.zip"
extracted_path = "sms_spam_collection"
data_file_path = Path(extracted_path) / "SMSSpamCollection.tsv"


def download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path):
    if data_file_path.exists():
        print(f"{data_file_path} already exists. Skipping download and extraction.")
        return

    # Downloading the file
    response = requests.get(url, stream=True, timeout=60)
    response.raise_for_status()
    with open(zip_path, "wb") as out_file:
        for chunk in response.iter_content(chunk_size=8192):
            if chunk:
                out_file.write(chunk)

    # Unzipping the file
    with zipfile.ZipFile(zip_path, "r") as zip_ref:
        zip_ref.extractall(extracted_path)

    # Add .tsv file extension
    original_file_path = Path(extracted_path) / "SMSSpamCollection"
    os.rename(original_file_path, data_file_path)
    print(f"File downloaded and saved as {data_file_path}")


try:
    download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path)
except (requests.exceptions.RequestException, TimeoutError) as e:
    print(f"Primary URL failed: {e}. Trying backup URL...")
    url = "https://f001.backblazeb2.com/file/LLMs-from-scratch/sms%2Bspam%2Bcollection.zip"
    download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path)

数据集保存为制表符分隔的文本文件，我们可以把它加载进 pandas DataFrame：

import pandas as pd

df = pd.read_csv(data_file_path, sep="\t", header=None, names=["Label", "Text"])
df

当我们检查类分布时，我们看到数据里 "ham"（也就是"not spam"）的出现频率远高于 "spam"：

print(df["Label"].value_counts())

为简单起见，也因为教育目的我们本来就偏好小数据集（它将使微调 LLM 更快成为可能），我们对数据集进行二次采样（下采样），让每个类包含 747 个实例。（除了下采样，还有其它几种处理类平衡的方法，但它们超出 LLM 书籍的范围；你可以在 imbalanced-learn 用户指南 里找到示例和更多信息。）

def create_balanced_dataset(df):
    
    # Count the instances of "spam"
    num_spam = df[df["Label"] == "spam"].shape[0]
    
    # Randomly sample "ham" instances to match the number of "spam" instances
    ham_subset = df[df["Label"] == "ham"].sample(num_spam, random_state=123)
    
    # Combine ham "subset" with "spam"
    balanced_df = pd.concat([ham_subset, df[df["Label"] == "spam"]])

    return balanced_df


balanced_df = create_balanced_dataset(df)
print(balanced_df["Label"].value_counts())

接下来，我们把字符串类标签 "ham" 和 "spam" 改成整数类标签 0 和 1：

balanced_df["Label"] = balanced_df["Label"].map({"ham": 0, "spam": 1})    

balanced_df

现在让我们定义一个函数，随机把数据集分成训练、验证和测试子集：

def random_split(df, train_frac, validation_frac):
    # Shuffle the entire DataFrame
    df = df.sample(frac=1, random_state=123).reset_index(drop=True)

    # Calculate split indices
    train_end = int(len(df) * train_frac)
    validation_end = train_end + int(len(df) * validation_frac)

    # Split the DataFrame
    train_df = df[:train_end]
    validation_df = df[train_end:validation_end]
    test_df = df[validation_end:]

    return train_df, validation_df, test_df

train_df, validation_df, test_df = random_split(balanced_df, 0.7, 0.1)
# Test size is implied to be 0.2 as the remainder

train_df.to_csv("train.csv", index=None)
validation_df.to_csv("validation.csv", index=None)
test_df.to_csv("test.csv", index=None)

6.3 创建数据加载器

注意，文本消息的长度不同；如果我们想在一个批次里组合多个训练示例，我们必须要么

1. 把所有消息截断到数据集或批次里最短消息的长度；
2. 把所有消息填充到数据集或批次里最长消息的长度。

我们选择选项 2，把所有消息填充到数据集里最长的消息。为此，我们用 <|endoftext|> 作为填充 token，如第 2 章讨论的：

import tiktoken

tokenizer = tiktoken.get_encoding("gpt2")
print(tokenizer.encode("<|endoftext|>", allowed_special={"<|endoftext|>"}))

下面的 SpamDataset 类识别训练数据集里最长的序列，并给其它序列添加填充 token 来匹配那个序列长度：

import torch
from torch.utils.data import Dataset


class SpamDataset(Dataset):
    def __init__(self, csv_file, tokenizer, max_length=None, pad_token_id=50256):
        self.data = pd.read_csv(csv_file)

        # Pre-tokenize texts
        self.encoded_texts = [
            tokenizer.encode(text) for text in self.data["Text"]
        ]

        if max_length is None:
            self.max_length = self._longest_encoded_length()
        else:
            self.max_length = max_length
            # Truncate sequences if they are longer than max_length
            self.encoded_texts = [
                encoded_text[:self.max_length]
                for encoded_text in self.encoded_texts
            ]

        # Pad sequences to the longest sequence
        self.encoded_texts = [
            encoded_text + [pad_token_id] * (self.max_length - len(encoded_text))
            for encoded_text in self.encoded_texts
        ]

    def __getitem__(self, index):
        encoded = self.encoded_texts[index]
        label = self.data.iloc[index]["Label"]
        return (
            torch.tensor(encoded, dtype=torch.long),
            torch.tensor(label, dtype=torch.long)
        )

    def __len__(self):
        return len(self.data)

    def _longest_encoded_length(self):
        max_length = 0
        for encoded_text in self.encoded_texts:
            encoded_length = len(encoded_text)
            if encoded_length > max_length:
                max_length = encoded_length
        return max_length
        # Note: A more pythonic version to implement this method
        # is the following, which is also used in the next chapter:
        # return max(len(encoded_text) for encoded_text in self.encoded_texts)

train_dataset = SpamDataset(
    csv_file="train.csv",
    max_length=None,
    tokenizer=tokenizer
)

print(train_dataset.max_length)

我们也把验证集和测试集填充到最长的训练序列。注意，验证集和测试集里比最长训练示例长的样本会通过 SpamDataset 代码里的 encoded_text[:self.max_length] 被截断。这个行为完全是可选的，如果我们在验证集和测试集两种情况里都设置 max_length=None，它也会很好地工作：

val_dataset = SpamDataset(
    csv_file="validation.csv",
    max_length=train_dataset.max_length,
    tokenizer=tokenizer
)
test_dataset = SpamDataset(
    csv_file="test.csv",
    max_length=train_dataset.max_length,
    tokenizer=tokenizer
)

接下来，我们用数据集实例化数据加载器，这和前面章节里创建数据加载器类似：

from torch.utils.data import DataLoader

num_workers = 0
batch_size = 8

torch.manual_seed(123)

train_loader = DataLoader(
    dataset=train_dataset,
    batch_size=batch_size,
    shuffle=True,
    num_workers=num_workers,
    drop_last=True,
)

val_loader = DataLoader(
    dataset=val_dataset,
    batch_size=batch_size,
    num_workers=num_workers,
    drop_last=False,
)

test_loader = DataLoader(
    dataset=test_dataset,
    batch_size=batch_size,
    num_workers=num_workers,
    drop_last=False,
)

作为验证步骤，我们遍历数据加载器，确保批次各包含 8 个训练示例，每个训练示例由 120 个 token 组成：

print("Train loader:")
for input_batch, target_batch in train_loader:
    pass

print("Input batch dimensions:", input_batch.shape)
print("Label batch dimensions", target_batch.shape)

最后，让我们打印每个数据集里的批次总数：

print(f"{len(train_loader)} training batches")
print(f"{len(val_loader)} validation batches")
print(f"{len(test_loader)} test batches")

6.4 用预训练权重初始化一个模型

在本节，我们初始化上一章用过的预训练模型。

CHOOSE_MODEL = "gpt2-small (124M)"
INPUT_PROMPT = "Every effort moves"

BASE_CONFIG = {
    "vocab_size": 50257,     # Vocabulary size
    "context_length": 1024,  # Context length
    "drop_rate": 0.0,        # Dropout rate
    "qkv_bias": True         # Query-key-value bias
}

model_configs = {
    "gpt2-small (124M)": {"emb_dim": 768, "n_layers": 12, "n_heads": 12},
    "gpt2-medium (355M)": {"emb_dim": 1024, "n_layers": 24, "n_heads": 16},
    "gpt2-large (774M)": {"emb_dim": 1280, "n_layers": 36, "n_heads": 20},
    "gpt2-xl (1558M)": {"emb_dim": 1600, "n_layers": 48, "n_heads": 25},
}

BASE_CONFIG.update(model_configs[CHOOSE_MODEL])

assert train_dataset.max_length <= BASE_CONFIG["context_length"], (
    f"Dataset length {train_dataset.max_length} exceeds model's context "
    f"length {BASE_CONFIG['context_length']}. Reinitialize data sets with "
    f"`max_length={BASE_CONFIG['context_length']}`"
)

from gpt_download import download_and_load_gpt2
from previous_chapters import GPTModel, load_weights_into_gpt
# If the `previous_chapters.py` file is not available locally,
# you can import it from the `llms-from-scratch` PyPI package.
# For details, see: https://github.com/rasbt/LLMs-from-scratch/tree/main/pkg
# E.g.,
# from llms_from_scratch.ch04 import GPTModel
# from llms_from_scratch.ch05 import download_and_load_gpt2, load_weights_into_gpt

model_size = CHOOSE_MODEL.split(" ")[-1].lstrip("(").rstrip(")")
settings, params = download_and_load_gpt2(model_size=model_size, models_dir="gpt2")

model = GPTModel(BASE_CONFIG)
load_weights_into_gpt(model, params)
model.eval();

为了确保模型被正确加载，让我们复查它能生成连贯的文本：

from previous_chapters import (
    generate_text_simple,
    text_to_token_ids,
    token_ids_to_text
)

# Alternatively:
# from llms_from_scratch.ch05 import (
#    generate_text_simple,
#    text_to_token_ids,
#    token_ids_to_text
# )


text_1 = "Every effort moves you"

token_ids = generate_text_simple(
    model=model,
    idx=text_to_token_ids(text_1, tokenizer),
    max_new_tokens=15,
    context_size=BASE_CONFIG["context_length"]
)

print(token_ids_to_text(token_ids, tokenizer))

在我们把模型微调成分类器之前，让我们看看模型能不能通过提示词就分类垃圾消息：

text_2 = (
    "Is the following text 'spam'? Answer with 'yes' or 'no':"
    " 'You are a winner you have been specially"
    " selected to receive $1000 cash or a $2000 award.'"
)

token_ids = generate_text_simple(
    model=model,
    idx=text_to_token_ids(text_2, tokenizer),
    max_new_tokens=23,
    context_size=BASE_CONFIG["context_length"]
)

print(token_ids_to_text(token_ids, tokenizer))

正如我们看到的，模型不是很擅长遵循指令。这是预期的，因为它只被预训练过，没有被指令微调过（指令微调会在下一章覆盖）。

6.5 添加分类头

在本节，我们修改预训练 LLM，为分类微调做好准备。让我们先看一下模型架构：

print(model)

上面，我们可以清楚地看到我们在第 4 章实现的架构。目标是替换并微调输出层。要做到这一点，我们首先冻结模型，也就是让所有层不可训练：

for param in model.parameters():
    param.requires_grad = False

然后，我们替换输出层（model.out_head），它最初把层输入映射到 50,257 维（词汇表的大小）。因为我们微调模型做二分类（预测 2 个类，"spam" 和 "not spam"），我们可以如下所示替换输出层，它默认是可训练的。注意我们用 BASE_CONFIG["emb_dim"]（在 "gpt2-small (124M)" 模型里等于 768）来让下面的代码更通用：

torch.manual_seed(123)

num_classes = 2
model.out_head = torch.nn.Linear(in_features=BASE_CONFIG["emb_dim"], out_features=num_classes)

从技术上来说，只训练输出层就够了。然而，正如作者在 Finetuning Large Language Models 里发现的，实验显示微调额外的层能显著提高表现。所以，我们也让最后一个 transformer 块和连接最后一个 transformer 块到输出层的最终 LayerNorm 模块可训练：

for param in model.trf_blocks[-1].parameters():
    param.requires_grad = True

for param in model.final_norm.parameters():
    param.requires_grad = True

我们仍然可以像之前章节那样使用这个模型。例如，让我们喂它一些文本输入：

inputs = tokenizer.encode("Do you have time")
inputs = torch.tensor(inputs).unsqueeze(0)
print("Inputs:", inputs)
print("Inputs dimensions:", inputs.shape) # shape: (batch_size, num_tokens)

和之前章节不同的是，它现在有两个输出维度而不是 50,257：

with torch.no_grad():
    outputs = model(inputs)

print("Outputs:\n", outputs)
print("Outputs dimensions:", outputs.shape) # shape: (batch_size, num_tokens, num_classes)

如前面章节讨论的，对每个输入 token，有一个输出向量。因为我们喂给模型一个含 4 个输入 token 的文本样本，上面输出由 4 个 2 维输出向量组成。

在第 3 章，我们讨论了注意力机制，它把每个输入 token 连接到每个其它输入 token。在第 3 章，我们随后还介绍了 GPT 类模型里使用的因果注意力掩码；这个因果掩码让当前 token 只能注意当前和之前的 token 位置。基于这个因果注意力机制，第 4 个（最后一个）token 在所有 token 里包含最多的信息，因为它是唯一包含所有其它 token 信息的 token。因此，我们特别对这个最后一个 token 感兴趣，我们将为垃圾分类任务微调它：

print("Last output token:", outputs[:, -1, :])

6.6 计算分类损失和准确率

在解释损失计算之前，让我们简短看一下模型输出如何被转成类标签。

和第 5 章类似，我们通过 softmax 函数把输出（logits）转成概率分数，然后通过 argmax 函数得到最大概率值的索引位置：

probas = torch.softmax(outputs[:, -1, :], dim=-1)
label = torch.argmax(probas)
print("Class label:", label.item())

注意，这里的 softmax 函数是可选的，如第 5 章解释的，因为最大的输出对应最大的概率分数：

logits = outputs[:, -1, :]
label = torch.argmax(logits)
print("Class label:", label.item())

我们可以把这个概念应用到计算所谓的分类准确率（classification accuracy），它计算给定数据集里正确预测的百分比。要计算分类准确率，我们可以把前面基于 argmax 的预测代码应用到数据集里的所有示例，并计算正确预测的比例，如下所示：

def calc_accuracy_loader(data_loader, model, device, num_batches=None):
    model.eval()
    correct_predictions, num_examples = 0, 0

    if num_batches is None:
        num_batches = len(data_loader)
    else:
        num_batches = min(num_batches, len(data_loader))
    for i, (input_batch, target_batch) in enumerate(data_loader):
        if i < num_batches:
            input_batch, target_batch = input_batch.to(device), target_batch.to(device)

            with torch.no_grad():
                logits = model(input_batch)[:, -1, :]  # Logits of last output token
            predicted_labels = torch.argmax(logits, dim=-1)

            num_examples += predicted_labels.shape[0]
            correct_predictions += (predicted_labels == target_batch).sum().item()
        else:
            break
    return correct_predictions / num_examples

让我们应用这个函数计算不同数据集的分类准确率：

if torch.cuda.is_available():
    device = torch.device("cuda")
elif torch.backends.mps.is_available():
    # Use PyTorch 2.9 or newer for stable mps results
    major, minor = map(int, torch.__version__.split(".")[:2])
    if (major, minor) >= (2, 9):
        device = torch.device("mps")
    else:
        device = torch.device("cpu")
else:
    device = torch.device("cpu")

print("Device:", device)

model.to(device) # no assignment model = model.to(device) necessary for nn.Module classes

torch.manual_seed(123) # For reproducibility due to the shuffling in the training data loader

train_accuracy = calc_accuracy_loader(train_loader, model, device, num_batches=10)
val_accuracy = calc_accuracy_loader(val_loader, model, device, num_batches=10)
test_accuracy = calc_accuracy_loader(test_loader, model, device, num_batches=10)

print(f"Training accuracy: {train_accuracy*100:.2f}%")
print(f"Validation accuracy: {val_accuracy*100:.2f}%")
print(f"Test accuracy: {test_accuracy*100:.2f}%")

正如我们看到的，预测准确率不是很好，因为我们还没有微调模型。

在我们开始微调（训练）之前，我们首先必须定义训练期间要优化的损失函数。目标是最大化模型的垃圾分类准确率；然而，分类准确率不是一个可微分的函数。因此，我们转而最小化交叉熵损失，作为最大化分类准确率的代理（你可以在作者免费提供的 Introduction to Deep Learning 课程的第 8 讲学到更多关于这个主题的内容）。

calc_loss_batch 函数这里和第 5 章一样，除了我们只对优化最后一个 token model(input_batch)[:, -1, :] 感兴趣，而不是所有 token model(input_batch)：

def calc_loss_batch(input_batch, target_batch, model, device):
    input_batch, target_batch = input_batch.to(device), target_batch.to(device)
    logits = model(input_batch)[:, -1, :]  # Logits of last output token
    loss = torch.nn.functional.cross_entropy(logits, target_batch)
    return loss

calc_loss_loader 和第 5 章完全一样：

# Same as in chapter 5
def calc_loss_loader(data_loader, model, device, num_batches=None):
    total_loss = 0.
    if len(data_loader) == 0:
        return float("nan")
    elif num_batches is None:
        num_batches = len(data_loader)
    else:
        # Reduce the number of batches to match the total number of batches in the data loader
        # if num_batches exceeds the number of batches in the data loader
        num_batches = min(num_batches, len(data_loader))
    for i, (input_batch, target_batch) in enumerate(data_loader):
        if i < num_batches:
            loss = calc_loss_batch(input_batch, target_batch, model, device)
            total_loss += loss.item()
        else:
            break
    return total_loss / num_batches

用 calc_loss_loader，我们在开始训练之前计算初始的训练、验证和测试集损失：

with torch.no_grad(): # Disable gradient tracking for efficiency because we are not training, yet
    train_loss = calc_loss_loader(train_loader, model, device, num_batches=5)
    val_loss = calc_loss_loader(val_loader, model, device, num_batches=5)
    test_loss = calc_loss_loader(test_loader, model, device, num_batches=5)

print(f"Training loss: {train_loss:.3f}")
print(f"Validation loss: {val_loss:.3f}")
print(f"Test loss: {test_loss:.3f}")

在下一节，我们训练模型来改进损失值，从而改进分类准确率。

6.7 在有监督数据上微调模型

在本节，我们定义并使用训练函数来改进模型的分类准确率。下面的 train_classifier_simple 函数和第 5 章预训练模型时用的 train_model_simple 函数几乎相同。仅有的两个区别是，我们现在：

1. 跟踪已见训练示例数（examples_seen），而不是已见 token 数；
2. 每个 epoch 之后计算准确率，而不是每个 epoch 之后打印示例文本。

# Overall the same as `train_model_simple` in chapter 5
def train_classifier_simple(model, train_loader, val_loader, optimizer, device, num_epochs,
                            eval_freq, eval_iter):
    # Initialize lists to track losses and examples seen
    train_losses, val_losses, train_accs, val_accs = [], [], [], []
    examples_seen, global_step = 0, -1

    # Main training loop
    for epoch in range(num_epochs):
        model.train()  # Set model to training mode

        for input_batch, target_batch in train_loader:
            optimizer.zero_grad() # Reset loss gradients from previous batch iteration
            loss = calc_loss_batch(input_batch, target_batch, model, device)
            loss.backward() # Calculate loss gradients
            optimizer.step() # Update model weights using loss gradients
            examples_seen += input_batch.shape[0] # New: track examples instead of tokens
            global_step += 1

            # Optional evaluation step
            if global_step % eval_freq == 0:
                train_loss, val_loss = evaluate_model(
                    model, train_loader, val_loader, device, eval_iter)
                train_losses.append(train_loss)
                val_losses.append(val_loss)
                print(f"Ep {epoch+1} (Step {global_step:06d}): "
                      f"Train loss {train_loss:.3f}, Val loss {val_loss:.3f}")

        # Calculate accuracy after each epoch
        train_accuracy = calc_accuracy_loader(train_loader, model, device, num_batches=eval_iter)
        val_accuracy = calc_accuracy_loader(val_loader, model, device, num_batches=eval_iter)
        print(f"Training accuracy: {train_accuracy*100:.2f}% | ", end="")
        print(f"Validation accuracy: {val_accuracy*100:.2f}%")
        train_accs.append(train_accuracy)
        val_accs.append(val_accuracy)

    return train_losses, val_losses, train_accs, val_accs, examples_seen

train_classifier_simple 里用的 evaluate_model 函数和我们第 5 章用的一样：

# Same as chapter 5
def evaluate_model(model, train_loader, val_loader, device, eval_iter):
    model.eval()
    with torch.no_grad():
        train_loss = calc_loss_loader(train_loader, model, device, num_batches=eval_iter)
        val_loss = calc_loss_loader(val_loader, model, device, num_batches=eval_iter)
    model.train()
    return train_loss, val_loss

训练在 M3 MacBook Air 笔记本电脑上大约花 5 分钟，在 V100 或 A100 GPU 上不到半分钟：

import time

start_time = time.time()

torch.manual_seed(123)

optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5, weight_decay=0.1)

num_epochs = 5
train_losses, val_losses, train_accs, val_accs, examples_seen = train_classifier_simple(
    model, train_loader, val_loader, optimizer, device,
    num_epochs=num_epochs, eval_freq=50, eval_iter=5,
)

end_time = time.time()
execution_time_minutes = (end_time - start_time) / 60
print(f"Training completed in {execution_time_minutes:.2f} minutes.")

和第 5 章类似，我们用 matplotlib 画训练集和验证集的损失函数：

import matplotlib.pyplot as plt

def plot_values(epochs_seen, examples_seen, train_values, val_values, label="loss"):
    fig, ax1 = plt.subplots(figsize=(5, 3))

    # Plot training and validation loss against epochs
    ax1.plot(epochs_seen, train_values, label=f"Training {label}")
    ax1.plot(epochs_seen, val_values, linestyle="-.", label=f"Validation {label}")
    ax1.set_xlabel("Epochs")
    ax1.set_ylabel(label.capitalize())
    ax1.legend()

    # Create a second x-axis for examples seen
    ax2 = ax1.twiny()  # Create a second x-axis that shares the same y-axis
    ax2.plot(examples_seen, train_values, alpha=0)  # Invisible plot for aligning ticks
    ax2.set_xlabel("Examples seen")

    fig.tight_layout()  # Adjust layout to make room
    plt.savefig(f"{label}-plot.pdf")
    plt.show()

epochs_tensor = torch.linspace(0, num_epochs, len(train_losses))
examples_seen_tensor = torch.linspace(0, examples_seen, len(train_losses))

plot_values(epochs_tensor, examples_seen_tensor, train_losses, val_losses)

上面，基于向下的斜率，我们看到模型学得很好。而且，训练和验证损失非常接近这个事实表明模型不会倾向于过拟合训练数据。类似地，我们可以在下面画准确率：

epochs_tensor = torch.linspace(0, num_epochs, len(train_accs))
examples_seen_tensor = torch.linspace(0, examples_seen, len(train_accs))

plot_values(epochs_tensor, examples_seen_tensor, train_accs, val_accs, label="accuracy")

基于上面的准确率图，我们可以看到模型在第 4 和 5 个 epoch 后达到相对较高的训练和验证准确率。然而，我们必须记住，我们之前在训练函数里指定了 eval_iter=5，这意味着我们只估计了训练和验证集表现。我们可以如下在完整数据集上计算训练、验证和测试集表现：

train_accuracy = calc_accuracy_loader(train_loader, model, device)
val_accuracy = calc_accuracy_loader(val_loader, model, device)
test_accuracy = calc_accuracy_loader(test_loader, model, device)

print(f"Training accuracy: {train_accuracy*100:.2f}%")
print(f"Validation accuracy: {val_accuracy*100:.2f}%")
print(f"Test accuracy: {test_accuracy*100:.2f}%")

我们可以看到，训练和验证集表现实际上相同。然而，基于略低的测试集表现，我们可以看到模型对训练数据过拟合了一点点，也对用于调整一些超参数（比如学习率）的验证数据过拟合了。然而，这是正常的，这个差距可能可以通过增加模型的 dropout 率（drop_rate）或优化器设置里的 weight_decay 进一步减小。

6.8 把 LLM 用作垃圾短信分类器

最后，让我们把微调过的 GPT 模型用起来。下面的 classify_review 函数实现的数据预处理步骤和我们之前实现的 SpamDataset 类似。然后，这个函数从模型返回预测的整数类标签，并返回对应的类名：

def classify_review(text, model, tokenizer, device, max_length=None, pad_token_id=50256):
    model.eval()

    # Prepare inputs to the model
    input_ids = tokenizer.encode(text)
    supported_context_length = model.pos_emb.weight.shape[0]
    # Note: In the book, this was originally written as pos_emb.weight.shape[1] by mistake
    # It didn't break the code but would have caused unnecessary truncation (to 768 instead of 1024)

    # Truncate sequences if they too long
    input_ids = input_ids[:min(max_length, supported_context_length)]
    assert max_length is not None, (
        "max_length must be specified. If you want to use the full model context, "
        "pass max_length=model.pos_emb.weight.shape[0]."
    )
    assert max_length <= supported_context_length, (
        f"max_length ({max_length}) exceeds model's supported context length ({supported_context_length})."
    )    
    # Alternatively, a more robust version is the following one, which handles the max_length=None case better
    # max_len = min(max_length,supported_context_length) if max_length else supported_context_length
    # input_ids = input_ids[:max_len]
    
    # Pad sequences to the longest sequence
    input_ids += [pad_token_id] * (max_length - len(input_ids))
    input_tensor = torch.tensor(input_ids, device=device).unsqueeze(0) # add batch dimension

    # Model inference
    with torch.no_grad():
        logits = model(input_tensor)[:, -1, :]  # Logits of the last output token
    predicted_label = torch.argmax(logits, dim=-1).item()

    # Return the classified result
    return "spam" if predicted_label == 1 else "not spam"

让我们在下面几个例子上试一下：

text_1 = (
    "You are a winner you have been specially"
    " selected to receive $1000 cash or a $2000 award."
)

print(classify_review(
    text_1, model, tokenizer, device, max_length=train_dataset.max_length
))

text_2 = (
    "Hey, just wanted to check if we're still on"
    " for dinner tonight? Let me know!"
)

print(classify_review(
    text_2, model, tokenizer, device, max_length=train_dataset.max_length
))

最后，让我们保存模型，以防我们之后想复用模型而不必再训练它：

torch.save(model.state_dict(), "review_classifier.pth")

然后，在新会话里，我们可以如下加载模型：

model_state_dict = torch.load("review_classifier.pth", map_location=device, weights_only=True)
model.load_state_dict(model_state_dict)

总结与要点

• 参见 ./gpt_class_finetune.py 脚本，一个用于分类微调的自包含脚本。
• 你可以在 ./exercise-solutions.ipynb 找到练习解答。
• 此外，感兴趣的读者可以在附录 E 找到用低秩适配（LoRA）做参数高效训练的简介。

关键概念

• 分类微调（classification finetuning）：把预训练模型微调成输出固定数量的类标签。
• 指令微调 vs 分类微调：指令微调是多任务通才；分类微调是单一任务的专门模型。
• 冻结（freezing）：让层不可训练；只微调输出层（和最后一层 transformer 块 + LayerNorm）来提升表现。
• 分类头（classification head）：替换输出层为 num_classes 维。
• 最后一个 token：由于因果注意力，最后一个 token 包含所有 token 的信息，分类时用它。
• 分类准确率：正确预测的百分比；不是可微分函数，所以训练用交叉熵损失。
• 填充：用 <|endoftext|> 把短消息填充到最长序列。

练习

练习解答见仓库的 ch06/01_main-chapter-code/exercise-solutions.ipynb。