第 47 章
第 47 章：全量 v2 微调实验
第 47 章：全量 v2 微调实验
本文整理自 Datawhale 开源项目 datawhalechina/jev-cookbook（CC BY-NC-SA 4.0），源文件：main/10_本地模型/notebooks/full_v2_finetuning.ipynb。
本 Notebook 把三套训练脚本组织成同一数据、同一拆分、同一评估口径的对照实验。它是启动与比较入口；真正的训练逻辑仍由 finetune_reviewed_jsonl.py 和 finetune_variant_jsonl.py 执行。
| 方法 | 更新哪些参数 | 训练目标 | 这组对照想回答什么 |
|---|---|---|---|
| Head-only | 只更新决策 head，encoder 冻结 | soft cross-entropy | 只适配输出头是否已足够？ |
| LoRA-SFT | 冻结 encoder 主权重，在注意力投影加 LoRA，并更新 head | soft cross-entropy | 少量可训练参数能否提升效果？ |
| RLCD-style | 更新 encoder 与 head，以探索采样、proper-score 奖励、组基线和辅助 CE 优化 | 本项目的 RLCD-style 实现 | 直接调整表征并加入概率决策信号是否有收益？ |
三者使用同一 train/dev/calibration/test 来源组切分与随机种子；学习率按方法分别设置。checkpoint 只按 dev 选择，定版后才评估 calibration/test。这里的 RLCD-style 是本项目的实验实现，不是上游官方 RLCD 配方的完整复现。
重要：首次打开时不会训练、下载模型或语料，也不会调用标注 API。 训练开关、伪标签授权和大文件下载均默认关闭。全量实验训练耗时较长，本 Notebook 只提供可审阅的启动配置；请在确认数据、GPU 和预算后自行开启训练。
当前 v2 数据是 DeepSeek 投票产生、尚待人工审核的伪标签。实验指标仅反映模型对这些代理标签的拟合，不是人工金标准确率或概率校准结论。重新生成标签只需 DeepSeek API key；训练、读取已有数据和运行 notebook 不需要 Jev API key。数据与训练流程说明见服务器项目中的 微调指南；历史全量实验指标见 留出集评估 JSON]。
AutoDL 一键启动
Notebook 放在 /root/autodl-tmp/full_v2_finetuning.ipynb，训练脚本在 /root/autodl-tmp/laya-training-project/laya/，候选数据、模型权重和实验输出均使用 AutoDL 持久盘路径。服务器已准备 /root/autodl-tmp/laya-venv、模型和数据后，在 SSH 终端执行一条命令即可启动 JupyterLab 并打开本 Notebook：
bash /root/autodl-tmp/start_autodl_full_v2_finetuning.sh默认只启动 JupyterLab，不会自动开始训练、下载或调用标注 API。Notebook 中 RUN_TRAINING、DOWNLOAD_MODEL_IF_NEEDED 和 DeepSeek 标注开关仍默认关闭；首次训练伪标签还需显式开启授权。Jupyter 只监听 127.0.0.1:8888。在本地电脑另开终端建立 SSH 隧道：
ssh -p 29470 -L 8888:127.0.0.1:8888 root@connect.nmb2.seetacloud.com然后打开启动日志给出的本地 JupyterLab 地址和 token。启动脚本会检查项目脚本、候选数据、模型文件及 JupyterLab 是否齐全，并将训练结果写入 /root/autodl-tmp/experiments/laya/。
from pathlib import Path
from datetime import datetime
import os
import uuid
import sys
cwd = Path.cwd().expanduser().resolve()
root_candidates = []
if os.environ.get("LAYA_PROJECT_ROOT"):
root_candidates.append(Path(os.environ["LAYA_PROJECT_ROOT"]).expanduser())
root_candidates.extend((cwd, *cwd.parents))
root_candidates.extend(parent / "laya-training-project" for parent in (cwd, *cwd.parents))
ROOT = next(
(candidate.resolve() for candidate in root_candidates
if (candidate / "laya" / "finetune_reviewed_jsonl.py").is_file()),
None,
)
if ROOT is None:
raise FileNotFoundError(
"找不到 Laya 训练脚本。请设置 LAYA_PROJECT_ROOT，或从项目目录及其子目录打开本 Notebook。"
)
sys.path.insert(0, str(ROOT))
LAYA_DIR = ROOT / "laya"
GENERATED_DIR = LAYA_DIR / "data_generation" / "generated" / "sharegpt_zh_38k"
DATA_DIR = GENERATED_DIR / "v2"
RAW_DIR = GENERATED_DIR / "raw"
POLICY_PATH = LAYA_DIR / "data_generation" / "sharegpt_policy_v2.json"
RAW_PATH = RAW_DIR / "sharegpt_zh_38K_format.jsonl"
CASES_PATH = DATA_DIR / "cases.jsonl"
VOTES_PATH = DATA_DIR / "deepseek_votes.jsonl"
AUTODL_CANDIDATES = Path(
"/root/autodl-tmp/datasets/laya-datasets/sharegpt_zh_38k/v2/laya_candidates.jsonl"
)
DEFAULT_CANDIDATES = (
AUTODL_CANDIDATES if AUTODL_CANDIDATES.is_file()
else DATA_DIR / "laya_candidates.jsonl"
)
CANDIDATES_PATH = Path(
os.environ.get("LAYA_CANDIDATES_PATH", str(DEFAULT_CANDIDATES))
).expanduser()
REVIEW_CSV_PATH = DATA_DIR / "human_review.csv"
AUTODL_MODEL_DIR = Path("/root/autodl-tmp/models/laya/multilingual")
DEFAULT_MODEL_DIR = (
AUTODL_MODEL_DIR if AUTODL_MODEL_DIR.is_dir()
else LAYA_DIR / "models" / "multilingual"
)
MODEL_DIR = Path(os.environ.get("LAYA_MODEL_DIR", str(DEFAULT_MODEL_DIR))).expanduser()
# AutoDL 预留数据盘优先；其他机器默认写入用户目录。可用 LAYA_OUTPUT_DIR 覆盖。
AUTODL_OUTPUT = Path("/root/autodl-tmp/experiments/laya")
DEFAULT_OUTPUT = AUTODL_OUTPUT if AUTODL_OUTPUT.exists() else Path.home() / "laya-runs"
OUTPUT_ROOT = Path(os.environ.get("LAYA_OUTPUT_DIR", str(DEFAULT_OUTPUT))).expanduser()
OUTPUT_ROOT.mkdir(parents=True, exist_ok=True)
RUN_ID = os.environ.get("LAYA_RUN_ID") or (datetime.now().strftime("full-v2-%Y%m%d-%H%M%S") + "-" + uuid.uuid4().hex[:6])
RUN_DIR = OUTPUT_ROOT / RUN_ID
SPLIT_DIR = OUTPUT_ROOT / (RUN_ID + "-splits")
print("仓库:", ROOT)
print("候选数据:", CANDIDATES_PATH)
print("模型目录:", MODEL_DIR)
print("实验输出:", RUN_DIR)
print("切分输出:", SPLIT_DIR)1. 检查训练环境
本 Notebook 不会自动替换 PyTorch，因为需要保留与本机 NVIDIA 驱动匹配的 CUDA wheel。若缺少 Transformers、Safetensors、NumPy 或 PEFT，可以用下面的开关安装；安装后重启 kernel 并从第一格重新运行。
import importlib.util
import subprocess
INSTALL_MISSING_PACKAGES = False # 远程环境默认不自动改依赖；确认后可显式开启
optional_packages = {
"transformers": "transformers>=4.45",
"safetensors": "safetensors>=0.4",
"numpy": "numpy>=1.26",
"peft": "peft>=0.17",
}
missing_packages = [
package_spec for module_name, package_spec in optional_packages.items()
if importlib.util.find_spec(module_name) is None
]
if importlib.util.find_spec("torch") is None:
raise RuntimeError(
"当前 Python 没有 PyTorch。请先按本机 NVIDIA 驱动安装 CUDA 版 PyTorch，再重启 Jupyter kernel。"
)
if missing_packages:
if not INSTALL_MISSING_PACKAGES:
raise RuntimeError("缺少依赖: " + ", ".join(missing_packages) + "。请在当前 kernel 对应环境安装 laya/requirements.txt 与 peft>=0.17，再重启 kernel。")
subprocess.run(
[sys.executable, "-m", "pip", "install", *missing_packages],
check=True,
)
raise RuntimeError("依赖已安装。请重启 Jupyter kernel，然后从第一格重新运行。")
import torch
import transformers
import safetensors
import numpy as np
import peft
print("Python:", sys.version.split()[0])
print("PyTorch:", torch.__version__, "| CUDA runtime:", torch.version.cuda)
print("Transformers:", transformers.__version__)
print("Safetensors:", safetensors.__version__, "| NumPy:", np.__version__, "| PEFT:", peft.__version__)
print("CUDA available:", torch.cuda.is_available())
if not torch.cuda.is_available():
mps_available = bool(getattr(torch.backends, "mps", None) and torch.backends.mps.is_available())
if mps_available:
raise RuntimeError(
"检测到 Apple MPS，但当前 Head-only、LoRA 和 RLCD-style 训练器只支持 CUDA。"
"请在 NVIDIA CUDA 机器上运行微调；MPS 可用于推理。"
)
raise RuntimeError(
"当前 kernel 没有可用 CUDA。请使用 Windows/Linux NVIDIA GPU 环境，并安装匹配的 CUDA 版 PyTorch。"
)
GPU_NAME = torch.cuda.get_device_name(0)
GPU_VRAM_GIB = torch.cuda.get_device_properties(0).total_memory / 1024**3
print("GPU:", GPU_NAME, "| 显存 GiB:", round(GPU_VRAM_GIB, 1))2. 获取 Laya 多语言 checkpoint
模型文件保存在仓库忽略目录 laya/models/multilingual，不会写进 Git。若本机没有完整 checkpoint，Notebook 会安装 ModelScope Hub CLI 并下载 multilingual 子目录。
import shutil
import sysconfig
DOWNLOAD_MODEL_IF_NEEDED = False # 模型约 644 MB；首次下载前请确认网络与磁盘空间
required_model_files = [
MODEL_DIR / "model.safetensors",
MODEL_DIR / "rl_agent_config.json",
MODEL_DIR / "encoder",
MODEL_DIR / "tokenizer",
]
missing_model_files = [str(path) for path in required_model_files if not path.exists()]
if missing_model_files:
if not DOWNLOAD_MODEL_IF_NEEDED:
raise FileNotFoundError(
"模型文件尚未就绪。请预先下载至 LAYA_MODEL_DIR，或确认空间后将 "
"DOWNLOAD_MODEL_IF_NEEDED 改为 True。缺少: " + ", ".join(missing_model_files)
)
subprocess.run([sys.executable, "-m", "pip", "install", "modelscope-hub"], check=True)
cli_name = "ms-hub.exe" if os.name == "nt" else "ms-hub"
cli_path = Path(sysconfig.get_path("scripts")) / cli_name
if not cli_path.is_file():
found_cli = shutil.which("ms-hub")
if found_cli:
cli_path = Path(found_cli)
else:
raise FileNotFoundError("找不到 ms-hub；安装 ModelScope Hub 后请重启 kernel 并重跑本格。")
subprocess.run(
[
str(cli_path), "download", "convaiinnovations/laya",
"--local-dir", str(MODEL_DIR.parent),
"--include", "multilingual/**",
],
check=True,
)
missing_model_files = [str(path) for path in required_model_files if not path.exists()]
if missing_model_files:
raise FileNotFoundError("模型文件仍不完整: " + ", ".join(missing_model_files))
print("多语言模型已就绪:", MODEL_DIR)3. 复用或构造全量对话决策数据
现有数据路径为 laya/data_generation/generated/sharegpt_zh_38k/v2/laya_candidates.jsonl。新环境没有候选集时，以下格子会按固定 revision 下载公开 ShareGPT 中文对话并筛出 1,900 个来源组。原始 assistant 回复不会进入 state 或标签提示。
DeepSeek 三轮投票需要 API key，Notebook 会读取环境变量 DEEPSEEK_API_KEY，或从 laya/.env 读取；不会回显密钥。打开标注开关前，请确认对数据的发送范围、授权和费用。
# 固定公开数据 revision，确保重建输入来源可追溯。
SOURCE_REVISION = "75412fc0a6a262899c6b99bfa35349d323d0c5c3"
SOURCE_URL = (
"https://www.modelscope.cn/datasets/AI-ModelScope/sharegpt_gpt4/resolve/"
+ SOURCE_REVISION + "/sharegpt_zh_38K_format.jsonl"
)
DOWNLOAD_SOURCE_IF_NEEDED = (os.environ.get("LAYA_DOWNLOAD_SOURCE", "0") == "1"
and not CANDIDATES_PATH.is_file()
and not RAW_PATH.is_file())
RUN_EXTRACTION_IF_NEEDED = not CANDIDATES_PATH.is_file() and not CASES_PATH.is_file()
RAW_DIR.mkdir(parents=True, exist_ok=True)
DATA_DIR.mkdir(parents=True, exist_ok=True)
if not CANDIDATES_PATH.is_file() and not RAW_PATH.is_file() and not DOWNLOAD_SOURCE_IF_NEEDED:
raise FileNotFoundError(
"找不到候选数据或原始语料。请设置 LAYA_CANDIDATES_PATH，或准备 RAW_PATH；"
"确认需要联网下载后，在启动前设置 LAYA_DOWNLOAD_SOURCE=1。"
)
if DOWNLOAD_SOURCE_IF_NEEDED:
from urllib.request import Request, urlopen
print("下载公开语料:", SOURCE_URL)
request = Request(SOURCE_URL, headers={"User-Agent": "Laya-dataset-notebook/1.0"})
temporary_raw = RAW_PATH.with_suffix(RAW_PATH.suffix + ".partial")
total_bytes = 0
with urlopen(request, timeout=60) as response, temporary_raw.open("wb") as out:
while True:
chunk = response.read(8 * 1024 * 1024)
if not chunk:
break
out.write(chunk)
total_bytes += len(chunk)
if total_bytes % (64 * 1024 * 1024) < len(chunk):
print("已下载 MiB:", round(total_bytes / 1024**2))
temporary_raw.replace(RAW_PATH)
print("语料文件 MiB:", round(RAW_PATH.stat().st_size / 1024**2, 1))
elif RAW_PATH.is_file():
print("复用已下载语料:", RAW_PATH)
if RUN_EXTRACTION_IF_NEEDED:
if not RAW_PATH.is_file():
raise FileNotFoundError("需要原始 ShareGPT JSONL: " + str(RAW_PATH))
command = [
sys.executable, str(LAYA_DIR / "data_generation" / "build_sharegpt_laya.py"),
"extract", "--raw", str(RAW_PATH), "--policy", str(POLICY_PATH),
"--out", str(CASES_PATH), "--manifest", str(DATA_DIR / "manifest.json"),
"--seed", "42", "--counts", "train=1200,dev=200,calibration=100,test=400",
]
print("本地筛选对话并保留固定来源组切分。")
subprocess.run(command, cwd=ROOT, check=True)
else:
print("候选/中间数据已存在，跳过下载与筛选。")3.1 可选：DeepSeek 生成三轮伪标签
候选数据缺少时，需先执行本节取得投票文件。把 RUN_DEEPSEEK_ANNOTATION 改为 True 后重新运行本格。API key 可预先放在当前进程的 DEEPSEEK_API_KEY 环境变量或 laya/.env；Notebook 只传递文件路径，不读取或打印密钥。
RUN_DEEPSEEK_ANNOTATION = False # 明确开启后才会产生 API 用量
ENV_FILE = LAYA_DIR / ".env"
if RUN_DEEPSEEK_ANNOTATION:
if not CASES_PATH.is_file():
raise FileNotFoundError("缺少案例文件；先运行上面的公开语料下载与筛选。")
if not os.environ.get("DEEPSEEK_API_KEY") and not ENV_FILE.is_file():
raise RuntimeError("请设置 DEEPSEEK_API_KEY 或在 laya/.env 中配置密钥，再运行标注。")
command = [
sys.executable, str(LAYA_DIR / "data_generation" / "build_sharegpt_laya.py"),
"annotate", "--cases", str(CASES_PATH), "--policy", str(POLICY_PATH),
"--votes-out", str(VOTES_PATH), "--batch-size", "10", "--workers", "4", "--votes", "3",
]
if ENV_FILE.is_file():
command.extend(["--env-file", str(ENV_FILE)])
subprocess.run(command, cwd=ROOT, check=True)
else:
print("DeepSeek API 标注未开启。")
if not VOTES_PATH.is_file() and not CANDIDATES_PATH.is_file():
print("要从零构造数据，请确认授权与预算后把 RUN_DEEPSEEK_ANNOTATION 改为 True。")3.2 合并投票并导出候选与人工审核表
已有候选集时会直接复用。若有 cases 和 votes、但还没有 candidates，则自动合并并生成逐题审核表。
if not CANDIDATES_PATH.is_file() and CASES_PATH.is_file() and VOTES_PATH.is_file():
if REVIEW_CSV_PATH.is_file():
raise FileExistsError(
"审核表已存在，为避免覆盖人工复核结果，已停止重新 assemble。"
"请先备份/迁移审核表，再设置新的 DATA_DIR。"
)
command = [
sys.executable, str(LAYA_DIR / "data_generation" / "build_sharegpt_laya.py"),
"assemble", "--cases", str(CASES_PATH), "--policy", str(POLICY_PATH),
"--votes", str(VOTES_PATH), "--out", str(CANDIDATES_PATH),
"--review-csv", str(REVIEW_CSV_PATH), "--manifest", str(DATA_DIR / "manifest.json"),
]
subprocess.run(command, cwd=ROOT, check=True)
elif CANDIDATES_PATH.is_file():
print("候选数据已存在，跳过 assemble。")
if not CANDIDATES_PATH.is_file():
raise FileNotFoundError(
"还没有全量候选数据: " + str(CANDIDATES_PATH) + "\n"
"如果从零构造，请先执行原始语料/筛选格，再审核数据授权并开启 RUN_DEEPSEEK_ANNOTATION。"
)
print("候选数据:", CANDIDATES_PATH, "| MiB:", round(CANDIDATES_PATH.stat().st_size / 1024**2, 1))
if REVIEW_CSV_PATH.is_file():
print("逐题人工审核表:", REVIEW_CSV_PATH)4. 固定拆分并核验数据
v2 按来源对话组切分：train 1,200、dev 200、calibration 100、test 400，共 1,900 组和 9,500 题。只用 train 更新权重、dev 选择 checkpoint；calibration/test 留到三种方法全部定版后再评估。
import hashlib
import json
def sha256_file(path):
digest = hashlib.sha256()
with Path(path).open("rb") as stream:
for block in iter(lambda: stream.read(8 * 1024 * 1024), b""):
digest.update(block)
return digest.hexdigest()
split_manifest_path = SPLIT_DIR / "manifest.json"
if split_manifest_path.is_file():
existing_manifest = json.loads(split_manifest_path.read_text(encoding="utf-8"))
if existing_manifest.get("source_sha256") != sha256_file(CANDIDATES_PATH):
raise RuntimeError(
"当前切分目录对应其他候选数据。请更改 LAYA_RUN_ID 或 SPLIT_DIR 后重新运行。"
)
elif SPLIT_DIR.exists() and any(SPLIT_DIR.iterdir()):
raise RuntimeError("切分输出目录非空但没有 manifest。请更改 LAYA_RUN_ID 后重跑。")
else:
command = [
sys.executable, str(LAYA_DIR / "data_generation" / "prepare_candidate_splits.py"),
"--input", str(CANDIDATES_PATH), "--output-dir", str(SPLIT_DIR),
]
subprocess.run(command, cwd=ROOT, check=True)
existing_manifest = json.loads(split_manifest_path.read_text(encoding="utf-8"))
print("切分清单:", split_manifest_path)
print("候选 SHA256:", existing_manifest["source_sha256"])
print("切分计数:", existing_manifest["planned_split_counts"])
print("导出文件:", existing_manifest["files"])from collections import Counter
split_rows = {}
for filename in ("train-dev.jsonl", "calibration.jsonl", "test.jsonl"):
path = SPLIT_DIR / filename
split_rows[filename] = [
json.loads(line) for line in path.read_text(encoding="utf-8").splitlines() if line.strip()
]
train_dev = split_rows["train-dev.jsonl"]
train_rows = [row for row in train_dev if row.get("split") == "train"]
dev_rows = [row for row in train_dev if row.get("split") == "dev"]
cal_rows = split_rows["calibration.jsonl"]
test_rows = split_rows["test.jsonl"]
assert (len(train_rows), len(dev_rows), len(cal_rows), len(test_rows)) == (1200, 200, 100, 400)
assert sum(len(row["qs"]) for row in train_rows) == 6000
assert sum(len(row["qs"]) for row in dev_rows) == 1000
assert sum(len(row["qs"]) for row in cal_rows) == 500
assert sum(len(row["qs"]) for row in test_rows) == 2000
all_rows = train_rows + dev_rows + cal_rows + test_rows
assert all((row.get("metadata") or {}).get("review_status") == "needs_human_review" for row in all_rows)
assert all(row.get("split") in {"train", "dev", "calibration", "test"} for row in all_rows)
print("records:", len(all_rows), "| questions:", sum(len(row["qs"]) for row in all_rows))
print("split:", dict(Counter(row["split"] for row in all_rows)))
print("review state:", dict(Counter(row["metadata"]["review_status"] for row in all_rows)))
print("question types:", dict(Counter(q["t"] for row in all_rows for q in row["qs"])))
print("全量 schema、计划拆分和审核状态检查通过。")5. 训练设置：显式启用伪标签并行实验
三种训练器共用同一个 train/dev 文件和随机种子。Head-only 只更新决策头；LoRA-SFT 更新 encoder LoRA 与决策头；RLCD-style 更新 encoder 与决策头，使用探索采样、组基线和交叉熵辅助项。这里的 RLCD-style 是本项目实验实现，不是对上游 RLCD 全部细节的完整复现。
候选标签尚未经过人工审核。若这次实验有意研究 DeepSeek 伪标签，请把 ALLOW_UNREVIEWED_PSEUDOLABELS 设为 True。该开关仅传递显式实验授权，不会更改数据审核字段，也不能将结果作为人工金标质量结论。
三种方法的数据拆分和训练轮数一致，但参数规模、目标函数与学习率不同；这是一组方法对照，不是严格控制所有变量的单因素消融。先串行运行可以减少显存争用并让日志更容易排查。RTX 3090 24 GB 是本项目验证过的并行配置；低于 20 GB 必须串行或降低 max_tokens。
| 方法 | 可训练部分 | 本 Notebook 参数 |
|---|---|---|
| Head-only | 决策 head | head LR 1e-4 |
| LoRA-SFT | Encoder 的 attention 投影 LoRA 与决策 head | LR 5e-5，rank 8、alpha 16、dropout 0.05 |
| RLCD-style | Encoder 与决策 head | LR 2e-5，4 个探索样本、标准差 0.5、CE 辅助权重 0.1 |
共同参数是 seed 42、最多 4 轮、dev patience 2、max_tokens 1024、max_seqs 4。
当前 RLCD-style 单元复现的是本项目此前的实验参数，不等于官方训练配方；该 run 的 accuracy 接近 LoRA，但 soft cross-entropy 明显更高。官方配方差异和原因排查见 微调指南。按官方核心参数在全量对话决策数据上复训的独立入口是 laya-training-project/laya/finetune_rlcd_official_jsonl.py，其参数、结果和 AutoDL 持久目录见 官方配方复跑报告]。这次使用单卡和项目自己的 321.9M 本地基座，属于适配复跑，不是官方 2×T4 / 421M typed-decisions Notebook 的逐项复现。
METHODS_TO_RUN = ["head-only", "lora-sft", "rlcd"]
ALLOW_UNREVIEWED_PSEUDOLABELS = False # 伪标签实验须由使用者显式改为 True
RUN_TRAINING = False # 确认参数、GPU 和数据后再改为 True
PARALLEL_TRAINING = False # 先串行；确认显存充足后再改为 True
EPOCHS = 4
PATIENCE = 2
MAX_TOKENS = 1024
MAX_SEQS = 4
SEED = 42
assert set(METHODS_TO_RUN) <= {"head-only", "lora-sft", "rlcd"}
assert len(METHODS_TO_RUN) == len(set(METHODS_TO_RUN))
if RUN_TRAINING:
if any(row["metadata"]["review_status"] == "needs_human_review" for row in train_rows):
if not ALLOW_UNREVIEWED_PSEUDOLABELS:
raise RuntimeError(
"数据尚未人工审核。确认仅做伪标签研究后，将 ALLOW_UNREVIEWED_PSEUDOLABELS 改为 True。"
)
if PARALLEL_TRAINING and GPU_VRAM_GIB < 20:
raise RuntimeError("当前 GPU 显存低于 20 GiB；请将 PARALLEL_TRAINING 设为 False。")
print("方法:", METHODS_TO_RUN)
print("训练开启:", RUN_TRAINING, "| 并行:", PARALLEL_TRAINING)
print("训练轮数:", EPOCHS, "| max_tokens:", MAX_TOKENS, "| max_seqs:", MAX_SEQS)
if not RUN_TRAINING:
print("当前只完成数据核验。确认后将 RUN_TRAINING=True；伪标签还需单独显式启用。")import time
from collections import OrderedDict
def build_training_command(method, output_dir):
common_args = [
"--data", str(SPLIT_DIR / "train-dev.jsonl"),
"--model-dir", str(MODEL_DIR),
"--output-dir", str(output_dir),
"--epochs", str(EPOCHS), "--patience", str(PATIENCE),
"--max-tokens", str(MAX_TOKENS), "--max-seqs", str(MAX_SEQS),
"--seed", str(SEED),
]
if ALLOW_UNREVIEWED_PSEUDOLABELS:
common_args.append("--allow-unreviewed-pseudolabels")
if method == "head-only":
return [
sys.executable, str(LAYA_DIR / "finetune_reviewed_jsonl.py"),
*common_args, "--head-lr", "1e-4",
]
if method == "lora-sft":
return [
sys.executable, str(LAYA_DIR / "finetune_variant_jsonl.py"),
"--method", "lora_sft", *common_args, "--lr", "5e-5", "--lora-rank", "8",
]
return [
sys.executable, str(LAYA_DIR / "finetune_variant_jsonl.py"),
"--method", "rlcd", *common_args, "--lr", "2e-5",
"--rl-samples", "4", "--exploration-std", "0.5", "--aux-weight", "0.1",
]
def tail_text(path, max_lines=4):
try:
return "\n".join(path.read_text(encoding="utf-8", errors="replace").splitlines()[-max_lines:])
except OSError:
return "(日志尚未生成)"
if RUN_TRAINING:
if any(row["metadata"]["review_status"] == "needs_human_review" for row in train_rows) and not ALLOW_UNREVIEWED_PSEUDOLABELS:
raise RuntimeError("当前数据是 needs_human_review 伪标签；训练前必须显式开启伪标签实验。")
RUN_DIR.mkdir(parents=True, exist_ok=True)
process_map = OrderedDict()
log_handles = {}
for method in METHODS_TO_RUN:
method_dir = RUN_DIR / method
if method_dir.exists() and any(method_dir.iterdir()):
raise RuntimeError("输出目录非空；为避免覆盖请设置新的 LAYA_RUN_ID: " + str(method_dir))
try:
launch_order = METHODS_TO_RUN if PARALLEL_TRAINING else list(METHODS_TO_RUN)
for method in launch_order:
method_dir = RUN_DIR / method
log_path = RUN_DIR / (method + ".log")
log_handles[method] = log_path.open("w", encoding="utf-8")
child_env = os.environ.copy()
child_env["PYTHONUNBUFFERED"] = "1"
process_map[method] = subprocess.Popen(
build_training_command(method, method_dir),
cwd=ROOT, env=child_env, stdout=log_handles[method],
stderr=subprocess.STDOUT,
)
print("已启动:", method, "| pid:", process_map[method].pid, "| 日志:", log_path)
if not PARALLEL_TRAINING:
while process_map[method].poll() is None:
time.sleep(20)
print(method, "仍在运行\n" + tail_text(log_path))
if process_map[method].returncode != 0:
raise subprocess.CalledProcessError(process_map[method].returncode, method)
if PARALLEL_TRAINING:
last_report = 0
while any(process.poll() is None for process in process_map.values()):
time.sleep(20)
now = time.time()
if now - last_report >= 30:
last_report = now
for method, process in process_map.items():
state = "running" if process.poll() is None else "exit=" + str(process.returncode)
print("\n[" + method + "] " + state + "\n" + tail_text(RUN_DIR / (method + ".log")))
failed = {name: p.returncode for name, p in process_map.items() if p.returncode != 0}
if failed:
raise RuntimeError("训练进程失败，请查看对应日志: " + str(failed))
except KeyboardInterrupt:
print("收到中断请求，正在停止本 Notebook 启动的训练进程。")
for process in process_map.values():
if process.poll() is None:
process.terminate()
for process in process_map.values():
try:
process.wait(timeout=10)
except subprocess.TimeoutExpired:
process.kill()
raise
finally:
for handle in log_handles.values():
handle.close()
print("训练结束。完整输出:", RUN_DIR)
else:
print("尚未启动训练。需要实际训练时先在上一格确认开关，再重跑本格。")6. 汇总训练报告
每个方法目录包含 experiment.json、checkpoint、训练日志和指标。下面只读取已经完成的结果；若刚才没有启动训练，可对已有 RUN_DIR 调用。
import json
training_summaries = {}
for method in METHODS_TO_RUN:
experiment_path = RUN_DIR / method / "experiment.json"
if not experiment_path.is_file():
print("尚无完成报告:", method, "|", experiment_path)
continue
experiment = json.loads(experiment_path.read_text(encoding="utf-8"))
training_summaries[method] = {
"best_epoch": experiment.get("best_epoch"),
"dev_before": experiment.get("dev_before"),
"dev_after": experiment.get("dev_after"),
"training_seconds": experiment.get("training_seconds"),
"peak_vram_gib": experiment.get("peak_vram_gib"),
"checkpoint": str(RUN_DIR / method / "model.safetensors"),
}
training_summaries7. 锁定 checkpoint 后运行留出评估
训练和 dev checkpoint 选择完成后，才使用 calibration 与 test。这里的 calibration 是预留 split 名称；当前脚本只报告伪标签 soft-CE 和 argmax accuracy，没有拟合温度缩放或声称真实概率校准。三种方法必须都有 checkpoint。
RUN_HOLDOUT_EVALUATION = False # 三个模型训练完成并检查后，再显式改为 True
if RUN_HOLDOUT_EVALUATION:
if any(row["metadata"]["review_status"] == "needs_human_review" for row in cal_rows + test_rows) and not ALLOW_UNREVIEWED_PSEUDOLABELS:
raise RuntimeError("留出集标签尚未人工审核；只做伪标签研究时需显式开启 ALLOW_UNREVIEWED_PSEUDOLABELS。")
required_checkpoints = [RUN_DIR / method / "model.safetensors" for method in ("head-only", "lora-sft", "rlcd")]
missing = [str(path) for path in required_checkpoints if not path.is_file()]
if missing:
raise FileNotFoundError("留出评估前需要三种方法的 checkpoint: " + ", ".join(missing))
command = [
sys.executable, str(LAYA_DIR / "evaluate_candidate_holdouts.py"),
"--split-dir", str(SPLIT_DIR), "--run-root", str(RUN_DIR),
"--max-tokens", str(MAX_TOKENS), "--max-seqs", str(MAX_SEQS),
"--allow-unreviewed-pseudolabels",
]
subprocess.run(command, cwd=ROOT, check=True)
else:
print("留出集尚未运行。确认 checkpoint 全部定版后，再将 RUN_HOLDOUT_EVALUATION 改为 True。")holdout_path = RUN_DIR / "holdout_metrics.json"
if holdout_path.is_file():
metrics = json.loads(holdout_path.read_text(encoding="utf-8"))
for method, result in metrics["models"].items():
cal = result["splits"]["calibration"]
test = result["splits"]["test"]
print(
method,
"| calibration CE/acc:",
round(cal["soft_cross_entropy"], 4), "/", round(cal["argmax_accuracy"], 4),
"| test CE/acc:",
round(test["soft_cross_entropy"], 4), "/", round(test["argmax_accuracy"], 4),
"| test questions:", test["questions"],
)
print("结果文件:", holdout_path)
else:
print("评估文件还未生成。")8. 产物、复现与解释边界
对照时先看三种方法的 checkpoint 选择轮数、训练时长和峰值显存，再并列比较 calibration/test 的 soft cross-entropy、argmax accuracy 与各题型结果。soft cross-entropy 衡量与投票 soft target 的差异；argmax accuracy 只看最大概率对应标签是否一致。两者都不是人工标注效果或概率校准指标。
实验和切分写在 OUTPUT_ROOT 下；AutoDL 启动脚本会将 LAYA_OUTPUT_DIR 指向持久盘。下载的模型和生成的数据在各自数据目录，不会进 Git。可将实验目录、manifest、CSV/JSONL 日志与报告整理后共享；请先按上游语料授权决定是否能分发原始语料和候选数据。
本 Notebook 的 full-v2 跑法对应已记录的 全量实验报告]。实验报告中的分数以未审核 DeepSeek 投票为参照，只能说明这组伪标签上的拟合/留出表现，不能解释为人工金标质量、生产效果或概率校准。用户应使用人工审核标签和独立业务评测集作正式结论。