第 30 章：模型评测实验

本文整理自 Datawhale 开源项目 datawhalechina/jev-cookbook（CC BY-NC-SA 4.0），源文件：main/06_模型评测/01_模型评测.ipynb。

本册来自 Micheal024 的 PR #6，内容按原样收录。 配套评测框架在 benchmark/（runner 账本、适配器、JevBench 公开题集与预置演示产物）。

第六章回答一个问题：怎么公平地评测一个判断模型？ 用 JevBench 风格的基准（231 道公开题：48 易 + 72 原版 + 111 难），同一套题分别跑本地 Laya 与真实 Jev，比较准确率、多数类底线、Brier、ECE 四个维度——并强调评测纪律：无重试、无回退、不修补答案、预算记账。

运行环境：benchmark/requirements.txt；题目集由 scripts/fetch_public_tasks.sh + scripts/convert_jevbench.py 生成 tasks/public_all.jsonl（本仓库已含转换好的版本）。无 Laya 服务 / 无 Key 时，分析部分使用仓库自带的演示产物照样可读。

Laya vs Jev — JevBench 评测 (本地笔记本)

工作目录：10_本地模型/benchmark/ 题目集：JevBench v1.1 + v1.2 公开题（共 231 道 = easy 48 + original 72 + hard 111） 模型：Laya multilingual（322M，本机 mps）+ Jev 1.13.0（TypeSafe API）

本笔记本端到端走通：

1. 检测环境 + 加载 .env
2. 校验 Laya serve /healthz
3. 加载 231 道题，做数据集 hash + family 分布
4. mock sanity baseline（无 key 也跑得通）
5. multi-run：Laya-local + Jev 并发跑完 231 题
6. summarize：每个 runner 算 accuracy / brier / ECE / p50 / cost / JevBench composite
7. compare：写 compare.json + compare.md，matplotlib 画两张图
8. 结论：4 维对比表 + 排名解读 + 失败用例抽样
9. cleanup：kill Laya serve 进程

注：本机硬件（Apple MPS）跑 multilingual checkpoint，速度与 JevBench 上游 Hetzner/RunPod 不可直接对比。 详见 cell 8 末段。

import json, os, sys, subprocess, time, statistics, hashlib, shutil
from pathlib import Path

# 通用解析：从当前工作目录向上找 laya/benchmark（兼容仓库根/notebooks/ 等启动位置）
PROJ = None
for _base in [Path.cwd(), *Path.cwd().parents]:
    _c = _base / 'laya' / 'benchmark'
    if (_c / 'llm_eval').exists():
        PROJ = _c.resolve(); break
assert PROJ is not None, 'benchmark dir not found: 请在 jev-cookbook 仓库内启动 Jupyter'

sys.path.insert(0, str(PROJ))
os.chdir(PROJ)

# Load .env into environment (no echo of secrets)
env_file = PROJ / '.env'
if env_file.exists():
    for line in env_file.read_text().splitlines():
        line = line.strip()
        if not line or line.startswith('#') or '=' not in line:
            continue
        k, v = line.split('=', 1)
        os.environ.setdefault(k.strip(), v.strip())

print('PROJ:', PROJ)
print('cwd:', os.getcwd())
print('python:', sys.executable)
print('TYPESAFE_API_KEY set:', bool(os.environ.get('TYPESAFE_API_KEY')))
print('venv OK:', (PROJ / 'venv' / 'bin' / 'python').exists())

1. 环境探针：Laya serve / Jev API key / 工具链

import urllib.request

# Laya serve liveness
healthz_url = 'http://127.0.0.1:8811/healthz'
try:
    with urllib.request.urlopen(healthz_url, timeout=3) as r:
        healthz = json.loads(r.read())
    print('Laya /healthz OK:', healthz)
    LAYA_ALIVE = True
except Exception as e:
    print('Laya /healthz FAILED:', repr(e))
    print('  -> run ./scripts/start_laya_serve.sh first')
    LAYA_ALIVE = False

# Laya /v1/models
try:
    with urllib.request.urlopen('http://127.0.0.1:8811/v1/models', timeout=3) as r:
        models = json.loads(r.read())
    print('Laya /v1/models:', models)
except Exception as e:
    print('Laya /v1/models failed:', repr(e))

# Jev key
TYPESAFE_KEY = os.environ.get('TYPESAFE_API_KEY', '')
print('Jev TYPESAFE_API_KEY:', 'set (length=' + str(len(TYPESAFE_KEY)) + ')' if TYPESAFE_KEY else 'MISSING')

# framework imports
from llm_eval.adapters import PROVIDERS, get_adapter
from llm_eval.task import load_tasks, dataset_hash
from llm_eval.summarize import public_export, load_results
from llm_eval.multi_config import MultiRunConfig, RunnerConfig
from llm_eval.multi_runner import run_multi
from llm_eval.compare import compare_run, write_markdown_table
print('llm_eval framework OK')
print('PROVIDERS:', sorted(PROVIDERS.keys()))
print('laya_local spec:', PROVIDERS['laya_local'])

2. 加载 231 道 JevBench 公开题

tasks = load_tasks('tasks/public_all.jsonl')
print(f'loaded {len(tasks)} tasks')
print(f'dataset_hash: {dataset_hash(tasks)[:16]}...')

from collections import Counter
families = Counter(t.family for t in tasks)
qtypes = Counter(t.question_type for t in tasks)
tiers = Counter(t.id.split('-')[0] for t in tasks)
print()
print('by tier:', dict(tiers))
print('by family (top 10):', dict(sorted(families.items(), key=lambda kv: -kv[1])[:10]))
print('by question_type:', dict(qtypes))

print()
print('--- example easy task ---')
for t in tasks:
    if t.id == 'easy-intent-00':
        print(json.dumps({
            'id': t.id, 'family': t.family, 'type': t.question_type,
            'instructions': t.instructions,
            'state': t.state,
            'labels': t.labels,
            'expected': t.expected,
        }, indent=2, ensure_ascii=False)[:600])
        break

print()
print('--- example hard task ---')
for t in tasks:
    if 'hard' in t.id:
        print(json.dumps({
            'id': t.id, 'family': t.family, 'type': t.question_type,
            'instructions': t.instructions[:100] + '...',
            'state': str(t.state)[:100] + '...',
            'labels': t.labels, 'expected': t.expected,
        }, indent=2, ensure_ascii=False))
        break

3. Mock sanity baseline（零成本）

from llm_eval.runner import run as runner_run

# Pick one task per question type
sample_ids = {
    'choice': next(t.id for t in tasks if t.question_type == 'choice'),
    'noul':   next(t.id for t in tasks if t.question_type == 'noul'),
    'score':  next(t.id for t in tasks if t.question_type == 'score'),
}
smoke_tasks = [t for t in tasks if t.id in sample_ids.values()]
print(f'smoke tasks: {[t.id for t in smoke_tasks]}')

mock = get_adapter(name='mock', model='perfect')
results = runner_run(
    tasks=smoke_tasks,
    adapter=mock,
    ledger_path='runs/smoke-mock/ledger.jsonl',
    cap_usd=1.0,
    out_path='runs/smoke-mock/results.jsonl',
    raw_dir='runs/smoke-mock/raw',
    price_in_per_m=0.0,
    price_out_per_m=0.0,
    delay_s=0.0,
    runner_name='mock-perfect',
    verbose=True,
)
print('mock smoke (perfect) summary:')
summary = public_export(results, smoke_tasks, run_meta={'runner': 'mock-perfect'})
print(json.dumps(summary, indent=2, ensure_ascii=False, default=str)[:500])

4. Mock 跑全量 231 题 → 看 floor

mock_uniform = get_adapter(name='mock', model='uniform-perfect')
results = runner_run(
    tasks=tasks,
    adapter=mock_uniform,
    ledger_path='runs/smoke-mock-uniform/ledger.jsonl',
    cap_usd=1.0,
    out_path='runs/smoke-mock-uniform/results.jsonl',
    raw_dir='runs/smoke-mock-uniform/raw',
    price_in_per_m=0.0,
    price_out_per_m=0.0,
    delay_s=0.0,
    runner_name='mock-uniform',
    verbose=True,
)
print('mock-uniform full summary:')
summary = public_export(results, tasks, run_meta={'runner': 'mock-uniform'})
print(json.dumps(summary, indent=2, ensure_ascii=False, default=str))

5. multi-run：Laya-local + Jev 并发跑全量

跑完输出：

• runs/notebook-demo/_tmp/laya-vs-jev-publicall/<slug>/{results.jsonl, summary.json, ledger.jsonl, raw/}
• runs/notebook-demo/_tmp/laya-vs-jev-publicall/_manifest.json

跑时把临时目录移到 real/ 终点。

cfg = MultiRunConfig(
    run_id='laya-vs-jev-publicall',
    tasks_path='tasks/public_all.jsonl',
    cap_usd=2.0,
    delay_s=0.2,
    out_root='runs/notebook-demo/_tmp',
    runners=[
        RunnerConfig(
            name='laya-local', adapter='laya_local',
            model='laya-multilingual', key_env='',
            price_in_per_m=None, price_out_per_m=None,
            skip_if_done=True,
        ),
        RunnerConfig(
            name='jev', adapter='typesafe',
            model='jev-latest', key_env='TYPESAFE_API_KEY',
            price_in_per_m=0.042, price_out_per_m=0.0,
            skip_if_done=True,
        ),
    ],
)

print('runners:')
for r in cfg.runners:
    print(f'  - {r.name}: adapter={r.adapter} model={r.model} '
          f'key_env={r.key_env!r} available={r.is_available()}')
print(f'tasks: {len(load_tasks(cfg.tasks_path))}')
print(f'cap_usd: {cfg.cap_usd}')

if not LAYA_ALIVE:
    print()
    print('!!! LAYA_ALIVE=False - multi-run will mark all laya-local tasks as error')
    print('    Start ./scripts/start_laya_serve.sh and re-run this cell')

manifest = run_multi(cfg)

# Move _tmp/// -> real//
src_root = Path('runs/notebook-demo/_tmp') / cfg.run_id
dst_root = Path('runs/notebook-demo/real')
for entry in manifest['runners']:
    src = Path(entry['dir'])
    dst = dst_root / entry['name']
    dst.mkdir(parents=True, exist_ok=True)
    if src.exists():
        for fn in ('results.jsonl', 'summary.json', 'ledger.jsonl'):
            s = src / fn
            if s.exists():
                shutil.copy(s, dst / fn)
        raw = src / 'raw'
        if raw.is_dir():
            shutil.copytree(raw, dst / 'raw', dirs_exist_ok=True)
print()
print('=== manifest ===')
print(json.dumps(manifest, indent=2, ensure_ascii=False, default=str)[:1500])

6. Summarize：每个 runner 的 metrics + JevBench composite

summaries = {}
for name in ('laya-local', 'jev'):
    res_path = Path(f'runs/notebook-demo/real/{name}/results.jsonl')
    if not res_path.exists():
        print(f'{name}: no results.jsonl')
        continue
    results = load_results(str(res_path))
    summary = public_export(results, tasks, run_meta={'runner': name})
    out = Path(f'runs/notebook-demo/real/{name}/summary.json')
    out.write_text(json.dumps(summary, indent=2, ensure_ascii=False, default=str))
    summaries[name] = summary
    print(f'--- {name} ---')
    print(json.dumps(summary, indent=2, ensure_ascii=False, default=str))
    print()

7. 对比 + 4 维图

import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt

multi_dir = Path('runs/notebook-demo/multi')
multi_dir.mkdir(parents=True, exist_ok=True)

src = Path('runs/notebook-demo/real')
tmp = multi_dir / '_compare_src'
tmp.mkdir(exist_ok=True)
for name in ('laya-local', 'jev'):
    s = src / name
    if not (s / 'results.jsonl').exists():
        continue
    d = tmp / name
    d.mkdir(exist_ok=True)
    shutil.copy(s / 'results.jsonl', d / 'results.jsonl')
    if (s / 'summary.json').exists():
        shutil.copy(s / 'summary.json', d / 'summary.json')

cmp_dict = compare_run(str(tmp))
(multi_dir / 'compare.json').write_text(
    json.dumps(cmp_dict, indent=2, ensure_ascii=False, default=str)
)
write_markdown_table(cmp_dict, str(multi_dir / 'compare.md'))
print('compare.json written')
print('compare.md:')
print((multi_dir / 'compare.md').read_text()[:1500])

chart_dir = multi_dir / 'charts'
chart_dir.mkdir(exist_ok=True)

names = [n for n in ('laya-local', 'jev') if n in summaries]
scores = [summaries[n].get('jevbench_score', {}).get('score') for n in names]

fig, ax = plt.subplots(figsize=(8, 4))
xs = [s if s is not None else 0 for s in scores]
bars = ax.barh(names, xs, color=['#1f77b4', '#d62728'])
ax.set_xlabel('JevBench Score (0-100)')
ax.set_title('Laya vs Jev - JevBench composite (231 tasks)')
ax.set_xlim(0, 100)
for b, v in zip(bars, scores):
    ax.text(b.get_width() + 1, b.get_y() + b.get_height()/2,
            f'{v:.1f}' if v is not None else 'n/a', va='center')
fig.tight_layout()
fig.savefig(chart_dir / 'main_score.png', dpi=120)
plt.close(fig)
print('saved main_score.png')

axes_names = ['intelligence', 'calibration', 'speed', 'cost']
data = {n: [summaries[n].get('jevbench_score', {}).get('axes', {}).get(a) for a in axes_names] for n in names}

fig, axes = plt.subplots(1, 4, figsize=(14, 4), sharey=True)
x = list(range(len(names)))
width = 0.6
for i, ax_name in enumerate(axes_names):
    vals = [data[n][i] for n in names]
    vals_plot = [v if v is not None else 0 for v in vals]
    axes[i].bar(x, vals_plot, width, color=['#1f77b4', '#d62728'])
    axes[i].set_title(ax_name)
    axes[i].set_xticks(x)
    axes[i].set_xticklabels(names, rotation=15)
    axes[i].set_ylim(0, 100)
    for j, v in enumerate(vals):
        axes[i].text(j, vals_plot[j] + 2, f'{v:.0f}' if v is not None else 'n/a',
                     ha='center', fontsize=9)
fig.suptitle('Laya vs Jev - 4 axes (0-100)')
fig.tight_layout()
fig.savefig(chart_dir / '4dim_compare.png', dpi=120)
plt.close(fig)
print('saved 4dim_compare.png')

print()
print('charts:')
for p in chart_dir.glob('*.png'):
    print(f'  - {p}')

8. 结论 / 4 维排名解读

print('=' * 70)
print('  Laya vs Jev - JevBench v1.1+1.2 public_all (231 tasks)')
print('=' * 70)
print()

print('--- accuracy / calibration / latency ---')
print(f'{"runner":<14} {"acc":>7} {"maj_acc":>8} {"brier":>7} {"ece":>6} '
      f'{"p50_s":>7} {"p95_s":>7} {"$ / 1k":>9} {"tokens_in":>10}')
for n in names:
    s = summaries[n]
    jb = s.get('jevbench_score', {})
    cost_per_1k = jb.get('cost_usd_per_1k')
    print(f'{n:<14} '
          f'{s.get("accuracy", 0) or 0:>7.3f} '
          f'{s.get("majority_class_accuracy", 0) or 0:>8.3f} '
          f'{(s.get("brier") or 0):>7.3f} '
          f'{(s.get("ece") or 0):>6.3f} '
          f'{(s.get("p50_s") or 0):>7.3f} '
          f'{(s.get("p95_s") or 0):>7.3f} '
          f'{(cost_per_1k or 0):>9.4f} '
          f'{(s.get("run_meta", {}).get("tokens_in") or 0):>10}')

print()
print('--- JevBench composite (4 axes, geometric mean) ---')
print(f'{"runner":<14} {"intel":>7} {"calib":>7} {"speed":>7} {"cost":>7} '
      f'{"composite":>10}')
for n in names:
    jb = summaries[n].get('jevbench_score', {})
    a = jb.get('axes', {})
    print(f'{n:<14} '
          f'{(a.get("intelligence") or 0):>7.1f} '
          f'{(a.get("calibration") or 0):>7.1f} '
          f'{(a.get("speed") or 0):>7.1f} '
          f'{(a.get("cost") or 0):>7.1f} '
          f'{(jb.get("score") or 0):>10.2f}')

print()
print('--- text verdict ---')

def pct(x): return f'{x*100:.1f}%' if x is not None else 'n/a'
def ms(x): return f'{x*1000:.0f} ms' if x is not None else 'n/a'

for n in names:
    s = summaries[n]
    jb = s.get('jevbench_score', {})
    a = jb.get('axes', {})
    print()
    print('**' + n + '**')
    print('  - accuracy = ' + pct(s.get('accuracy')) +
          ' (vs majority-class floor ' + pct(s.get('majority_class_accuracy')) + ')')
    print(f'  - brier = {s.get("brier"):.3f}, ECE = {s.get("ece"):.3f}')
    print('  - latency p50 = ' + ms(s.get('p50_s')) + ', p95 = ' + ms(s.get('p95_s')))
    print(f'  - cost = ${jb.get("cost_usd_per_1k") or 0:.4f} / 1k decisions')
    print(f'  - JevBench Score = {jb.get("score"):.2f}')
    if jb.get('intelligence_penalty_applied'):
        print('  - low-Intelligence penalty APPLIED (Intelligence < 50)')

print()
print('--- evidence boundaries ---')
print('\n1. 题集：JevBench 公开题是英文 short decision；Laya multilingual 在英文题上不一定比 english checkpoint 强（README §1）。')
print('2. 硬件：本机 Apple Silicon（MPS）+ 系统 Python 3.14 + torch 2.13。JevBench 上游在 Hetzner 服务器/RunPod GPU 上测量。p50_s 不可直接横向对比。')
print('3. Cost 列：Laya 是自托管（cost = $0，Speed = 100），Jev 是 API（$0.04/1k decisions）。Laya 在 Cost 这一轴天然占优；想看 raw 数字请看 accuracy / latency 而非 composite。')
print('4. JevBench v1.2 vs v1.4：上游在 v1.4 引入密封题 + 4 维调和均值；本评测沿用 v1.2/v1.3 公式（4 维几何平均 + low-Intelligence penalty），与上游公开数字口径不完全一致。')

9. 失败用例抽样 + cleanup

for n in names:
    res_path = Path(f'runs/notebook-demo/real/{n}/results.jsonl')
    if not res_path.exists():
        continue
    results = load_results(str(res_path))
    wrongs = [r for r in results if not r.get('correct_value')]
    print(f'--- {n}: {len(wrongs)} wrong of {len(results)} ---')
    for r in wrongs[:3]:
        tid = r['task_id']
        expected = next(t.expected for t in tasks if t.id == tid)
        state = str(next(t.state for t in tasks if t.id == tid))[:80]
        top = r.get('probs', {})
        top_label = max(top, key=top.get) if top else '-'
        top_prob = top.get(top_label, 0) if top else 0
        print(f'  [{tid}] expected={expected!r} got={top_label!r} '
              f'p={top_prob:.3f} state={state!r}')
    print()

print('=== cleanup ===')
pid_file = Path('runs/laya_serve.pid')
if pid_file.exists():
    pid = int(pid_file.read_text().strip())
    try:
        os.kill(pid, 15)
        print(f'Laya serve (pid={pid}) SIGTERM sent')
        for _ in range(10):
            try:
                os.kill(pid, 0)
            except ProcessLookupError:
                print('Laya serve stopped')
                pid_file.unlink()
                break
            time.sleep(0.5)
        else:
            os.kill(pid, 9)
            print('Laya serve SIGKILL sent')
            pid_file.unlink()
    except ProcessLookupError:
        print(f'pid={pid} already gone')
        pid_file.unlink()
else:
    print('no pid file at runs/laya_serve.pid - nothing to stop')