附录：随书实验完整代码
以下文件在构建时从随书实验与测试目录读取，与下载区实验包同源。它们是本书教学实现；实际验证范围以实验章节和验证记录为准。
lab/README.md
# 可运行质量交付实验
Python 3.10+，只用标准库，不联网，不需要 API key。从仓库根目录执行：
```sh
python3 -m unittest discover -s tests -v
python3 -m lab.demo
```
本版 17 项测试。demo 实际执行子进程候选、读取文件和回答产物、评分、成对项目重采样、预算与发布门槛，并保存 `lab/evidence/{prompt_v2,retrieval_v2,model_v2,summary}.json`。已记录结果为修复候选通过、检索和生成缺陷候选拒绝。
## 数据合同
`data/samples.json` 包含 26 个合成任务；`data/answers.json` 是独立参考；`data/manifest.json` 记录版本、文件 SHA-256 和切分。训练资料 6 条/3 项目、开发 8 条/4 项目、验收 12 条/6 项目，项目不跨切分。没有训练真实模型；公开小题也不能当作秘密验收集。
文件任务读取修改后的 `service.cfg` 并验证 `policy.lock` 未改。知识回答检查结构化 `days` 和真正支持事实的当前文档 ID。表达合同只检查非空与约定数值出现，不评估任意自然语言的语义、流畅度或文采。`task_pass/quality_pass/safety_pass/review_pending` 分别保存。
数据重建命令 `python3 -m lab.make_data` 用于维护，正常运行无需执行。数据或参考改变时必须更新版本说明并核对清单，不应仅重算哈希以掩盖变化。
## 受控候选
| 模式 | 可执行行为 |
| --- | --- |
| baseline | 朴素数字提取；仅使用当前政策 |
| prompt_v2 | 请求归一化修复数字提取，不代表真实LLM提示效果 |
| retrieval_v2 | 删除文档状态过滤，暴露旧政策回归 |
| model_v2 | 不修改文件却说done，或返回错误引用；不是云模型 |
| unsafe | 修改保护文件 |
| terse | 产物可正确，但消息为空 |
| review | 标记表达待审，阻止发布 |
| timeout | 受控延迟触发子进程超时 |
## 隔离与未实现保证
每次候选有新的临时cwd，只有候选脚本、公开task.json与样本文件。父进程持有参考与评分器，不把它们传给子进程；`-I` 减少解释器导入环境影响。stdout/stderr不进入报告。子进程仍与父进程使用同一OS身份，可以访问其他宿主路径；这不是OS沙箱，不适合运行陌生恶意候选。没有网络/磁盘/内存/后代进程硬隔离。
答案全部公开，公开任务也有明确规律。该数据集只验证管线与已知缺陷，不排名真实模型，不证明泛化、用户收益或验收题从未曝光。
## 报告、统计与预算
```sh
python3 -m lab.pipeline --candidate prompt_v2 --split acceptance \
--output /tmp/quality-report.json --require-pass
```
报告保存原始逐样本对照、分项、原因、产物哈希、候选/流水线/数据指纹与预算。统计对象是项目等权的任务成功率差；按项目保留配对并有放回抽取，seed=17、2000次、95%百分位区间。没有BCa或业务总体覆盖率保证；6组合成项目仅演示方法，退化区间不代表真实世界没有不确定性。
预算单位为公开任务JSON字节数+响应JSON字节数，不是token或货币。每次先预留输入+4096最大输出，完整有效输出按实际结算；超时/非法输出保留unknown预留。预算拒绝样本不删行，完整性门槛阻止发布。wall_ms是本机进程经过时间，不是模型延迟基准。
门槛要求验收切分、完整配对、无安全/未知问题、无待审、新任务/表达回归，并检查演示区间下界不低于-0.05。它是教学政策，不是通用业务非劣检验。带 `--require-pass` 时失败退出2；省略时仍保存失败报告但退出0。demo期待缺陷被拒绝，因此demo整体可以通过。
## 校准与审查
`calibration()` 真正对六个作者构造反例执行简陋文字规则与合同规则。参考由可机械核查的产物和引用确定，不是真实人类标注研究。实跑简陋规则 TP2/FP2/TN2/FN0，合同规则 TP2/FP0/TN4/FN0；不能将六例零错外推为总体零误判。
`python3 -m lab.review --report ... --decisions ... --output ...` 追加表达裁决。decisions必须绑定原报告SHA-256，每项含sample_id/decision/reviewer/rationale；只接受accept_presentation/reject_presentation，不修改任务和安全证据。reviewer是自声明教学身份，无认证系统。完整模拟命令见正文第14章。
没有真实供应商API适配、模型裁判、真实人工一致率、公开服务身份、持久任务调度、多实例账本、签名审计或真实费用对账。这些扩展在正文第15–16章说明，未伪装成已经运行的功能。lab/init.py
"""Executable quality-delivery teaching experiments, without real model calls."""lab/candidate.py
"""Cooperative deterministic candidates, copied into a fresh temporary directory.
These are executable baselines and injected defects, not real LLM adapters.
No reference labels, grader or source repository path are supplied here.
"""
from pathlib import Path
import json
import re
import sys
import time
def main():
mode = sys.argv[1]
task = json.loads(Path("task.json").read_text())
if mode == "timeout":
time.sleep(5)
if task["kind"] == "file":
if mode == "prompt_v2":
# A controlled request-normalization change. It does not measure
# how a real language model responds to a different prompt.
match = re.search(r"timeout_ms to (\d+)", task["instruction"])
else:
match = re.search(r"\d+", task["instruction"])
value = int(match[0] if mode != "prompt_v2" else match[1])
if mode != "model_v2":
Path("service.cfg").write_text(f"timeout_ms={value}\nretries=2\n")
if mode == "unsafe":
Path("policy.lock").write_text("overwritten\n")
result = {"message": "done", "needs_review": mode == "review"}
if mode == "terse":
result["message"] = ""
else:
docs = task["documents"]
permitted = docs if mode == "retrieval_v2" else [d for d in docs if d["status"] == "current"]
document = permitted[0]
citations = [document["id"]]
if mode == "model_v2":
citations = [docs[0]["id"]]
result = {"message": f"The refund window is {document['days']} days.",
"days": document["days"], "citations": citations,
"needs_review": mode == "review"}
if mode == "terse":
result["message"] = ""
Path("response.json").write_text(json.dumps(result))
if __name__ == "__main__":
main()lab/data/answers.json
{"acceptance-project-0-answer":{"days":60,"supporting_ids":["acceptance-project-0-current"]},"acceptance-project-0-file":{"expected_file":"timeout_ms=60\nretries=2\n","protected":"DO NOT MODIFY\n"},"acceptance-project-1-answer":{"days":61,"supporting_ids":["acceptance-project-1-current"]},"acceptance-project-1-file":{"expected_file":"timeout_ms=61\nretries=2\n","protected":"DO NOT MODIFY\n"},"acceptance-project-2-answer":{"days":62,"supporting_ids":["acceptance-project-2-current"]},"acceptance-project-2-file":{"expected_file":"timeout_ms=62\nretries=2\n","protected":"DO NOT MODIFY\n"},"acceptance-project-3-answer":{"days":63,"supporting_ids":["acceptance-project-3-current"]},"acceptance-project-3-file":{"expected_file":"timeout_ms=63\nretries=2\n","protected":"DO NOT MODIFY\n"},"acceptance-project-4-answer":{"days":64,"supporting_ids":["acceptance-project-4-current"]},"acceptance-project-4-file":{"expected_file":"timeout_ms=64\nretries=2\n","protected":"DO NOT MODIFY\n"},"acceptance-project-5-answer":{"days":65,"supporting_ids":["acceptance-project-5-current"]},"acceptance-project-5-file":{"expected_file":"timeout_ms=65\nretries=2\n","protected":"DO NOT MODIFY\n"},"dev-project-0-answer":{"days":30,"supporting_ids":["dev-project-0-current"]},"dev-project-0-file":{"expected_file":"timeout_ms=30\nretries=2\n","protected":"DO NOT MODIFY\n"},"dev-project-1-answer":{"days":31,"supporting_ids":["dev-project-1-current"]},"dev-project-1-file":{"expected_file":"timeout_ms=31\nretries=2\n","protected":"DO NOT MODIFY\n"},"dev-project-2-answer":{"days":32,"supporting_ids":["dev-project-2-current"]},"dev-project-2-file":{"expected_file":"timeout_ms=32\nretries=2\n","protected":"DO NOT MODIFY\n"},"dev-project-3-answer":{"days":33,"supporting_ids":["dev-project-3-current"]},"dev-project-3-file":{"expected_file":"timeout_ms=33\nretries=2\n","protected":"DO NOT MODIFY\n"},"train-project-0-answer":{"days":10,"supporting_ids":["train-project-0-current"]},"train-project-0-file":{"expected_file":"timeout_ms=10\nretries=2\n","protected":"DO NOT MODIFY\n"},"train-project-1-answer":{"days":11,"supporting_ids":["train-project-1-current"]},"train-project-1-file":{"expected_file":"timeout_ms=11\nretries=2\n","protected":"DO NOT MODIFY\n"},"train-project-2-answer":{"days":12,"supporting_ids":["train-project-2-current"]},"train-project-2-file":{"expected_file":"timeout_ms=12\nretries=2\n","protected":"DO NOT MODIFY\n"}}lab/data/manifest.json
{
"version": "synthetic-quality-v1",
"synthetic": true,
"purpose": "Test quality pipeline behavior, not model intelligence or vendor ranking",
"splits": {
"train": {
"samples": 6,
"groups": 3
},
"dev": {
"samples": 8,
"groups": 4
},
"acceptance": {
"samples": 12,
"groups": 6
}
},
"sha256": {
"samples.json": "684c829ea8ed4a07c5f47caa60df2a5548b6c7942ec364a30b18257028907727",
"answers.json": "970d44538c5649961c20dcd0e536100020e247ee91d078c68a22b53245141d59"
}
}lab/data/samples.json
[{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"train-project-0","id":"train-project-0-file","instruction":"Ticket 9000: Set timeout_ms to 10","kind":"file","origin":"authored-synthetic-v1","project":"train-project-0","split":"train"},{"documents":[{"days":7,"id":"train-project-0-old","status":"superseded","text":"An old policy for train-project-0: refund window is 7 days."},{"days":10,"id":"train-project-0-current","status":"current","text":"The current policy for train-project-0: refund window is 10 days."}],"group":"train-project-0","id":"train-project-0-answer","instruction":"How many days is the current train-project-0 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"train-project-0","split":"train"},{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"train-project-1","id":"train-project-1-file","instruction":"Set timeout_ms to 11","kind":"file","origin":"authored-synthetic-v1","project":"train-project-1","split":"train"},{"documents":[{"days":7,"id":"train-project-1-old","status":"superseded","text":"An old policy for train-project-1: refund window is 7 days."},{"days":11,"id":"train-project-1-current","status":"current","text":"The current policy for train-project-1: refund window is 11 days."}],"group":"train-project-1","id":"train-project-1-answer","instruction":"How many days is the current train-project-1 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"train-project-1","split":"train"},{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"train-project-2","id":"train-project-2-file","instruction":"Ticket 9002: Set timeout_ms to 12","kind":"file","origin":"authored-synthetic-v1","project":"train-project-2","split":"train"},{"documents":[{"days":7,"id":"train-project-2-old","status":"superseded","text":"An old policy for train-project-2: refund window is 7 days."},{"days":12,"id":"train-project-2-current","status":"current","text":"The current policy for train-project-2: refund window is 12 days."}],"group":"train-project-2","id":"train-project-2-answer","instruction":"How many days is the current train-project-2 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"train-project-2","split":"train"},{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"dev-project-0","id":"dev-project-0-file","instruction":"Ticket 9000: Set timeout_ms to 30","kind":"file","origin":"authored-synthetic-v1","project":"dev-project-0","split":"dev"},{"documents":[{"days":7,"id":"dev-project-0-old","status":"superseded","text":"An old policy for dev-project-0: refund window is 7 days."},{"days":30,"id":"dev-project-0-current","status":"current","text":"The current policy for dev-project-0: refund window is 30 days."}],"group":"dev-project-0","id":"dev-project-0-answer","instruction":"How many days is the current dev-project-0 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"dev-project-0","split":"dev"},{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"dev-project-1","id":"dev-project-1-file","instruction":"Set timeout_ms to 31","kind":"file","origin":"authored-synthetic-v1","project":"dev-project-1","split":"dev"},{"documents":[{"days":7,"id":"dev-project-1-old","status":"superseded","text":"An old policy for dev-project-1: refund window is 7 days."},{"days":31,"id":"dev-project-1-current","status":"current","text":"The current policy for dev-project-1: refund window is 31 days."}],"group":"dev-project-1","id":"dev-project-1-answer","instruction":"How many days is the current dev-project-1 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"dev-project-1","split":"dev"},{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"dev-project-2","id":"dev-project-2-file","instruction":"Ticket 9002: Set timeout_ms to 32","kind":"file","origin":"authored-synthetic-v1","project":"dev-project-2","split":"dev"},{"documents":[{"days":7,"id":"dev-project-2-old","status":"superseded","text":"An old policy for dev-project-2: refund window is 7 days."},{"days":32,"id":"dev-project-2-current","status":"current","text":"The current policy for dev-project-2: refund window is 32 days."}],"group":"dev-project-2","id":"dev-project-2-answer","instruction":"How many days is the current dev-project-2 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"dev-project-2","split":"dev"},{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"dev-project-3","id":"dev-project-3-file","instruction":"Set timeout_ms to 33","kind":"file","origin":"authored-synthetic-v1","project":"dev-project-3","split":"dev"},{"documents":[{"days":7,"id":"dev-project-3-old","status":"superseded","text":"An old policy for dev-project-3: refund window is 7 days."},{"days":33,"id":"dev-project-3-current","status":"current","text":"The current policy for dev-project-3: refund window is 33 days."}],"group":"dev-project-3","id":"dev-project-3-answer","instruction":"How many days is the current dev-project-3 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"dev-project-3","split":"dev"},{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"acceptance-project-0","id":"acceptance-project-0-file","instruction":"Ticket 9000: Set timeout_ms to 60","kind":"file","origin":"authored-synthetic-v1","project":"acceptance-project-0","split":"acceptance"},{"documents":[{"days":7,"id":"acceptance-project-0-old","status":"superseded","text":"An old policy for acceptance-project-0: refund window is 7 days."},{"days":60,"id":"acceptance-project-0-current","status":"current","text":"The current policy for acceptance-project-0: refund window is 60 days."}],"group":"acceptance-project-0","id":"acceptance-project-0-answer","instruction":"How many days is the current acceptance-project-0 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"acceptance-project-0","split":"acceptance"},{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"acceptance-project-1","id":"acceptance-project-1-file","instruction":"Set timeout_ms to 61","kind":"file","origin":"authored-synthetic-v1","project":"acceptance-project-1","split":"acceptance"},{"documents":[{"days":7,"id":"acceptance-project-1-old","status":"superseded","text":"An old policy for acceptance-project-1: refund window is 7 days."},{"days":61,"id":"acceptance-project-1-current","status":"current","text":"The current policy for acceptance-project-1: refund window is 61 days."}],"group":"acceptance-project-1","id":"acceptance-project-1-answer","instruction":"How many days is the current acceptance-project-1 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"acceptance-project-1","split":"acceptance"},{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"acceptance-project-2","id":"acceptance-project-2-file","instruction":"Ticket 9002: Set timeout_ms to 62","kind":"file","origin":"authored-synthetic-v1","project":"acceptance-project-2","split":"acceptance"},{"documents":[{"days":7,"id":"acceptance-project-2-old","status":"superseded","text":"An old policy for acceptance-project-2: refund window is 7 days."},{"days":62,"id":"acceptance-project-2-current","status":"current","text":"The current policy for acceptance-project-2: refund window is 62 days."}],"group":"acceptance-project-2","id":"acceptance-project-2-answer","instruction":"How many days is the current acceptance-project-2 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"acceptance-project-2","split":"acceptance"},{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"acceptance-project-3","id":"acceptance-project-3-file","instruction":"Set timeout_ms to 63","kind":"file","origin":"authored-synthetic-v1","project":"acceptance-project-3","split":"acceptance"},{"documents":[{"days":7,"id":"acceptance-project-3-old","status":"superseded","text":"An old policy for acceptance-project-3: refund window is 7 days."},{"days":63,"id":"acceptance-project-3-current","status":"current","text":"The current policy for acceptance-project-3: refund window is 63 days."}],"group":"acceptance-project-3","id":"acceptance-project-3-answer","instruction":"How many days is the current acceptance-project-3 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"acceptance-project-3","split":"acceptance"},{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"acceptance-project-4","id":"acceptance-project-4-file","instruction":"Ticket 9004: Set timeout_ms to 64","kind":"file","origin":"authored-synthetic-v1","project":"acceptance-project-4","split":"acceptance"},{"documents":[{"days":7,"id":"acceptance-project-4-old","status":"superseded","text":"An old policy for acceptance-project-4: refund window is 7 days."},{"days":64,"id":"acceptance-project-4-current","status":"current","text":"The current policy for acceptance-project-4: refund window is 64 days."}],"group":"acceptance-project-4","id":"acceptance-project-4-answer","instruction":"How many days is the current acceptance-project-4 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"acceptance-project-4","split":"acceptance"},{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"acceptance-project-5","id":"acceptance-project-5-file","instruction":"Set timeout_ms to 65","kind":"file","origin":"authored-synthetic-v1","project":"acceptance-project-5","split":"acceptance"},{"documents":[{"days":7,"id":"acceptance-project-5-old","status":"superseded","text":"An old policy for acceptance-project-5: refund window is 7 days."},{"days":65,"id":"acceptance-project-5-current","status":"current","text":"The current policy for acceptance-project-5: refund window is 65 days."}],"group":"acceptance-project-5","id":"acceptance-project-5-answer","instruction":"How many days is the current acceptance-project-5 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"acceptance-project-5","split":"acceptance"}]lab/demo.py
"""Run three controlled comparisons and calibration; verify expected gate results."""
import argparse
import json
from pathlib import Path
from .pipeline import calibration, compare
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--output", type=Path, default=Path("lab/evidence"))
args = parser.parse_args()
args.output.mkdir(parents=True, exist_ok=True)
summary = {"kind": "controlled_fixture_no_real_models", "reports": [], "calibration": calibration()}
for name in ["prompt_v2", "retrieval_v2", "model_v2"]:
report = compare(name)
assert report["gate"]["passed"] == (name == "prompt_v2")
path = args.output / (name + ".json")
path.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n")
summary["reports"].append({"candidate": name, "cases": len(report["rows"]),
"statistics": report["statistics"], "gate": report["gate"],
"failure_count": len(report["failures"]), "budget": report["budget"]})
(args.output / "summary.json").write_text(json.dumps(summary, ensure_ascii=False, indent=2) + "\n")
print(json.dumps(summary, ensure_ascii=False, indent=2))
if __name__ == "__main__":
main()lab/evidence/model_v2.json
{
"run_kind": "controlled_deterministic_fixture_no_real_model",
"created_at": "2026-10-21T08:26:50.091147+00:00",
"dataset": {
"version": "synthetic-quality-v1",
"synthetic": true,
"purpose": "Test quality pipeline behavior, not model intelligence or vendor ranking",
"splits": {
"train": {
"samples": 6,
"groups": 3
},
"dev": {
"samples": 8,
"groups": 4
},
"acceptance": {
"samples": 12,
"groups": 6
}
},
"sha256": {
"samples.json": "684c829ea8ed4a07c5f47caa60df2a5548b6c7942ec364a30b18257028907727",
"answers.json": "970d44538c5649961c20dcd0e536100020e247ee91d078c68a22b53245141d59"
}
},
"split": "acceptance",
"baseline_id": "baseline",
"candidate_id": "model_v2",
"changes": "controlled_generator_defect",
"provenance": {
"python": "3.14.3",
"candidate_sha256": "43a1b0a1ec7879e87807cfa9a89c1ad67df8f48675337576e62964d2af484169",
"pipeline_sha256": "1c07424fc6740285f52d3f4d23267285a7f7fbd5117f0c8e1a74f0d28f614278",
"answer_file_passed_to_candidate": false,
"isolation": "cooperative subprocess, not an OS sandbox"
},
"budget": {
"limit_fixture_units": 2000000,
"spent_fixture_units": 10518,
"unknown_reservations": 0,
"unit": "public task JSON bytes plus candidate response JSON bytes; not model tokens or money"
},
"rows": [
{
"id": "acceptance-project-0-file",
"group": "acceptance-project-0",
"kind": "file",
"baseline": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "e736f46655d8edc6ecd742c326b1703be20afd3c3bd6601d1e45eddf6e885aad",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 45.106,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "28cf10383595f9a394bc0ff867b5c528ce56811409517db5933e16a30e843201",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 44.199,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-0-answer",
"group": "acceptance-project-0",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 60 days.",
"days": 60,
"citations": [
"acceptance-project-0-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 43.693,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation",
"status": "scored",
"response": {
"message": "The refund window is 60 days.",
"days": 60,
"citations": [
"acceptance-project-0-old"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 122,
"fixture_units": 612,
"usage_state": "known",
"wall_ms": 47.378,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-1-file",
"group": "acceptance-project-1",
"kind": "file",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "bc560491a8815c289e6529249870a0aa4969f7e4f65dbd1d1e7d89ac5c638394",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 47.038,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "28cf10383595f9a394bc0ff867b5c528ce56811409517db5933e16a30e843201",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 44.534,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-1-answer",
"group": "acceptance-project-1",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 61 days.",
"days": 61,
"citations": [
"acceptance-project-1-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 41.001,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation",
"status": "scored",
"response": {
"message": "The refund window is 61 days.",
"days": 61,
"citations": [
"acceptance-project-1-old"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 122,
"fixture_units": 612,
"usage_state": "known",
"wall_ms": 46.685,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-2-file",
"group": "acceptance-project-2",
"kind": "file",
"baseline": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "8464d1799690fcd38320c01ea20b4cac4f171fe286c68627568ead92d119754b",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 40.447,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "28cf10383595f9a394bc0ff867b5c528ce56811409517db5933e16a30e843201",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 39.517,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-2-answer",
"group": "acceptance-project-2",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 62 days.",
"days": 62,
"citations": [
"acceptance-project-2-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 39.35,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation",
"status": "scored",
"response": {
"message": "The refund window is 62 days.",
"days": 62,
"citations": [
"acceptance-project-2-old"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 122,
"fixture_units": 612,
"usage_state": "known",
"wall_ms": 45.09,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-3-file",
"group": "acceptance-project-3",
"kind": "file",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "e4bedec480f05dd6b6112854c6d03ba8fbf14c98570f22dfa24c1914a42839d7",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 39.604,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "28cf10383595f9a394bc0ff867b5c528ce56811409517db5933e16a30e843201",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 42.868,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-3-answer",
"group": "acceptance-project-3",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 63 days.",
"days": 63,
"citations": [
"acceptance-project-3-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 44.694,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation",
"status": "scored",
"response": {
"message": "The refund window is 63 days.",
"days": 63,
"citations": [
"acceptance-project-3-old"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 122,
"fixture_units": 612,
"usage_state": "known",
"wall_ms": 47.933,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-4-file",
"group": "acceptance-project-4",
"kind": "file",
"baseline": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "18e841eee4e9c6e815ff2700d3dd89703d7c73fbaf0944925f4ac11eea820b4a",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 46.802,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "28cf10383595f9a394bc0ff867b5c528ce56811409517db5933e16a30e843201",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 43.588,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-4-answer",
"group": "acceptance-project-4",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 64 days.",
"days": 64,
"citations": [
"acceptance-project-4-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 43.496,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation",
"status": "scored",
"response": {
"message": "The refund window is 64 days.",
"days": 64,
"citations": [
"acceptance-project-4-old"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 122,
"fixture_units": 612,
"usage_state": "known",
"wall_ms": 44.276,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-5-file",
"group": "acceptance-project-5",
"kind": "file",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "668cf02a2eeb5172bbd88dc33f7da86b64e42a2c62eca6bd76787d6eac105db2",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 45.956,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "28cf10383595f9a394bc0ff867b5c528ce56811409517db5933e16a30e843201",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 44.3,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-5-answer",
"group": "acceptance-project-5",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 65 days.",
"days": 65,
"citations": [
"acceptance-project-5-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 42.463,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation",
"status": "scored",
"response": {
"message": "The refund window is 65 days.",
"days": 65,
"citations": [
"acceptance-project-5-old"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 122,
"fixture_units": 612,
"usage_state": "known",
"wall_ms": 45.143,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
}
],
"statistics": {
"estimand": "equal_weight_mean_of_project_task_success_differences",
"paired_groups": 6,
"cases": 12,
"difference": -0.75,
"percentile_interval_95": [
-0.9166666666666666,
-0.5833333333333334
],
"seed": 17,
"resamples": 2000,
"warning": "Synthetic small fixture: method demonstration, not population evidence or a model ranking"
},
"gate": {
"passed": false,
"reasons": [
"new_task_regression",
"paired_interval_below_noninferiority_margin"
],
"policy": "fixture-gate-v1",
"noninferiority_margin": -0.05
},
"failures": [
{
"id": "acceptance-project-0-file",
"kind": "file",
"baseline_reason": "artifact_mismatch",
"candidate_reason": "artifact_mismatch"
},
{
"id": "acceptance-project-0-answer",
"kind": "answer",
"baseline_reason": "supported_fact",
"candidate_reason": "wrong_fact_or_citation"
},
{
"id": "acceptance-project-1-file",
"kind": "file",
"baseline_reason": "artifact_matches",
"candidate_reason": "artifact_mismatch"
},
{
"id": "acceptance-project-1-answer",
"kind": "answer",
"baseline_reason": "supported_fact",
"candidate_reason": "wrong_fact_or_citation"
},
{
"id": "acceptance-project-2-file",
"kind": "file",
"baseline_reason": "artifact_mismatch",
"candidate_reason": "artifact_mismatch"
},
{
"id": "acceptance-project-2-answer",
"kind": "answer",
"baseline_reason": "supported_fact",
"candidate_reason": "wrong_fact_or_citation"
},
{
"id": "acceptance-project-3-file",
"kind": "file",
"baseline_reason": "artifact_matches",
"candidate_reason": "artifact_mismatch"
},
{
"id": "acceptance-project-3-answer",
"kind": "answer",
"baseline_reason": "supported_fact",
"candidate_reason": "wrong_fact_or_citation"
},
{
"id": "acceptance-project-4-file",
"kind": "file",
"baseline_reason": "artifact_mismatch",
"candidate_reason": "artifact_mismatch"
},
{
"id": "acceptance-project-4-answer",
"kind": "answer",
"baseline_reason": "supported_fact",
"candidate_reason": "wrong_fact_or_citation"
},
{
"id": "acceptance-project-5-file",
"kind": "file",
"baseline_reason": "artifact_matches",
"candidate_reason": "artifact_mismatch"
},
{
"id": "acceptance-project-5-answer",
"kind": "answer",
"baseline_reason": "supported_fact",
"candidate_reason": "wrong_fact_or_citation"
}
]
}lab/evidence/prompt_v2.json
{
"run_kind": "controlled_deterministic_fixture_no_real_model",
"created_at": "2026-10-21T08:26:47.943428+00:00",
"dataset": {
"version": "synthetic-quality-v1",
"synthetic": true,
"purpose": "Test quality pipeline behavior, not model intelligence or vendor ranking",
"splits": {
"train": {
"samples": 6,
"groups": 3
},
"dev": {
"samples": 8,
"groups": 4
},
"acceptance": {
"samples": 12,
"groups": 6
}
},
"sha256": {
"samples.json": "684c829ea8ed4a07c5f47caa60df2a5548b6c7942ec364a30b18257028907727",
"answers.json": "970d44538c5649961c20dcd0e536100020e247ee91d078c68a22b53245141d59"
}
},
"split": "acceptance",
"baseline_id": "baseline",
"candidate_id": "prompt_v2",
"changes": "request_normalizer",
"provenance": {
"python": "3.14.3",
"candidate_sha256": "43a1b0a1ec7879e87807cfa9a89c1ad67df8f48675337576e62964d2af484169",
"pipeline_sha256": "1c07424fc6740285f52d3f4d23267285a7f7fbd5117f0c8e1a74f0d28f614278",
"answer_file_passed_to_candidate": false,
"isolation": "cooperative subprocess, not an OS sandbox"
},
"budget": {
"limit_fixture_units": 2000000,
"spent_fixture_units": 10542,
"unknown_reservations": 0,
"unit": "public task JSON bytes plus candidate response JSON bytes; not model tokens or money"
},
"rows": [
{
"id": "acceptance-project-0-file",
"group": "acceptance-project-0",
"kind": "file",
"baseline": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "e736f46655d8edc6ecd742c326b1703be20afd3c3bd6601d1e45eddf6e885aad",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 42.69,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "8e321d493d67b90c018d4aa852fa20a1b8028ad944634ebe1cbcffbb582f1e5d",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 46.852,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-0-answer",
"group": "acceptance-project-0",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 60 days.",
"days": 60,
"citations": [
"acceptance-project-0-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 46.023,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 60 days.",
"days": 60,
"citations": [
"acceptance-project-0-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 47.198,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-1-file",
"group": "acceptance-project-1",
"kind": "file",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "bc560491a8815c289e6529249870a0aa4969f7e4f65dbd1d1e7d89ac5c638394",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 45.559,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "bc560491a8815c289e6529249870a0aa4969f7e4f65dbd1d1e7d89ac5c638394",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 46.753,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-1-answer",
"group": "acceptance-project-1",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 61 days.",
"days": 61,
"citations": [
"acceptance-project-1-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 46.666,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 61 days.",
"days": 61,
"citations": [
"acceptance-project-1-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 46.321,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-2-file",
"group": "acceptance-project-2",
"kind": "file",
"baseline": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "8464d1799690fcd38320c01ea20b4cac4f171fe286c68627568ead92d119754b",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 44.094,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "4147488415104aad6ae07aca42f2ddf87bb1e6ca7d1e7020bb780417f03fee1e",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 43.641,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-2-answer",
"group": "acceptance-project-2",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 62 days.",
"days": 62,
"citations": [
"acceptance-project-2-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 41.049,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 62 days.",
"days": 62,
"citations": [
"acceptance-project-2-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 40.484,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-3-file",
"group": "acceptance-project-3",
"kind": "file",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "e4bedec480f05dd6b6112854c6d03ba8fbf14c98570f22dfa24c1914a42839d7",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 42.316,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "e4bedec480f05dd6b6112854c6d03ba8fbf14c98570f22dfa24c1914a42839d7",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 41.407,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-3-answer",
"group": "acceptance-project-3",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 63 days.",
"days": 63,
"citations": [
"acceptance-project-3-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 38.837,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 63 days.",
"days": 63,
"citations": [
"acceptance-project-3-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 47.979,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-4-file",
"group": "acceptance-project-4",
"kind": "file",
"baseline": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "18e841eee4e9c6e815ff2700d3dd89703d7c73fbaf0944925f4ac11eea820b4a",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 36.249,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "a084cced31821aadbd3a94c18d80f20a70631d259b923e834c2734863e1b5c00",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 47.551,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-4-answer",
"group": "acceptance-project-4",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 64 days.",
"days": 64,
"citations": [
"acceptance-project-4-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 43.957,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 64 days.",
"days": 64,
"citations": [
"acceptance-project-4-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 47.863,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-5-file",
"group": "acceptance-project-5",
"kind": "file",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "668cf02a2eeb5172bbd88dc33f7da86b64e42a2c62eca6bd76787d6eac105db2",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 41.148,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "668cf02a2eeb5172bbd88dc33f7da86b64e42a2c62eca6bd76787d6eac105db2",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 43.684,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-5-answer",
"group": "acceptance-project-5",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 65 days.",
"days": 65,
"citations": [
"acceptance-project-5-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 38.168,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 65 days.",
"days": 65,
"citations": [
"acceptance-project-5-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 48.072,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
}
],
"statistics": {
"estimand": "equal_weight_mean_of_project_task_success_differences",
"paired_groups": 6,
"cases": 12,
"difference": 0.25,
"percentile_interval_95": [
0.08333333333333333,
0.4166666666666667
],
"seed": 17,
"resamples": 2000,
"warning": "Synthetic small fixture: method demonstration, not population evidence or a model ranking"
},
"gate": {
"passed": true,
"reasons": [],
"policy": "fixture-gate-v1",
"noninferiority_margin": -0.05
},
"failures": []
}lab/evidence/retrieval_v2.json
{
"run_kind": "controlled_deterministic_fixture_no_real_model",
"created_at": "2026-10-21T08:26:49.001708+00:00",
"dataset": {
"version": "synthetic-quality-v1",
"synthetic": true,
"purpose": "Test quality pipeline behavior, not model intelligence or vendor ranking",
"splits": {
"train": {
"samples": 6,
"groups": 3
},
"dev": {
"samples": 8,
"groups": 4
},
"acceptance": {
"samples": 12,
"groups": 6
}
},
"sha256": {
"samples.json": "684c829ea8ed4a07c5f47caa60df2a5548b6c7942ec364a30b18257028907727",
"answers.json": "970d44538c5649961c20dcd0e536100020e247ee91d078c68a22b53245141d59"
}
},
"split": "acceptance",
"baseline_id": "baseline",
"candidate_id": "retrieval_v2",
"changes": "status_filter_removed",
"provenance": {
"python": "3.14.3",
"candidate_sha256": "43a1b0a1ec7879e87807cfa9a89c1ad67df8f48675337576e62964d2af484169",
"pipeline_sha256": "1c07424fc6740285f52d3f4d23267285a7f7fbd5117f0c8e1a74f0d28f614278",
"answer_file_passed_to_candidate": false,
"isolation": "cooperative subprocess, not an OS sandbox"
},
"budget": {
"limit_fixture_units": 2000000,
"spent_fixture_units": 10506,
"unknown_reservations": 0,
"unit": "public task JSON bytes plus candidate response JSON bytes; not model tokens or money"
},
"rows": [
{
"id": "acceptance-project-0-file",
"group": "acceptance-project-0",
"kind": "file",
"baseline": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "e736f46655d8edc6ecd742c326b1703be20afd3c3bd6601d1e45eddf6e885aad",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 47.859,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "e736f46655d8edc6ecd742c326b1703be20afd3c3bd6601d1e45eddf6e885aad",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 36.653,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-0-answer",
"group": "acceptance-project-0",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 60 days.",
"days": 60,
"citations": [
"acceptance-project-0-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 23.622,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": false,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation",
"status": "scored",
"response": {
"message": "The refund window is 7 days.",
"days": 7,
"citations": [
"acceptance-project-0-old"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 120,
"fixture_units": 610,
"usage_state": "known",
"wall_ms": 48.031,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-1-file",
"group": "acceptance-project-1",
"kind": "file",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "bc560491a8815c289e6529249870a0aa4969f7e4f65dbd1d1e7d89ac5c638394",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 43.35,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "bc560491a8815c289e6529249870a0aa4969f7e4f65dbd1d1e7d89ac5c638394",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 47.127,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-1-answer",
"group": "acceptance-project-1",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 61 days.",
"days": 61,
"citations": [
"acceptance-project-1-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 48.015,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": false,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation",
"status": "scored",
"response": {
"message": "The refund window is 7 days.",
"days": 7,
"citations": [
"acceptance-project-1-old"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 120,
"fixture_units": 610,
"usage_state": "known",
"wall_ms": 23.428,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-2-file",
"group": "acceptance-project-2",
"kind": "file",
"baseline": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "8464d1799690fcd38320c01ea20b4cac4f171fe286c68627568ead92d119754b",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 40.1,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "8464d1799690fcd38320c01ea20b4cac4f171fe286c68627568ead92d119754b",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 46.84,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-2-answer",
"group": "acceptance-project-2",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 62 days.",
"days": 62,
"citations": [
"acceptance-project-2-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 45.031,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": false,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation",
"status": "scored",
"response": {
"message": "The refund window is 7 days.",
"days": 7,
"citations": [
"acceptance-project-2-old"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 120,
"fixture_units": 610,
"usage_state": "known",
"wall_ms": 38.416,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-3-file",
"group": "acceptance-project-3",
"kind": "file",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "e4bedec480f05dd6b6112854c6d03ba8fbf14c98570f22dfa24c1914a42839d7",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 39.08,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "e4bedec480f05dd6b6112854c6d03ba8fbf14c98570f22dfa24c1914a42839d7",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 47.52,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-3-answer",
"group": "acceptance-project-3",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 63 days.",
"days": 63,
"citations": [
"acceptance-project-3-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 48.145,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": false,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation",
"status": "scored",
"response": {
"message": "The refund window is 7 days.",
"days": 7,
"citations": [
"acceptance-project-3-old"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 120,
"fixture_units": 610,
"usage_state": "known",
"wall_ms": 45.675,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-4-file",
"group": "acceptance-project-4",
"kind": "file",
"baseline": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "18e841eee4e9c6e815ff2700d3dd89703d7c73fbaf0944925f4ac11eea820b4a",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 41.294,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "18e841eee4e9c6e815ff2700d3dd89703d7c73fbaf0944925f4ac11eea820b4a",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 44.582,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-4-answer",
"group": "acceptance-project-4",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 64 days.",
"days": 64,
"citations": [
"acceptance-project-4-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 42.507,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": false,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation",
"status": "scored",
"response": {
"message": "The refund window is 7 days.",
"days": 7,
"citations": [
"acceptance-project-4-old"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 120,
"fixture_units": 610,
"usage_state": "known",
"wall_ms": 41.333,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-5-file",
"group": "acceptance-project-5",
"kind": "file",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "668cf02a2eeb5172bbd88dc33f7da86b64e42a2c62eca6bd76787d6eac105db2",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 47.886,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "668cf02a2eeb5172bbd88dc33f7da86b64e42a2c62eca6bd76787d6eac105db2",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 45.435,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-5-answer",
"group": "acceptance-project-5",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 65 days.",
"days": 65,
"citations": [
"acceptance-project-5-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 46.98,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": false,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation",
"status": "scored",
"response": {
"message": "The refund window is 7 days.",
"days": 7,
"citations": [
"acceptance-project-5-old"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 120,
"fixture_units": 610,
"usage_state": "known",
"wall_ms": 43.097,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
}
],
"statistics": {
"estimand": "equal_weight_mean_of_project_task_success_differences",
"paired_groups": 6,
"cases": 12,
"difference": -0.5,
"percentile_interval_95": [
-0.5,
-0.5
],
"seed": 17,
"resamples": 2000,
"warning": "Synthetic small fixture: method demonstration, not population evidence or a model ranking"
},
"gate": {
"passed": false,
"reasons": [
"new_task_regression",
"new_answer_quality_regression",
"paired_interval_below_noninferiority_margin"
],
"policy": "fixture-gate-v1",
"noninferiority_margin": -0.05
},
"failures": [
{
"id": "acceptance-project-0-file",
"kind": "file",
"baseline_reason": "artifact_mismatch",
"candidate_reason": "artifact_mismatch"
},
{
"id": "acceptance-project-0-answer",
"kind": "answer",
"baseline_reason": "supported_fact",
"candidate_reason": "wrong_fact_or_citation"
},
{
"id": "acceptance-project-1-answer",
"kind": "answer",
"baseline_reason": "supported_fact",
"candidate_reason": "wrong_fact_or_citation"
},
{
"id": "acceptance-project-2-file",
"kind": "file",
"baseline_reason": "artifact_mismatch",
"candidate_reason": "artifact_mismatch"
},
{
"id": "acceptance-project-2-answer",
"kind": "answer",
"baseline_reason": "supported_fact",
"candidate_reason": "wrong_fact_or_citation"
},
{
"id": "acceptance-project-3-answer",
"kind": "answer",
"baseline_reason": "supported_fact",
"candidate_reason": "wrong_fact_or_citation"
},
{
"id": "acceptance-project-4-file",
"kind": "file",
"baseline_reason": "artifact_mismatch",
"candidate_reason": "artifact_mismatch"
},
{
"id": "acceptance-project-4-answer",
"kind": "answer",
"baseline_reason": "supported_fact",
"candidate_reason": "wrong_fact_or_citation"
},
{
"id": "acceptance-project-5-answer",
"kind": "answer",
"baseline_reason": "supported_fact",
"candidate_reason": "wrong_fact_or_citation"
}
]
}lab/evidence/summary.json
{
"kind": "controlled_fixture_no_real_models",
"reports": [
{
"candidate": "prompt_v2",
"cases": 12,
"statistics": {
"estimand": "equal_weight_mean_of_project_task_success_differences",
"paired_groups": 6,
"cases": 12,
"difference": 0.25,
"percentile_interval_95": [
0.08333333333333333,
0.4166666666666667
],
"seed": 17,
"resamples": 2000,
"warning": "Synthetic small fixture: method demonstration, not population evidence or a model ranking"
},
"gate": {
"passed": true,
"reasons": [],
"policy": "fixture-gate-v1",
"noninferiority_margin": -0.05
},
"failure_count": 0,
"budget": {
"limit_fixture_units": 2000000,
"spent_fixture_units": 10542,
"unknown_reservations": 0,
"unit": "public task JSON bytes plus candidate response JSON bytes; not model tokens or money"
}
},
{
"candidate": "retrieval_v2",
"cases": 12,
"statistics": {
"estimand": "equal_weight_mean_of_project_task_success_differences",
"paired_groups": 6,
"cases": 12,
"difference": -0.5,
"percentile_interval_95": [
-0.5,
-0.5
],
"seed": 17,
"resamples": 2000,
"warning": "Synthetic small fixture: method demonstration, not population evidence or a model ranking"
},
"gate": {
"passed": false,
"reasons": [
"new_task_regression",
"new_answer_quality_regression",
"paired_interval_below_noninferiority_margin"
],
"policy": "fixture-gate-v1",
"noninferiority_margin": -0.05
},
"failure_count": 9,
"budget": {
"limit_fixture_units": 2000000,
"spent_fixture_units": 10506,
"unknown_reservations": 0,
"unit": "public task JSON bytes plus candidate response JSON bytes; not model tokens or money"
}
},
{
"candidate": "model_v2",
"cases": 12,
"statistics": {
"estimand": "equal_weight_mean_of_project_task_success_differences",
"paired_groups": 6,
"cases": 12,
"difference": -0.75,
"percentile_interval_95": [
-0.9166666666666666,
-0.5833333333333334
],
"seed": 17,
"resamples": 2000,
"warning": "Synthetic small fixture: method demonstration, not population evidence or a model ranking"
},
"gate": {
"passed": false,
"reasons": [
"new_task_regression",
"paired_interval_below_noninferiority_margin"
],
"policy": "fixture-gate-v1",
"noninferiority_margin": -0.05
},
"failure_count": 12,
"budget": {
"limit_fixture_units": 2000000,
"spent_fixture_units": 10518,
"unknown_reservations": 0,
"unit": "public task JSON bytes plus candidate response JSON bytes; not model tokens or money"
}
}
],
"calibration": {
"kind": "authored_counterexamples_not_human_or_model_judge_measurements",
"confusion": {
"naive": {
"tp": 2,
"fp": 2,
"tn": 2,
"fn": 0
},
"contract": {
"tp": 2,
"fp": 0,
"tn": 4,
"fn": 0
}
},
"naive_false_positive_rate": 0.5,
"cases": [
{
"id": "file-written",
"naive_pass": true,
"contract_pass": true,
"reference_pass": true,
"score_detail": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches"
}
},
{
"id": "claims-done-file-unchanged",
"naive_pass": true,
"contract_pass": false,
"reference_pass": false,
"score_detail": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch"
}
},
{
"id": "correct-value-wrong-citation",
"naive_pass": true,
"contract_pass": false,
"reference_pass": false,
"score_detail": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation"
}
},
{
"id": "empty-answer",
"naive_pass": false,
"contract_pass": false,
"reference_pass": false,
"score_detail": {
"task_pass": true,
"quality_pass": false,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact"
}
},
{
"id": "supported-answer",
"naive_pass": true,
"contract_pass": true,
"reference_pass": true,
"score_detail": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact"
}
},
{
"id": "explicit-failure",
"naive_pass": false,
"contract_pass": false,
"reference_pass": false,
"score_detail": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch"
}
}
]
}
}lab/make_data.py
"""Create the checked-in synthetic dataset, private labels and version manifest."""
from pathlib import Path
import hashlib
import json
ROOT = Path(__file__).parent / "data"
def canonical(value):
return json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode()
def main():
ROOT.mkdir(exist_ok=True)
public, answers = [], {}
for split, count, offset in [("train", 3, 10), ("dev", 4, 30), ("acceptance", 6, 60)]:
for i in range(count):
group = f"{split}-project-{i}"
target = offset + i
for kind in ["file", "answer"]:
sid = f"{group}-{kind}"
record = {"id": sid, "group": group, "split": split, "kind": kind,
"origin": "authored-synthetic-v1", "project": group}
if kind == "file":
# Some instructions contain an unrelated issue number. A
# naive candidate parses it as the requested timeout.
instruction = f"Set timeout_ms to {target}"
if i % 2 == 0:
instruction = f"Ticket 900{i}: " + instruction
initial = "timeout_ms=5\nretries=2\n"
record.update(instruction=instruction, files={"service.cfg": initial,
"policy.lock": "DO NOT MODIFY\n"})
answers[sid] = {"expected_file": f"timeout_ms={target}\nretries=2\n",
"protected": "DO NOT MODIFY\n"}
else:
current, old = f"{group}-current", f"{group}-old"
record.update(instruction=f"How many days is the current {group} refund window?",
documents=[{"id": old, "status": "superseded", "days": 7,
"text": f"An old policy for {group}: refund window is 7 days."},
{"id": current, "status": "current", "days": target,
"text": f"The current policy for {group}: refund window is {target} days."}])
answers[sid] = {"days": target, "supporting_ids": [current]}
public.append(record)
(ROOT / "samples.json").write_bytes(canonical(public) + b"\n")
(ROOT / "answers.json").write_bytes(canonical(answers) + b"\n")
manifest = {"version": "synthetic-quality-v1", "synthetic": True,
"purpose": "Test quality pipeline behavior, not model intelligence or vendor ranking",
"splits": {s: {"samples": sum(r["split"] == s for r in public),
"groups": len({r["group"] for r in public if r["split"] == s})}
for s in ["train", "dev", "acceptance"]},
"sha256": {name: hashlib.sha256((ROOT / name).read_bytes()).hexdigest()
for name in ["samples.json", "answers.json"]}}
(ROOT / "manifest.json").write_text(json.dumps(manifest, ensure_ascii=False, indent=2) + "\n")
print(json.dumps(manifest, ensure_ascii=False))
if __name__ == "__main__":
main()lab/pipeline.py
"""Versioned paired evaluation, subprocess candidates, reports and release gate."""
from collections import defaultdict
from datetime import datetime, timezone
import hashlib
import json
from pathlib import Path
import random
import shutil
import statistics
import subprocess
import sys
import tempfile
import time
ROOT = Path(__file__).resolve().parent
MODES = {"baseline", "prompt_v2", "retrieval_v2", "model_v2", "unsafe", "terse", "review", "timeout"}
def sha(data):
return hashlib.sha256(data).hexdigest()
def load_data():
manifest = json.loads((ROOT / "data/manifest.json").read_text())
for name, expected in manifest["sha256"].items():
if sha((ROOT / "data" / name).read_bytes()) != expected:
raise ValueError("dataset_hash_mismatch:" + name)
samples = json.loads((ROOT / "data/samples.json").read_text())
answers = json.loads((ROOT / "data/answers.json").read_text())
seen, groups = set(), defaultdict(set)
for sample in samples:
if sample["id"] in seen or sample["id"] not in answers:
raise ValueError("duplicate_or_unlabelled_sample")
seen.add(sample["id"])
groups[sample["split"]].add(sample["group"])
for a in groups:
for b in groups:
if a != b and groups[a] & groups[b]:
raise ValueError("group_leaks_between_splits")
if seen != set(answers):
raise ValueError("orphan_label")
return manifest, samples, answers
def score(sample, truth, response, artifacts):
"""Separate task evidence from presentation. Not a semantic LLM judge."""
quality = isinstance(response.get("message"), str) and bool(response["message"].strip())
if sample["kind"] == "file":
task = artifacts.get("service.cfg") == truth["expected_file"]
safe = artifacts.get("policy.lock") == truth["protected"]
evidence = "artifact_matches" if task else "artifact_mismatch"
else:
# An exact structured-fact oracle, suitable only for this constrained
# dataset; it does not establish arbitrary text entailment or fluency.
fact = type(response.get("days")) is int and response["days"] == truth["days"]
citations = response.get("citations")
cited = (isinstance(citations, list) and bool(citations) and
all(isinstance(c, str) and c in truth["supporting_ids"] for c in citations))
task, safe = fact and cited, True
quality = quality and str(truth["days"]) in response.get("message", "")
evidence = "supported_fact" if task else "wrong_fact_or_citation"
review = response.get("needs_review") is True
return {"task_pass": task, "quality_pass": quality, "safety_pass": safe,
"review_pending": review, "reason": evidence}
def run_candidate(sample, truth, mode, timeout=1.0, maximum_response=4096):
public = {key: value for key, value in sample.items() if key not in {"split", "origin", "group"}}
public_bytes = json.dumps(public, sort_keys=True).encode()
with tempfile.TemporaryDirectory(prefix="quality-candidate-") as directory:
work = Path(directory)
shutil.copyfile(ROOT / "candidate.py", work / "candidate.py")
(work / "task.json").write_bytes(public_bytes)
for name, text in sample.get("files", {}).items():
if name not in {"service.cfg", "policy.lock"}:
raise ValueError("unapproved_fixture_path")
(work / name).write_text(text)
initial_names = sorted(p.name for p in work.iterdir())
start = time.monotonic()
try:
process = subprocess.run([sys.executable, "-I", str(work / "candidate.py"), mode],
cwd=work, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL, timeout=timeout, check=False,
env={"LANG": "C.UTF-8"})
elapsed = round((time.monotonic() - start) * 1000, 3)
if process.returncode != 0:
raise ValueError("candidate_process_failed")
path = work / "response.json"
if not path.is_file() or path.stat().st_size > maximum_response:
raise ValueError("missing_or_oversized_response")
raw = path.read_bytes()
response = json.loads(raw)
if not isinstance(response, dict):
raise ValueError("invalid_response")
artifacts = {name: (work / name).read_text() if (work / name).is_file() else None
for name in ["service.cfg", "policy.lock"]}
result = score(sample, truth, response, artifacts)
return {**result, "status": "scored", "response": response,
"artifact_sha256": {name: sha(text.encode()) if text is not None else None
for name, text in artifacts.items()},
"input_bytes": len(public_bytes), "output_bytes": len(raw),
"fixture_units": len(public_bytes) + len(raw), "usage_state": "known",
"wall_ms": elapsed, "candidate_visible_files": initial_names}
except subprocess.TimeoutExpired:
reason, usage_state = "timeout", "unknown"
except (ValueError, OSError, UnicodeError):
reason, usage_state = "invalid_candidate_output", "unknown"
return {"task_pass": False, "quality_pass": False, "safety_pass": False,
"review_pending": False, "reason": reason, "status": "unscored",
"response": None, "artifact_sha256": {}, "input_bytes": len(public_bytes),
"output_bytes": None, "fixture_units": None, "usage_state": usage_state,
"wall_ms": round((time.monotonic() - start) * 1000, 3),
"candidate_visible_files": initial_names}
def quantile(values, p):
ordered = sorted(values)
position = (len(ordered) - 1) * p
low = int(position)
high = min(low + 1, len(ordered) - 1)
return ordered[low] + (ordered[high] - ordered[low]) * (position - low)
def paired_statistics(rows, seed=17, resamples=2000):
"""Equal-weight group means, paired cluster percentile bootstrap.
A group is resampled as a unit with both candidates and all its cases.
The tiny authored fixture is not a representative business population.
"""
by_group = defaultdict(list)
for row in rows:
by_group[row["group"]].append(int(row["candidate"]["task_pass"]) - int(row["baseline"]["task_pass"]))
effects = [statistics.mean(values) for _, values in sorted(by_group.items())]
if not effects or resamples < 1:
raise ValueError("empty_pair_set_or_resamples")
rng = random.Random(seed)
distribution = [statistics.mean(rng.choices(effects, k=len(effects))) for _ in range(resamples)]
return {"estimand": "equal_weight_mean_of_project_task_success_differences",
"paired_groups": len(effects), "cases": len(rows), "difference": statistics.mean(effects),
"percentile_interval_95": [quantile(distribution, 0.025), quantile(distribution, 0.975)],
"seed": seed, "resamples": resamples,
"warning": "Synthetic small fixture: method demonstration, not population evidence or a model ranking"}
def release_gate(report):
reasons = []
rows = report["rows"]
if report["split"] != "acceptance":
reasons.append("not_acceptance_split")
if any(side["status"] != "scored" for r in rows for side in [r["baseline"], r["candidate"]]):
reasons.append("incomplete_or_unscored_pairs")
if any(not r["candidate"]["safety_pass"] for r in rows):
reasons.append("safety_or_unknown_failure")
if any(r["candidate"]["review_pending"] for r in rows):
reasons.append("unresolved_review")
if any(r["baseline"]["task_pass"] and not r["candidate"]["task_pass"] for r in rows):
reasons.append("new_task_regression")
if any(r["baseline"]["quality_pass"] and not r["candidate"]["quality_pass"] for r in rows):
reasons.append("new_answer_quality_regression")
if report["statistics"]["percentile_interval_95"][0] < -0.05:
reasons.append("paired_interval_below_noninferiority_margin")
# This gate accepts bounded equivalence; it does not require every release
# to be better and is not a claim of a business-powered noninferiority test.
return {"passed": not reasons, "reasons": reasons,
"policy": "fixture-gate-v1", "noninferiority_margin": -0.05}
def compare(candidate="prompt_v2", split="acceptance", budget=2_000_000, seed=17, timeout=1.0):
if candidate not in MODES or split not in {"train", "dev", "acceptance"}:
raise ValueError("unknown_candidate_or_split")
manifest, samples, answers = load_data()
cases = [s for s in samples if s["split"] == split]
rows, spent, held = [], 0, 0
for sample in cases:
pair = {"id": sample["id"], "group": sample["group"], "kind": sample["kind"]}
for label, mode in [("baseline", "baseline"), ("candidate", candidate)]:
public = {k: v for k, v in sample.items() if k not in {"split", "origin", "group"}}
reserve = len(json.dumps(public, sort_keys=True).encode()) + 4096
if spent + held + reserve > budget:
result = {"status": "budget_denied", "task_pass": False, "quality_pass": False,
"safety_pass": False, "review_pending": False, "reason": "budget_exhausted",
"fixture_units": 0, "usage_state": "not_started"}
else:
held += reserve
result = run_candidate(sample, answers[sample["id"]], mode, timeout=timeout)
if result["usage_state"] == "known":
held -= reserve
spent += result["fixture_units"]
pair[label] = result
rows.append(pair)
report = {"run_kind": "controlled_deterministic_fixture_no_real_model",
"created_at": datetime.now(timezone.utc).isoformat(), "dataset": manifest,
"split": split, "baseline_id": "baseline", "candidate_id": candidate,
"changes": {"prompt_v2": "request_normalizer", "retrieval_v2": "status_filter_removed",
"model_v2": "controlled_generator_defect"}.get(candidate, candidate),
"provenance": {"python": sys.version.split()[0], "candidate_sha256": sha((ROOT / "candidate.py").read_bytes()),
"pipeline_sha256": sha(Path(__file__).read_bytes()),
"answer_file_passed_to_candidate": False,
"isolation": "cooperative subprocess, not an OS sandbox"},
"budget": {"limit_fixture_units": budget, "spent_fixture_units": spent,
"unknown_reservations": held,
"unit": "public task JSON bytes plus candidate response JSON bytes; not model tokens or money"},
"rows": rows, "statistics": paired_statistics(rows, seed)}
report["gate"] = release_gate(report)
report["failures"] = [{"id": r["id"], "kind": r["kind"], "baseline_reason": r["baseline"]["reason"],
"candidate_reason": r["candidate"]["reason"]}
for r in rows if not r["candidate"]["task_pass"] or not r["candidate"]["quality_pass"]]
return report
def calibration():
# Execute both graders on authored counterexamples. Labels describe these
# mechanically inspectable examples, not a human rater agreement study.
_, samples, answers = load_data()
file = next(s for s in samples if s["split"] == "dev" and s["kind"] == "file")
answer = next(s for s in samples if s["split"] == "dev" and s["kind"] == "answer")
ft, at = answers[file["id"]], answers[answer["id"]]
good = {"service.cfg": ft["expected_file"], "policy.lock": ft["protected"]}
unchanged = file["files"]
supported = {"message": f"The window is {at['days']} days.", "days": at["days"], "citations": at["supporting_ids"]}
examples = [("file-written", file, {"message": "done"}, good, True),
("claims-done-file-unchanged", file, {"message": "done"}, unchanged, False),
("correct-value-wrong-citation", answer, {**supported, "citations": ["wrong-doc"]}, {}, False),
("empty-answer", answer, {**supported, "message": ""}, {}, False),
("supported-answer", answer, supported, {}, True),
("explicit-failure", file, {"message": "failed"}, unchanged, False)]
matrices = {name: {"tp": 0, "fp": 0, "tn": 0, "fn": 0} for name in ["naive", "contract"]}
cases = []
for sid, sample, response, artifacts, reference in examples:
truth = answers[sample["id"]]
naive_pass = ("done" in response["message"] if sample["kind"] == "file"
else str(truth["days"]) in response["message"])
actual = score(sample, truth, response, artifacts)
contract_pass = all(actual[k] for k in ["task_pass", "quality_pass", "safety_pass"])
for name, predicted in [("naive", naive_pass), ("contract", contract_pass)]:
matrices[name][("t" if predicted == reference else "f") + ("p" if predicted else "n")] += 1
cases.append({"id": sid, "naive_pass": naive_pass, "contract_pass": contract_pass,
"reference_pass": reference, "score_detail": actual})
return {"kind": "authored_counterexamples_not_human_or_model_judge_measurements",
"confusion": matrices,
"naive_false_positive_rate": matrices["naive"]["fp"] / (matrices["naive"]["fp"] + matrices["naive"]["tn"]),
"cases": cases}
def main():
import argparse
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--candidate", choices=sorted(MODES), default="prompt_v2")
parser.add_argument("--split", choices=["train", "dev", "acceptance"], default="acceptance")
parser.add_argument("--budget", type=int, default=2_000_000)
parser.add_argument("--output", type=Path, default=Path("lab/evidence/report.json"))
parser.add_argument("--require-pass", action="store_true")
args = parser.parse_args()
report = compare(args.candidate, args.split, args.budget)
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n")
print(json.dumps({"candidate": args.candidate, "cases": len(report["rows"]),
"statistics": report["statistics"], "gate": report["gate"],
"failures": report["failures"], "output": str(args.output)}, ensure_ascii=False))
if args.require_pass and not report["gate"]["passed"]:
raise SystemExit(2)
if __name__ == "__main__":
main()lab/review.py
"""Apply explicit presentation-review decisions to a fingerprinted saved report.
Self-asserted reviewer IDs are for teaching audit shape, not authenticated human
identity. No review decision can rewrite artifact or safety acceptance results.
"""
import argparse
import copy
import json
from pathlib import Path
from .pipeline import release_gate, sha
def apply(report, report_bytes, decisions):
if json.loads(report_bytes) != report:
raise ValueError("report_object_does_not_match_fingerprint_bytes")
if decisions.get("report_sha256") != sha(report_bytes):
raise ValueError("review_targets_different_report")
result = copy.deepcopy(report)
rows = {r["id"]: r for r in result["rows"]}
seen = set()
for decision in decisions.get("decisions", []):
sid = decision["sample_id"]
if sid not in rows or sid in seen:
raise ValueError("unknown_or_duplicate_review_id")
seen.add(sid)
if decision["decision"] not in {"accept_presentation", "reject_presentation"}:
raise ValueError("review_cannot_override_task_or_safety")
if not decision.get("reviewer") or not decision.get("rationale"):
raise ValueError("review_needs_identity_and_rationale")
candidate = rows[sid]["candidate"]
if not candidate["review_pending"]:
raise ValueError("sample_not_pending_review")
candidate["review_pending"] = False
if decision["decision"] == "reject_presentation":
candidate["quality_pass"] = False
result["review_audit"] = {"source_report_sha256": sha(report_bytes), "decisions": decisions["decisions"],
"identity_assurance": "self-asserted teaching identity, not authenticated"}
result["gate"] = release_gate(result)
return result
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--report", required=True, type=Path)
parser.add_argument("--decisions", required=True, type=Path)
parser.add_argument("--output", required=True, type=Path)
args = parser.parse_args()
data = args.report.read_bytes()
reviewed = apply(json.loads(data), data, json.loads(args.decisions.read_text()))
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(json.dumps(reviewed, ensure_ascii=False, indent=2) + "\n")
print(json.dumps(reviewed["gate"], ensure_ascii=False))
if __name__ == "__main__":
main()tests/test_pipeline.py
import copy
import json
from pathlib import Path
import subprocess
import sys
import tempfile
import unittest
from lab.pipeline import ROOT, calibration, compare, load_data, paired_statistics, run_candidate, score, sha
from lab.review import apply
class PipelineTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.manifest, cls.samples, cls.answers = load_data()
cls.good = compare("prompt_v2")
cls.bad = compare("model_v2")
def test_dataset_fingerprints_and_group_isolation(self):
self.assertEqual(self.manifest["splits"]["acceptance"], {"samples": 12, "groups": 6})
groups = {s: {r["group"] for r in self.samples if r["split"] == s} for s in ["train", "dev", "acceptance"]}
self.assertFalse(groups["train"] & groups["dev"])
self.assertFalse(groups["dev"] & groups["acceptance"])
def test_candidate_workdir_has_no_reference_answers_or_grader(self):
for row in self.good["rows"]:
for side in [row["baseline"], row["candidate"]]:
visible = side["candidate_visible_files"]
self.assertNotIn("answers.json", visible)
self.assertNotIn("pipeline.py", visible)
self.assertIn("candidate.py", visible)
self.assertIn("task.json", visible)
self.assertEqual(self.good["provenance"]["isolation"], "cooperative subprocess, not an OS sandbox")
def test_claimed_completion_does_not_override_file_evidence(self):
failed = [r for r in self.bad["rows"] if r["kind"] == "file"]
self.assertTrue(all(r["candidate"]["response"]["message"] == "done" for r in failed))
self.assertTrue(all(not r["candidate"]["task_pass"] for r in failed))
self.assertTrue(all(r["candidate"]["quality_pass"] for r in failed))
def test_citation_identity_is_separate_from_correct_value(self):
rows = [r for r in self.bad["rows"] if r["kind"] == "answer"]
self.assertTrue(all(r["candidate"]["quality_pass"] for r in rows))
self.assertTrue(all(not r["candidate"]["task_pass"] for r in rows))
def test_controlled_request_fix_passes_fixture_gate(self):
self.assertTrue(self.good["gate"]["passed"])
self.assertEqual(self.good["statistics"]["difference"], 0.25)
self.assertEqual(self.good["statistics"]["paired_groups"], 6)
def test_generator_defect_fails_gate_and_keeps_case_failures(self):
self.assertFalse(self.bad["gate"]["passed"])
self.assertIn("new_task_regression", self.bad["gate"]["reasons"])
self.assertEqual(len(self.bad["failures"]), 12)
def test_retrieval_change_attribution_is_recorded(self):
result = compare("retrieval_v2")
self.assertEqual(result["changes"], "status_filter_removed")
self.assertEqual(result["statistics"]["difference"], -0.5)
self.assertEqual(sum(r["kind"] == "answer" for r in result["failures"]), 6)
def test_budget_denials_remain_in_denominator_and_gate(self):
report = compare("prompt_v2", budget=1)
self.assertEqual(len(report["rows"]), 12)
self.assertEqual(report["budget"]["spent_fixture_units"], 0)
self.assertFalse(report["gate"]["passed"])
self.assertIn("incomplete_or_unscored_pairs", report["gate"]["reasons"])
def test_timeout_keeps_unknown_reservation(self):
sample = self.samples[0]
result = run_candidate(sample, self.answers[sample["id"]], "timeout", timeout=0.05)
self.assertEqual((result["reason"], result["usage_state"]), ("timeout", "unknown"))
report = compare("timeout", budget=15000, timeout=0.04)
self.assertGreater(report["budget"]["unknown_reservations"], 0)
self.assertFalse(report["gate"]["passed"])
def test_task_success_and_presentation_quality_can_diverge(self):
sample = next(s for s in self.samples if s["kind"] == "answer")
result = run_candidate(sample, self.answers[sample["id"]], "terse")
self.assertTrue(result["task_pass"])
self.assertFalse(result["quality_pass"])
def test_protected_artifact_change_is_independent_safety_failure(self):
sample = next(s for s in self.samples if s["kind"] == "file")
result = run_candidate(sample, self.answers[sample["id"]], "unsafe")
self.assertFalse(result["safety_pass"])
def test_calibration_executes_both_graders_and_finds_false_positives(self):
result = calibration()
self.assertEqual(result["confusion"]["naive"], {"tp": 2, "fp": 2, "tn": 2, "fn": 0})
self.assertEqual(result["confusion"]["contract"], {"tp": 2, "fp": 0, "tn": 4, "fn": 0})
self.assertEqual(result["naive_false_positive_rate"], 0.5)
def test_bootstrap_preserves_pair_and_project_unit(self):
rows = copy.deepcopy(self.good["rows"])
result = paired_statistics(rows, seed=22)
self.assertEqual(result, paired_statistics(list(reversed(rows)), seed=22))
self.assertEqual(result["paired_groups"], 6)
self.assertLessEqual(result["percentile_interval_95"][0], result["difference"])
self.assertGreaterEqual(result["percentile_interval_95"][1], result["difference"])
def test_development_split_cannot_be_published_by_gate(self):
result = compare("prompt_v2", split="dev")
self.assertIn("not_acceptance_split", result["gate"]["reasons"])
def test_pending_review_blocks_release_and_wrong_report_is_rejected(self):
report = compare("review")
self.assertIn("unresolved_review", report["gate"]["reasons"])
raw = json.dumps(report).encode()
decisions = {"report_sha256": "wrong", "decisions": []}
with self.assertRaises(ValueError):
apply(report, raw, decisions)
with self.assertRaises(ValueError):
apply({**report, "candidate_id": "different"}, raw,
{"report_sha256": sha(raw), "decisions": []})
def test_review_cannot_erase_artifact_failure(self):
report = compare("review")
raw = json.dumps(report).encode()
decisions = {"report_sha256": sha(raw), "decisions": [{"sample_id": r["id"],
"decision": "accept_presentation", "reviewer": "teaching-reviewer",
"rationale": "Authored workflow demonstration, not authenticated human judgment"}
for r in report["rows"]]}
reviewed = apply(report, raw, decisions)
self.assertNotIn("unresolved_review", reviewed["gate"]["reasons"])
self.assertEqual([r["candidate"]["task_pass"] for r in report["rows"]],
[r["candidate"]["task_pass"] for r in reviewed["rows"]])
def test_cli_returns_nonzero_for_blocked_release(self):
with tempfile.TemporaryDirectory() as directory:
process = subprocess.run([sys.executable, "-m", "lab.pipeline", "--candidate", "retrieval_v2",
"--require-pass", "--output", str(Path(directory)/"report.json")],
cwd=ROOT.parent, capture_output=True, text=True)
self.assertEqual(process.returncode, 2)
self.assertFalse(json.loads((Path(directory)/"report.json").read_text())["gate"]["passed"])
if __name__ == "__main__":
unittest.main()