让样本、评分与发布决定可以复核
2026-10-21
一个知识助手回答得更流畅，可能只是删掉了不确定措辞；一个编程代理说“已经修好”，文件却可能保持原样。把最终文本交给另一个模型打一个总分，并不能自动解决这些问题。质量交付需要连接样本、参考证据、执行产物、评分器、比较方法与发布责任。
本书写给已经能调用模型、检索器或工具的开发者。我们关注的不是再搭一个聊天窗口，而是回答一个具体问题：当提示、检索、生成器或运行策略发生变化时，如何知道哪些任务改善、哪些任务退步，以及这份证据是否足以支持发布。
随书代码只有 Python 标准库。它包含 26 个明确标注为合成的小样本，分为训练资料、开发和验收三份，按项目组隔离。两类任务分别要求修改文件，以及根据当前政策给出带引用的结构化回答。答案与验收器由父进程持有，候选只接收任务和临时工作区。
候选是确定性程序和可控缺陷，不是真实语言模型。所谓
prompt_v2
在实验中是请求归一化替身，retrieval_v2
是检索过滤变更，model_v2
是生成器缺陷变体。这些名字描述流水线中的变更位置，不构成真实提示效果、供应商性能或模型排名。
你将实际看到：文件没有改却被文字评分误判为成功，答案数值正确却引用旧资料，开发集的好结果被发布门槛拒绝，预算不足时样本仍留在分母，人工裁决无法抹掉确定的产物失败，以及按项目成对重采样生成的区间。
第 1–4 章建立决策、数据与验收合同；第 5–8 章讨论评分器、受控对照和统计；第 9–12 章连接预算、争议、归因与发布；第 13–14 章运行完整实验和故障回归；第 15–16 章说明真实模型接入与持续维护需要增加什么。
如果希望先动手，可以从第 13 章开始。请保留报告中的失败记录，再回到前面的原理解释它们。不要为了获得绿色检查，把未知、争议或预算拒绝样本删掉。
本版没有调用真实模型，没有测量业务总体，没有收集真实人工标注一致性，也没有建立操作系统沙箱。公开的小型合成验收集可以验证流水线行为，不能被当作未曝光的真实模型基准。统计区间是方法演示，不能弥补样本代表性与独立性不足。
NIST AI RMF、SciPy 和 Inspect 的第一方资料用于核对测量、重采样以及数据/评分职责等概念。具体发布策略与实验结构是本书工程设计，不代表这些项目背书。来源、读取时间和范围见附录。
评测从一个决定开始。例如：检索器升级后，是否允许把新版本交给同一批用户？这个问题包含使用范围、可接受损失、必须保持的性质和可回退版本。若只写“评估一下效果”，团队往往会在看到结果之后才选择最有利的指标。
一个 AI 应用版本至少包含代码、提示模板、模型标识与参数、检索配置、数据快照、工具权限和评分器。只保存模型名字，无法解释为什么同一个名字在两次运行中得出不同结果。供应商服务可能变化，索引也可能在后台更新。
发布记录应能够回答：这次比较的是哪两个完整配置？哪个字段发生变化？哪些外部状态没有被固定？如果变更同时包含新模型、新提示和新文档，就只能评估整体结果，不能凭总分把收益归功于其中一个因素。
随书报告保存候选脚本哈希、流水线哈希、数据与答案哈希、候选 ID 和变更位置。它没有提供真实供应商快照保证；这个缺口必须在接入真实系统时补上。
一次结果可以支持“修复了已知反例”“没有发现指定回归”“在这组样本上平均提高”，也可能为更广使用范围提供统计证据。这几句话的强度不同。修复一个反例并不证明整个产品改善；未检出回归也不代表风险为零。
本书小实验主要支持前两类结论，并演示第三类的计算。它没有代表性业务样本，因此不会把区间下界大于零解释为真实用户总体必然获益。NIST AI RMF 的测量功能强调评测上下文、方法、限制与不确定性；它不是一张通过某个分数即可取得的认证证书。[S1]
可以把门槛分成三部分。第一部分是不可被平均值抵消的约束，例如修改了保护文件。第二部分是样本与运行完整性，例如有未执行样本、未知消耗或未解决争议。第三部分才是允许的任务质量变化与不确定区间。
本实验采用保守的样本级规则：新出现的任务或表达合同回归会阻止发布；候选安全失败、未完成配对、待审样本也会阻止发布；按项目计算的区间下界不能低于演示用的 -0.05。这个数字是教学政策，不是通用业务标准。
门槛允许与基线保持相当的版本，不要求每次发布都严格提高。安全修补、依赖升级或运维简化也可能有价值，条件是其质量代价满足事先约定。相反，平均分提高不能豁免明确的越权或重要功能损失。
自动流水线负责按固定规则计算，业务负责人决定范围与风险容忍度，数据与评分维护者解释证据缺口。三种责任不能全部交给一个模型裁判。人工例外若被允许，应记录批准人、理由、影响范围与到期时间，不能通过临时改阈值让旧报告看似通过。
练习。 选择一次你最近做过的提示或检索更新，写出发布对象、候选变更、两项不可平均的约束、一个可接受退化范围和回退版本。先不要写“准确率越高越好”；明确哪一种失败会真正伤害使用者。
一行“问题—答案”不足以描述一个可维护任务。你还需要知道答案依据哪一版事实、允许采取什么动作、哪些资源不得修改、样本来自哪里，以及它代表哪一种使用场景。否则，同一输出可能因评审者心中的背景不同而得到相反结论。
稳定 ID 用来连接执行结果、人工意见和历史故障。ID 不应随列表排序变化；删除一条数据以后，其他任务也不应被重新编号。本实验使用项目、切分和任务类别组成可读 ID，生产数据也可以用不可推测标识，并单独保存受控来源映射。
来源记录包括采集时间、权利范围、脱敏方式、使用限制和版本。真实用户内容不应因为“只用于内部评测”而自动获得无限保存许可。脱敏还可能改变任务难度，处理前后的差异应进入数据说明。
Inspect 的数据结构把 input、target、id、metadata 和样本文件分开表达，说明任务输入与评分参考本来就是不同职责。[S3] 本书借用这一分层思想，自己实现轻量合同，没有依赖或运行 Inspect。
文件任务的公开部分包含初始文件和修改指令；私有参考包含应得到的完整文件内容，以及保护文件必须保留的字节。知识回答公开问题和可用文档；参考保存正确天数与真正支持该事实的文档 ID。
“私有”在这里描述输入职责，而不是宣称仓库文件已经被操作系统保护。实验把答案留在父进程，只复制候选脚本、任务和工作区文件到临时目录。仓库本身是公开的，候选与父进程也使用同一身份；恶意程序仍可能读宿主文件。第 6 章会明确这个教学隔离的边界。
manifest.json 给出 samples.json 和
answers.json 的 SHA-256。加载器核对哈希、ID
唯一性、标签完整性和跨切分的项目重叠。只改答案而不更新清单，会被拒绝，避免悄悄用新标准解释旧报告。
哈希能证明字节一致，不能解释语义变化。因此版本说明还应记录新增、删除、标签修正和适用范围。某条参考答案被纠正后，可以用新评分器重评旧产物，但新旧结论要并列保存，不能覆盖历史结果而抹去判断过程。
本实验的数据版本为 synthetic-quality-v1，26
条任务全部由作者构造。它们有明确参考结果，却不代表真实流量分布，也不适合用来训练或排名真实语言模型。
设想有 100 条任务，其中 20 条因连接失败没有执行。如果只对其余 80 条计算正确率，一个不稳定的系统可能看起来更优秀。报告应同时保存计划样本数、启动数、可评分数、成功数以及失败原因，并声明所用分母。
随书流水线把预算拒绝与执行失败保留在逐样本行中；发布门槛会拒绝不完整配对。它仍会生成演示统计，便于检查数据，但不允许用这些不完整结果支持发布。计算一个数和允许解释这个数，是两项不同的检查。
练习。 为一条生产失败建立样本卡：公开输入、私有参考、原始证据位置、保护条件、来源范围和版本。再设计一次标签更正，说明旧报告应如何保留，而不是只替换标准答案。
把列表随机分成三份，不一定获得三个独立的数据集。同一客户的多个问法、同一工单的重述、相同文档的不同切块，都可能把答案线索带到另一份数据里。模型或工程人员学到的是同一来源，而非可推广的新能力。
训练资料允许用于拟合模型或构造方法；开发集允许调提示、检索与阈值；验收集用于在配置冻结之后比较候选。在本实验中没有任何模型训练，train
只是展示这类资料应该如何与后续阶段隔开，不应写成“已经训练了一个模型”。
验收不是禁止维护的数据墓地。业务范围改变、事实过期或发现错误标签时，需要正式版本更新。关键是区分数据维护与为了当前候选过关而调题。后者会让“未参与开发”的声明失去依据。
本实验每个项目有两条关联任务：一条文件配置修改，一条政策回答。训练资料包含 3 个项目、6 条任务；开发包含 4 个项目、8 条任务；验收包含 6 个项目、12 条任务。项目 ID 不跨切分，统计时也以项目作为重采样单位。
真实应用的单位可能是客户、仓库、工单、文档来源或时间段。应先画出数据之间的关系，再决定切分。若同一仓库的修复任务被分到开发和验收两边，某些代码、测试和修改习惯可能已经被开发过程观察到。
时间切分适合评估未来事实或漂移问题，但也不是万能的。同一事件的重复抓取可能跨过时间边界；后续摘要还可能包含早期样本的标准答案。来源追踪与去重应结合使用。
本书为了可复现，公开样本生成器、数据和答案。即使程序没有把答案文件传给候选，读者仍可以查看全部内容。因此实验验证的是“管线按切分规则处理”，不是“候选从未见过验收题”。两种保证不能互换。
生产环境若需要保密验收，应控制数据访问，隔离候选运行身份，记录题目曝光，限制同一验收集上的反复试探。单纯隐藏文件名或依靠提示词要求“不看答案”都不能提供同样保证。
如果在同一验收集上试过几十个提示，然后只报告最好的一个，即使没有训练模型，也已经利用验收反馈选择了策略。一次比较的区间没有自动包含这种选择过程的不确定性。
可行做法包括：把大多数迭代留在开发集；保留一次性或定期更新的确认集；记录尝试过的候选数量；对重要发布追加新的独立场景。具体样本量需要结合效应、成本和风险设计，本书不会给出一个通用最小数。
练习。 某知识库用同一份政策生成了 20 种问题，随机分到开发和验收两份。请指出泄漏路径，并用来源或客户分组重做划分。重新切分后分数下降，不一定说明系统退步，也可能是原先的证据过于乐观。
一个配置文件已经正确修改，最终回复却为空；另一个回复礼貌、完整地宣称“done”，磁盘上却什么都没有改变。这两种情况都不能用一个笼统的回答分数解释。验收首先应对应使用者真正要得到的结果。
随书文件任务要求修改 service.cfg，并保持
policy.lock
原样。评分器读取真实工作区产物，对照参考文件完整内容，分别记录
task_pass 与
safety_pass。候选的成功声明不能覆盖文件证据，保护文件被改也不能被正确配置抵消。
完整字节比较适合本实验的小型确定性配置；真实代码任务可能需要解析结果、编译、行为测试以及资源约束。不要机械地把空白差异视为功能失败，也不要只因为测试通过，就推断所有未覆盖行为都正确。验收器本身需要版本与反例。
回答任务询问当前退款天数。公开资料包含旧版和当前政策。参考分别给出正确数值与支持该数值的文档 ID；一个数值正确但引用旧政策的输出仍会失败。
这种检查是有约束的结构化事实验收，不是任意自然语言蕴含识别。模型可能用不同措辞正确回答，也可能在长文本中同时包含对与错。本实验只测试规定的
days 字段和引用集合，不声称解决了开放域事实核查。
quality_pass
检查非空消息，回答任务还检查约定数值是否在消息中出现。这是最低表达合同，用来演示任务与表达可以分离；它不能衡量文采、可读性、礼貌、专业深度或完整语义质量。
例如 model_v2
对文件任务只返回“done”。表达合同可以通过，产物验收失败。terse
变体则能给出正确结构化答案与引用，却把消息设为空；任务验收可以通过，表达合同失败。这两个反例比给每个输出打一个总分更能说明责任边界。
真实产品可追加引用覆盖率、可理解性量表或专家评分，但应先定义评分对象、取值与判断依据。若使用模型裁判，必须单独校准，不应让裁判的流畅解释代替任务证据。
候选超时、输出格式损坏或缺少产物时，报告保存 unscored
与具体原因，而不是伪造一次正常评分。它们在发布层面会阻止完整性通过，但仍应与“执行完整、答案确实错误”分开分析。
review_pending
表示表达意见仍待裁决；它不能让候选跳过确定性任务要求。即使有人认为回复写得不错，文件没有修改这一事实也不会改变。人工意见适合解决规则未覆盖的表达争议，不应改写已经明确的产物记录。
练习。 为你的系统列出一个任务完成但表达差的例子，以及一个表达好但任务失败的例子。为二者分别选择可观察证据，再决定哪些条件会阻止发布。避免让“最终回答像成功”同时承担执行、事实和安全三项职责。
评测系统至少有两个会出错的对象：候选应用与评分器。评分器漏掉引用问题，可能使更差的系统被判为更好；评分器过于依赖固定措辞，又可能把合理表达判错。增加评分次数不能修复一个系统性的判断偏差。
校准集应包含正常、边界、恶意或误导性样本，并为每个样本写出参考判断和理由。参考标签不应该只是从当前评分器的输出复制而来，否则校准会变成自我验证。
随书 calibration()
构造六个可机械检查的反例，真正调用两种评分规则。简陋规则只看文件回复里是否包含
done，或答案文本是否含正确数值；合同规则结合产物、引用和最低表达要求。它们不是对真实模型裁判的测量，也不是人类评审一致性研究。
其中两个关键反例是“说完成、文件未改”与“数值正确、引用错误”。程序保存实际判定、参考判定与详细分项，因此可以逐例解释误判来源，而不是只展示一个漂亮的准确率。
本书把“应通过”的输出定义为正类。TP 是正确放行，FP 是错误放行，TN 是正确拒绝，FN 是错误拒绝。定义正类之后，才有意义讨论误报与漏报，不能把中文名称混着使用。
本版实跑结果如下：
| 评分规则 | TP | FP | TN | FN |
|---|---|---|---|---|
| 简陋文字/数值匹配 | 2 | 2 | 2 | 0 |
| 产物与引用合同 | 2 | 0 | 4 | 0 |
简陋规则在四个应拒绝反例中放过两个，假阳性率为 2/(2+2)=0.5。合同规则在这六例里没有错误，不表示它在真实任务中完美。反例就是为特定缺陷构造的，样本很少，也不是随机抽样总体。
修改评分器可能改变所有历史分数。应保存旧评分器、原始产物和新旧判定差异，检查变化来自修正错误、扩大适用范围还是引入新的偏差。不能把评分器更宽松造成的分数上升，当作候选能力提升。
一个实用顺序是先固定候选产物，只换评分器；校准完成后，再用同一版评分器比较候选。若必须同时修改，应把两条变化分别报告，否则归因不可辨认。
真实模型裁判可能受答案长度、顺序、措辞、自信程度或受测文本中的指令影响。它的版本、提示、输入顺序和重复次数都应进入记录。对能用文件和结构化条件检查的任务，优先保留这些独立证据；裁判适合补充它们难以覆盖的表达或专业判断。
Inspect 文档区分文本匹配、自定义规则和模型评分，并把聚合指标作为评分器职责的一部分。[S2] 它提供了组织接口的参考，但不会替你证明某个裁判适合自己的业务。
练习。 为当前评分器增加三个它应当拒绝的输出，以及两个措辞不同但应通过的输出。让同一批固定产物分别经过旧、新评分器，逐例解释每个变化。不要只挑选有利于新评分器的反例。
如果候选能够修改评分脚本或读取标准答案，分数再精确也缺少可信基础。但另一个极端也不正确：只要把程序放进子进程，就宣称实现了安全沙箱。隔离必须按实际机制与威胁模型描述。
父进程加载参考答案，创建新的临时目录，复制 candidate.py
与 task.json，再放入该任务的初始文件。子进程通过绝对路径的
Python 解释器，以 -I 隔离模式运行；它的标准输入关闭，stdout
与 stderr 不进入评测日志。
候选写出 response.json
与任务产物，父进程在子进程结束后读取并评分。评分器和答案文件没有被复制到工作区，候选也不会得到父进程中的
truth 对象。测试检查
candidate_visible_files，防止将来改动时无意把评分材料混进输入。
Python 的 -I
影响解释器路径和环境处理，不限制系统调用、网络、绝对路径或进程权限。这里的保证是“没有通过协议给候选答案”，而不是“候选无法从其他途径获取答案”。
随书候选都是作者可审查的合作程序，因此适合演示输入分离与生命周期。它们使用与父进程相同的操作系统身份，公开仓库中的答案也不是秘密。恶意候选可以尝试访问宿主文件；当前实现不能作为执行陌生代码的公共服务。
生产评测需要按风险增加文件系统、身份、网络、资源与进程边界。评分器应保存在候选不能写的位置，答案与凭据需要独立访问控制，产物导出也要验证。即便使用容器，还需要核对挂载、用户、网络和宿主权限，而非仅检查“是否有 Docker”。
父进程限制候选响应文件最多 4096 字节，要求 JSON 对象。候选超时、退出异常、缺失文件或输出格式不合法，都会记录为不可正常评分，而不是凭空得到一个成功或免费结果。
响应文件上限不是候选总磁盘、内存或 CPU 配额；本实验没有针对恶意写盘实现硬隔离。stdout 不保留可以减少偶然泄漏和无界日志，但它也不替代运行资源控制。所有限制都应说清约束对象。
不要只相信候选附带的“文件哈希”。本实验由父进程读取工作区中的真实文件，计算哈希并保存，再检查完整内容与保护条件。候选声明只进入 response，不能取代独立验收。
真实代码任务还要考虑后台进程是否继续修改工作区、符号链接是否指向外部，以及验收时看到的是哪一版内容。可靠隔离与稳定快照应在评分前建立；无法保证时，要将结论标为有条件，而不是默认为已经解决。
练习。 画出候选、答案、评分器、工作区和报告五个对象的读写关系。标出本实验真正阻止了哪条路径，又有哪些路径只是合作约定。将“不能读答案”改写成可以验证的权限要求。
基线通过 75%，候选通过 80%，不能直接告诉你哪类任务改善。也许候选修复了五个新问题，却损坏了四个原本可靠的重要功能；总分只剩一个百分点的净差。发布者需要看到变化的构成。
比较时，两边应使用相同任务、初始文件、证据版本与评分器。否则差值可能来自难度变化。对随机系统，还要记录运行参数和重复方式；相同 seed 并不保证不同供应商共享相同随机过程。
随书 rows 对每个样本保存 baseline 与 candidate
两份结果，包含分项判定、失败原因、响应、产物哈希、用量状态和可见文件清单。任务
ID 连接两边，不能用结果列表当前顺序默默对齐。
本实验候选确定且离线，不需要用重复运行估计模型采样变化。多跑十次同一个确定程序不会产生十份独立业务样本，也不应把它们计为更大的总体。
逐样本任务结果可分为都成功、修复、回归、都失败。修复说明新版本解决了哪些问题；回归尤其需要分类与审查；都失败则提醒我们，版本没有破坏旧功能不等于它已经足够好。
在验收集里，基线对三条带工单号的文件指令取错数字，六条知识回答都正确。prompt_v2
的请求归一化修复三条文件任务，没有引入新失败。12 条任务中成功比例差为
3/12=0.25。
这里修复的是确定性数字提取缺陷。它不是语言模型理解能力提高了 25 个百分点，也不是证明这组题之外仍有相同收益。报告保留具体变更与样本 ID，就是为了防止这个结论被过度解读。
model_v2
可能给出正确天数与流畅消息，却引用旧文档；它的最低表达检查仍可能通过，任务验收失败。文件任务里，done
也不能证明产物存在。若只比较质量字段，会漏掉这些确定回归。
应分别展示任务、表达、安全、运行完整性和人工待审情况。不同指标之间可以有关联，但不能通过一个加权平均让严重失误消失。权重也需要业务理由与事先记录，不能在看到候选表现以后临时调整。
当某条任务从成功变失败，报告至少要能定位它的输入版本、候选配置、输出、验收依据和失败分类。对含敏感内容的真实样本，完整产物应保存在受控存储中，报告仅暴露必要定位符；公开样例则可使用专门构造的脱敏反例。
本实验所有文本都是合成数据，因而能公开完整响应，但这不是生产日志的默认保留策略。保留可解释证据与无差别保存用户内容，是两件不同的事。
练习。 选择一份比较报告，分别列出三条修复、三条回归与三条共同失败。若回归为空，挑选最重要的共同失败解释当前系统仍有哪些边界，而不是把报告写成“无问题”。
一个差值可以准确描述当前样本，却不足以表明未来任务上的变化。置信区间尝试表达抽样带来的不确定性，其解释依赖数据来源、独立单位、估计对象和方法条件。给数字加上“95%”不会自动使样本具有代表性。
本实验先在每个项目内计算候选减基线的任务成功差，再对项目等权平均。每个项目有一条文件任务和一条回答任务，因此当前数据上，它也等于全样本平均差；项目规模不同时，这两种估计对象可能不同。
项目等权回答“平均项目受影响多少”，任务等权更接近“平均任务受影响多少”。都可能合理，但报告必须写明选择。不能在结果不利时才从一种权重换到另一种。
paired_statistics()
把同项目的两种任务与两边候选一起保留，再有放回抽取项目，计算每次项目均值差。程序固定
seed=17，执行 2000 次重采样，取经验分布的 2.5% 与 97.5%
分位数，得到百分位区间。
重采样的关键不是循环次数，而是保留依赖关系。将同一个项目里的任务当作独立记录，或把基线与候选分开抽样，会改变不确定性结构。SciPy 的 bootstrap 文档明确区分成对重采样和不同区间方法；本书使用标准库实现简单百分位版本，没有调用 SciPy，也没有实现其 BCa 修正。[S4]
| 受控候选 | 项目均值差 | 演示用 95% 百分位区间 |
|---|---|---|
请求归一化 prompt_v2 |
0.25 | [0.0833, 0.4167] |
删除状态过滤 retrieval_v2 |
-0.50 | [-0.50, -0.50] |
生成缺陷 model_v2 |
-0.75 | [-0.9167, -0.5833] |
这些数字来自 6 个合成项目，不是实际业务抽样。检索变体每个项目都发生相同 -0.5 差值，所以重采样区间退化成一点；它不代表现实中没有不确定性，只说明当前经验分布没有呈现项目差异。
本书不会把小合成集上的正区间解释为真实模型显著改善。百分位 bootstrap 在小样本、离散指标或偏斜分布下可能表现不佳；增加重采样次数只会更稳定地近似当前经验分布，不会补充缺失场景。
区间跨零也不等于证明没有影响；区间较宽可能意味着数据不足。反过来，预设的非劣界值不能在看到结果以后随意放宽。若要做业务级非劣检验，应先设计样本量、独立单位和决策风险，必要时请统计专业人员审查。
同时查看许多切片、指标与候选，最后只报告最好的一项，会引入选择偏差。本书三组变体用于说明已知故障，不是搜索最优真实模型。生产中应预先区分主要指标、探索性分析和确认性比较，完整记录候选试验过程。
练习。 给某个项目复制十条几乎一样的任务。分别按任务和按项目重采样，解释为什么区间可能改变；再说明新增重复是否真的增加了独立信息。切勿把更窄的区间自动当作更可信。
评测也会消耗资源。模型调用、环境启动、执行时间、存储与人工复核都可能成为瓶颈。一个预算策略如果只留下便宜而容易成功的任务，就会悄悄改变样本分布，使看起来更好的结果来自选择过程而非系统改善。
本实验用公开任务 JSON 的字节数，加候选响应 JSON 的字节数，定义
fixture_units。它不是模型
token、货币费用，也不包含解释器、CPU、临时磁盘和人工成本。这样定义只是为了让预留与结算可以在没有外部服务时复现。
每次运行前，流水线预留输入字节数加 4096 字节的最大响应额度；完整响应通过格式检查后，以实际字节结算并释放差额。配置的总预算同时约束已结算与尚未解除的预留。
候选超时或输出不可核实时，实验把预留标为未知并保留。这是一项保守的教学政策；它不声称字节本身产生了账单，也不把缺少结果解释为零消耗。真实模型接口需要依据供应商的可信用量与费用合同重新设计单位。
python3 -m lab.pipeline --candidate prompt_v2 --budget 1 \
--output /tmp/quality-budget-denied.json --require-pass预算为 1 时，任何正常输入都无法获得预留，报告仍包含计划的 12
个验收任务，而不是生成一个空列表后声称没有失败。命令以 2 退出，门槛记录
incomplete_or_unscored_pairs。
这份报告可以诊断预算策略，却不能支持候选质量比较。两边都未执行而被记成失败时，演示统计可能给出零差值；完整性门槛正是为了防止“同样没运行，所以没有退步”成为发布理由。
wall_ms
是本机候选子进程的实际经过时间，包含解释器启动等开销。它会随机器和调度变化，不能当作模型推理延迟，更不能据此排名云服务。实验不会把一次本机毫秒差异写成性能改善。
真实系统需要区分端到端延迟、排队、模型服务、工具执行与人工等待。成本也要按完成任务和失败尝试分别查看。一个便宜但频繁失败的候选，可能在每个成功任务上的综合成本更高。
本实现为了易读，先运行基线，再运行候选。它适合验证确定性质量行为，不适合比较存在缓存、负载或时段影响的真实性能。接入服务后，应考虑交错或随机顺序，同时保存配对关系。
若预算只够完成部分任务，应事先定义分层抽样与停止政策，不能跑到看到喜欢的结果就停。计划变化需成为报告的一部分。对重要发布，完整性不足通常意味着补充实验，而非直接放宽门槛。
练习。 让预算足够执行前几对但不足以完成全部任务，检查报告仍保留哪些行、未知预留如何影响后续接纳、门槛为何拒绝。再设计一份真实费用合同，分别标注哪些字段可从服务端确认，哪些只是本地估算。
表达质量往往包含需要专业判断的部分。多个引用都可能合理，回答的详细程度也可能因使用场景而异。把这类争议偷偷改成一个分数，会掩盖判断责任；把所有判断都留给人工，又难以稳定执行。
文件确实没有修改，通常不是表达偏好的争议。一个政策引用无法支持当前天数，也不是因为评审者不喜欢措辞。应先保留这些可确定的合同，再把规则覆盖不了的表达问题交给审查。
本实验的 review 候选会给输出标记
needs_review=true。它是工作流演示，不证明现实中存在多少争议，也不表示已经完成真实人工标注。任何待审项都会阻止发布，避免候选通过自称“待审”逃避质量要求。
lab.review 要求 decisions 文件携带原报告原始字节的
SHA-256。决定中包含样本 ID、reviewer、decision 与
rationale。若报告换了候选、答案或其他内容，旧决定不能悄悄套上去。
仅有 reviewer 字符串不提供身份认证。随书代码记录的是审计形状，不是组织权限系统；生产中需要认证身份、角色授权、时间与签署机制。书中出现的教学评审者不能当作真实专家参与研究的证据。
程序只接受 accept_presentation 与
reject_presentation。接受表达可以关闭该样本的待审状态，却不能把
task_pass=false 改为
true，也不能撤销保护文件被改动的安全失败。拒绝表达则会使质量条件失败。
这个设计有意限制裁决范围。若确定性验收器本身错误，应通过有版本的标签或评分器修正重新评估，而不是在人工表里用“看起来可以”覆盖产物证据。两种操作有不同影响，必须分别保存。
真实评审应给出清楚量表、正反例、盲化方式与冲突升级路径。可对部分输出重复标注，分析一致与分歧；不过一致不等于正确，不同专业背景可能共享同一个偏差。
本版没有真实评审者研究，没有报告人类一致率或模型裁判与人类的一致率。六个校准反例由作者构造并用产物事实核查；它们的角色是测试代码行为，而非估计现实评审群体的意见分布。
审核工具输出新报告，并记录来源报告哈希与决定列表，不改写原文件。这样可以同时查看“机器检查与待审状态下是什么结论”，以及“追加表达决定之后是什么结论”。
如果一份报告在表达审查后仍因产物回归被拒绝，应如实保留。审查不是给发布找一个签字，而是补足合同里确实需要人的判断。每一次例外都应该能解释它改变了什么，又没有改变什么。
练习。 为一条待审输出构造两种表达决定。先故意改错报告哈希，确认工具拒绝；再使用正确哈希，检查任务与安全字段仍保持原样。不要把这项模拟称为真实人工校准实验。
发现回归以后，最急迫的问题通常是“为什么”。总体差值只能指出结果变化，归因需要控制比较。一次性更换所有组件，再凭经验挑一个解释，无法区分假设与证据。
随书实验固定数据、参考和评分器，分别运行三种变体。prompt_v2
改请求归一化，让带工单号的文件指令提取真正的目标数值；retrieval_v2
取消文档状态过滤；model_v2
注入只宣称完成或替换引用的生成器缺陷。
这些都是确定程序中的受控变化，不是真实 LLM 的提示效果研究。名称用于展示变更进入流水线的位置。真实模型接入后，必须重新运行数据与重复实验，不能继承这里的数值。
请求归一化修复三条文件任务，对知识回答没有改变。检索变体把旧政策排在可用资料首位，六条知识回答都退步，原来的三条文件失败仍存在，所以失败记录共有九条，而新回归不是九条。
生成器变体的文件正文仍说 done，真实配置未改；知识回答数值可以正确，引用却来自旧文档。它制造了任务层的全面失败，但最低表达指标不一定下降。若只看一项质量分，就可能选错组件继续优化。
共同失败说明两边都没有解决的问题；新回归说明候选损坏了基线已具备的行为。二者都值得研究，但在发布中承担不同角色。候选修复若干旧问题的收益，也不能自动抵消新出现的保护条件失败。
报告中的逐样本 reason 是分类线索，不应被当作完整根因证明。例如
wrong_fact_or_citation
包含事实与引用两类机制；进一步分析要查看响应和参考。真实系统可继续细分，但分类必须与实际可观察证据对应。
同一个提示可能在不同模型上产生不同作用，检索文档长度也会改变上下文预算。单因素对照适合定位局部变化，不能保证多个改进组合起来仍然改善。
重要组合可以增加配对的因子实验，事先选择需要研究的交互。若比较次数很多，需要控制选择偏差并保留独立确认。不要把所有组合都试一遍，然后只发表最高分配置而省略搜索过程。
确认根因之后，把最小输入、触发条件和验收证据保存为开发集回归样本，并记录它来自哪个故障。若它已经被用于修复，就不应继续称为未曝光验收题。
验证修复要回到同一失败条件，同时检查附近的正常场景，避免只记住一个 ID 或固定答案。当前数据生成器按规则构造项目值，便于改变实体和值进行邻近实验；这仍然不是对真实业务广泛有效性的证明。
练习。 在报告里找出检索变体的六个新回归和三个共同失败。分别提出一个可验证的根因假设。再把请求归一化修复与检索缺陷组合，说明为什么不能直接相加两个单独报告的总分。
发布门槛不是报告最后一行的装饰。它应该真正影响进程退出状态，使 CI 能阻止不满足合同的候选，同时把失败原因留给维护者。绿色检查还需要对应明确范围，不能借此宣称整个产品已安全可靠。
随书 release_gate()
首先检查是否为验收集，以及配对是否完整、候选是否有安全或未知失败、是否仍有待审项。开发集上的漂亮表现会被
not_acceptance_split
拒绝；预算拒绝和超时不能从分母里消失。
这类门槛表达的是证据可用性。假如运行不完整，即使计算得到候选与基线相同的差值，也不应继续把它解释成非劣证据。把检查顺序写清楚，有助于团队先补实验，再争论分数。
当前教学政策拒绝任何新的任务回归与最低表达回归，并检查项目差值百分位区间的下界不低于 -0.05。这里样本级规则比整体区间更严格；保留两项是为了展示不同层次的决策依据，不应把它们误称为经过统计功效设计的业务检验。
真实项目可能允许某些低风险场景轻微退化，以换取更重要收益。这样的权衡应事先按业务范围规定；不能看到新版本失败以后，再把该样本降级成无关紧要。
python3 -m lab.pipeline --candidate retrieval_v2 \
--output /tmp/quality-blocked.json --require-pass该命令实际以状态码 2 结束。报告包括
new_task_regression、new_answer_quality_regression
和 paired_interval_below_noninferiority_margin。省略
--require-pass
时，程序仍生成失败报告，但命令本身仅承担分析任务，退出为
0；两种使用方式需要区分。
CI 的实验测试包含对失败退出码的检查，避免未来重构只打印 FAIL 却继续成功。完整 demo 则期望一个修复候选通过、两个缺陷候选失败；它自身成功表示这些预期行为被验证，并不表示三个候选都适合发布。
至少保留输入和答案指纹、候选及评分代码版本、完整配对结果、配置、预算状态、人工决定、门槛版本与原因。对含私有数据的业务报告，证据应分级存储，公开报告可以只保留定位符和汇总。
本实验完整响应可公开，是因为数据全部是合成的；报告本身没有实现访问控制、签名、不可篡改日志或审批身份系统。哈希便于发现字节变化，但不证明谁有权签署该版本。
发布记录应关联实际部署配置、回退目标和监测窗口。评测通过后部署了另一个提示或索引版本，先前报告便不能直接覆盖它。回退后也要确认旧资源仍然可用，权限与数据合同没有被迁移破坏。
练习。 去掉 --require-pass
运行缺陷候选，观察报告与退出码的区别。然后为自己的 CI
写下“哪些步骤允许只产出分析、哪些步骤必须阻止发布”，避免把一份失败报告误当作一个失败的工作流。
现在把前面的数据、候选、评分与门槛连起来。本章所有输出来自随书程序实际运行；可变化的时间与本机经过毫秒不当作固定答案。先保留原始报告，再选择其中需要解释的字段。
代码要求 Python 3.10+，没有第三方运行依赖、真实模型密钥或联网调用。请从仓库根目录执行：
python3 -m unittest discover -s tests -v
python3 -m lab.demo本版实跑 17 项测试通过。demo 创建
lab/evidence/，保存三份完整比较与
summary.json。它不是只运行三个评分函数：每条任务、每个候选都会进入新的临时工作区，由独立
Python 子进程生成真实响应与文件，再由父进程验收。
lab/data/manifest.json 记录训练资料 6 条、开发 8
条、验收 12
条。正常使用不需要重新生成数据；python3 -m lab.make_data
是维护命令，会重建数据文件及版本清单。修改样本后应正式更新版本说明，不能只靠重算哈希掩盖改变。
下面节选自 prompt_v2 的实际摘要：
{
"candidate": "prompt_v2",
"cases": 12,
"statistics": {
"paired_groups": 6,
"difference": 0.25,
"percentile_interval_95": [0.08333333333333333, 0.4166666666666667],
"seed": 17,
"resamples": 2000
},
"gate": {"passed": true, "reasons": [], "policy": "fixture-gate-v1"},
"failure_count": 0
}这表示请求归一化替身修复了已知的数字提取问题，且满足当前合成验收门槛。报告同时标注没有真实模型调用、隔离不是 OS 沙箱、小样本不是总体研究。展示正区间时不能把这些限制裁掉。
完整报告里的每一行都包含 baseline 和 candidate。选取一条带 Ticket 的文件任务，对照初始配置、指令中的工单号、目标值与两边产物哈希，可以看出究竟修复了什么，而非只读总分。
| 候选 | 实际差值 | failure_count | 发布结果 |
|---|---|---|---|
retrieval_v2 |
-0.50 | 9 | 拒绝 |
model_v2 |
-0.75 | 12 | 拒绝 |
检索变体的九条失败包含六条新出现的回答回归和三条基线已有的文件失败。不要把 failure_count 直接当作新增回归数。生成器变体十二条任务都没有满足最终任务合同，但一些回复仍通过最低表达检查。
完整 demo 自身应当正常退出，因为它验证了这两个缺陷确实被拒绝。若你删除某个重要评分条件，导致坏候选通过，demo 的断言反而会失败。
python3 -m lab.pipeline --candidate prompt_v2 --split acceptance \
--output /tmp/quality-prompt.json --require-pass命令保存完整报告，并把发布决定映射为进程退出状态。将候选改为
retrieval_v2，应该得到报告和退出码 2。把 split 改为
dev，即使修复表现不错，也会被验收范围检查拒绝。
如果报 dataset_hash_mismatch，先核查数据是否被改动，不要立即跳过验证。若候选超时，查看单项 reason 和未知预留，而不是将它改写为普通答案错误。若缺少某个样本 ID，核查清单、过滤与标签完整性。
可以先将预算降到 1，检查所有计划任务仍在报告中、没有消耗被伪造、门槛拒绝不完整配对。再恢复预算，运行原候选，确认先前拒绝没有修改数据或候选状态。
不要为了复制书中的时间戳或 wall_ms 而调程序。真正应稳定的是数据哈希、候选逻辑、样本级判定与固定 seed 下的演示统计。运行环境差异应被记录，而非藏起来。
本章针对容易被成功演示隐藏的几种情况做验证。每个实验都有明确故障与可观察结果，避免把内部函数形状当作唯一证据。需要注意，以下仍是合作式程序与合成任务，不是恶意代码安全测试或真实人工研究。
python3 -m unittest discover -s tests -v -k claimed_completion
python3 -m unittest discover -s tests -v -k citation_identity第一项核对生成器虽返回 done，真实文件仍不满足目标；第二项核对答案数值和消息看似合理，却没有引用支持它的当前资料。它们都会使任务验收失败，而最低表达指标仍可为 true。
这两项测试防止“让另一个简单字符串规则给最终回答评分”替代实际任务验收。对于真实模型输出，也应保留相同证据链；不能因为换成模型裁判，就删除文件、引用和权限条件。
python3 -m unittest discover -s tests -v -k calibration校准真正调用两种规则，不是只打印一张手写矩阵。简陋规则在四个应拒绝反例中放过两个；合同规则在六个构造反例中全部判断正确。你可以打开
summary.json 的
calibration.cases，逐条查看响应与参考判断。
改参练习是增加“正确数字出现在一段否定句中”的反例。当前最低表达规则只做出现性检查，不会理解否定语义，新的反例应暴露这个范围。正确做法是修订表达合同或交给经校准的更合适判断，不是声称现有规则已经理解自然语言。
python3 -m unittest discover -s tests -v -k timeout测试用受控候选延迟触发子进程超时。父进程终止直接子进程，将运行标为不可评分与 usage unknown，报告保留预留，并阻止发布。这里没有真实费用账单；保留预留是为了验证资源承诺在未知结果下不会被无依据地释放。
程序没有完整的恶意后代进程管理和 OS 资源沙箱，不能用这项测试证明不可信程序已经被彻底清理。真实执行服务需要独立身份、进程组与资源边界。
先产生待审报告，再为它生成明确标注为模拟的决定文件：
python3 -m lab.pipeline --candidate review --output /tmp/pending-quality.json
python3 - <<'PY'
import hashlib, json
from pathlib import Path
p = Path('/tmp/pending-quality.json')
r = json.loads(p.read_text())
d = {'report_sha256': hashlib.sha256(p.read_bytes()).hexdigest(),
'decisions': [{'sample_id': x['id'], 'decision': 'accept_presentation',
'reviewer': 'teaching-simulation',
'rationale': 'Workflow demonstration; not real human evaluation'}
for x in r['rows']]}
Path('/tmp/quality-decisions.json').write_text(json.dumps(d))
PY
python3 -m lab.review --report /tmp/pending-quality.json \
--decisions /tmp/quality-decisions.json --output /tmp/reviewed-quality.json裁决之后，unresolved_review
被解除，任务与安全判定保持原样。这不是“人工批准使候选变正确”，而是演示审核只能改变指定层次。若把
decisions 的 report_sha256 改错，工具必须拒绝，防止跨版本误用。
预算测试保留全部 12 个计划样本；超时测试保存未知消耗；审核输出新报告而非覆盖旧文件。三者共同避免“为了总分更好而删行”的路径。
新增故障以后，应记录它属于数据、候选、验收、表达、资源还是发布策略问题。能够指出是哪一层需要改变，比在所有地方一起调整阈值更有价值。
随书流水线把基础合同跑通，不代表真实模型已经被评估。把确定候选换成模型 API，会引入随机性、端点版本、流式中断、费用、外部状态和更复杂的表达。现有结果的数值不能直接沿用。
适配器接收公开任务、允许的证据和工作区，产生结构化响应与实际产物。私有参考仍留在评分侧。不要为了方便把答案拼进提示，也不要让候选获取评分脚本写权限。
对文件任务，模型可以提出修改，由受控执行层写入工作区，最后读取真实产物验收；对知识回答，可保留结构化事实与引用字段，另收集自然语言回答以做独立表达判断。现有 task、quality、safety 分项应继续存在。
本书没有提供任何供应商 API 的可执行适配，没有配置真实 key，也没有验证特定模型支持结构化输出或工具调用。接入者需要按所选服务当前官方合同另行实现认证、协议翻译与错误处理，并保存版本和运行证据。
同一任务重复采样可以估计运行波动，多个独立项目用于研究场景差异。把同一任务的十次输出当作十个独立客户，会低估依赖性。报告应保留项目、任务与重复层级，选择与估计对象相符的方法。
如果服务没有稳定 seed 或版本固定能力，应如实记录。冻结提示和参数只能控制部分变化，不能保证供应商后端完全相同。真实实验可能需要交错运行候选，避免时段、负载和缓存影响。
真实用量可能包括输入、输出、缓存、图像、工具结构或服务内部计量，字节单位不能充当它们的通用替代。应区分本地估算与服务端确认，并处理重试、失败请求和账单迟到。
没有最终用量时，应保留 unknown，而非将它填为零。供应商是否对某类失败计费，需要按所选合同核对。流水线的预算上限还应包含并发预留与评测停止策略，防止昂贵的异常持续扩大。
真实模型会产生代码块、附加解释、拒答、格式变体与冲突陈述。当前字段检查只能覆盖约定子集。新增解析容错可能引入误判，因此必须先在固定产物上校准，再比较候选。
模型裁判应配有代表性人工参考和反例，独立考察错误放行、错误拒绝、顺序影响以及对不可信内容的反应。它不能替代本来能确定检查的产物，也不应在缺少校准时与生产门槛直接绑定。
可以先运行少量只读、低费用任务，核对端点、用量和结构；再扩展工具动作、长任务和更复杂文档。每一阶段明确可以停止的条件，保存失败与未覆盖范围。
只有在任务样本、标签、隔离、评分、统计与成本都具有足够证据后，才讨论业务发布结论。当前教学管线为这些接口提供可审查起点，没有替你完成剩下的工作。
练习。 写一份真实模型适配计划，分别列出已经有依据的接口、需要查询的供应商合同、需要实跑的异常路径和不允许沿用的教学数字。不要先填一个模型名称再把其余保证默认补齐。
质量交付不会在一次发布之后结束。用户、文档、任务与模型服务会变化，旧样本会过期，评分器也可能暴露新的盲点。一个长期有用的评测系统需要清楚的数据维护、运行与发布责任。
线上问题先保存受控的现场与最小复现，确认授权和隐私处理后进入开发回归。用它修复系统以后，它已经是开发知识，不宜继续称为隐藏验收题。重要能力可以追加新的独立场景作为后续确认。
失败分类应服务下一步动作：标签错就修数据；产物不符合目标就查候选与执行；引用失效就查资料和检索；评分误判就校准验收；预算拒绝就补运行计划。不要把所有问题都变成一句“提示不够强”。
当政策事实发生变化，旧报告可能仍然正确描述当时合同，但不再适用于当前业务。应保留时间与版本，明确重评范围。重评旧产物可以帮助研究变化，却不等于旧候选在新世界里已经重新执行成功。
评分器升级也应先跑固定反例与历史产物，解释分数变化。删除一种检查可能让失败减少；新增检查可能让旧报告变差。两者都不应直接归因为模型变化。
生产化需要持久任务队列、可靠产物存储、访问控制、候选隔离、运行配额、审计与恢复。报告签名与审批身份属于另一层保证，不能由一个 SHA-256 字符串自动提供。
随书代码用单进程、临时目录和合作式候选，适合在个人环境与 CI 里复现列明行为。它没有多节点调度、敌对代码沙箱、认证审查门户或真实费用对账。设计生产服务时，应对这些缺口分别立项，而不是把演示脚本直接开放到公网。
一次候选交付至少应包含：使用范围与变更说明，数据与参考版本，评分器及候选指纹，完整逐样本比较，统计方法和限制，预算与异常情况，待审和裁决记录，发布门槛与最终决定，以及实际部署和回退配置。
本书自身也沿这条链路交付。Markdown 同时生成网页与 EPUB；校验器逐章比较正文、链接、搜索和代码包；浏览器检查全部章节的手机布局、主题与无 JavaScript 阅读；EPUBCheck 检查格式。它们各有范围，不能用格式通过替代事实或实验审阅。
选择一个真实但可以合法使用的小场景，建立来源清楚的训练资料、开发与验收切分。先保留确定性基线，再增加一个受控变化。用独立产物证据验收，制作至少三种能骗过旧评分器的反例，保存逐样本配对变化。
接着定义独立单位与估计对象，说明当前数据不足以支持哪些结论；为预算拒绝、超时、未知与人工争议写清停止规则。只有门槛确实能拒绝坏候选，交付包又能解释失败，才允许进入下一阶段真实模型实验。
这项任务的完成标准不是得到一个更高的平均数，而是其他开发者能够重现一次修复、一种回归和一个被正确阻止的发布，并明确知道仍有哪些保证没有建立。
第 2 章样本卡至少应有稳定 ID、公开输入、参考证据、保护约束、来源与版本。标签修正必须创建新版本或有明确补丁记录，旧报告保留其原用标签，不能只更新答案而让历史结论失去上下文。
第 3 章同一政策的不同问法仍共享来源，应先按文档版本、组织或项目等真实关联切分。时间边界还要检查重复抓取和后续摘要。分数下降可能来自更严格的独立性，不能只按新旧百分比判断系统变化。
任务成功、表达与保护条件应分别记录。配置已改但回复空，可以是任务通过而表达失败；回复 done 而文件未改，则是表达最低合同通过而任务失败。评分器的正反例应来自可独立检查的产物，不能直接复制它自己输出的标签。
混淆矩阵本书以应通过为正类。假阳性率是 FP/(FP+TN)，不是 FP/全部样本；本实验为 2/4。六个反例上合同规则全部正确，只说明这些已构造情形被识别，不是总体误判率为零。
按任务复制重复记录，会改变任务等权结果，却未增加项目独立信息。重采样应保留候选配对与项目内关联，说明估计对象。区间退化为一点，只表示当前经验效应没有变化，不能证明真实总体完全确定。
开发集运行不能直接通过验收门槛；预算拒绝应保留计划行。省略
--require-pass 允许程序仅生成报告，并不改变报告中的
gate=false。CI 真正阻止发布需要使用该参数或检查报告决定。
模拟裁决应绑定报告哈希，限制影响层次。接受表达不能把错误产物改为正确；评分器或标签确实有错时，应走有版本的修订与重评。检索变体九条失败中只有六条是新回归，其余三条基线也失败。
ModuleNotFoundError 通常表示没有在仓库根目录执行
python3 -m ...。数据哈希不匹配时，先检查文件改动与版本说明。真实模型适配未提供，不应寻找隐藏的
API key
参数。子进程不能直接导入私有评分模块是刻意的输入分离，但不等于宿主已经有敌对代码隔离。
如果测试失败，保留完整名称、环境、命令和报告；不要直接调大阈值或删除样本。先判断失败属于环境、数据、候选、评分、预算还是发布合同，再修改对应责任层。
本版于 2026-10-21 读取下列第一方资料，保存获取内容的 SHA-256 与结论映射。资料原文权利属于各自权利人，仓库不打包完整文档副本；以下为独立摘要，不能替代原始说明。
来源：NIST AI 100-1。重点核对 MEASURE 功能、评测背景、测试集和工具的记录、不确定性与持续测量。AI RMF 是自愿使用的风险管理框架，不是本实验取得的认证，也没有替本书规定 -0.05 发布阈值。
本文关于发布责任、证据范围与持续维护的组织方式属于工程推导。没有声称通过随书测试即符合某项法规、行业规范或业务安全标准。
来源：Inspect 官方评分器说明。文档区分文本提取/匹配、自定义规则和模型评分，并将度量汇总与评分器连接。它支持本书分离评分机制与指标的讨论。
随书代码没有安装或运行 Inspect，也没有测量其框架行为。当前最低表达合同不能当作任意自然语言质量评分，模型裁判接入与校准仍是未运行的扩展任务。
来源：Inspect 官方数据集说明。核对 Sample 中 input、target、id、metadata 与文件/环境设置等职责。本文借此说明公开任务、评分参考和执行环境需要分别建模。
本书自己的临时目录方案不是 Inspect 沙箱实现，也没有因为数据字段分开就获得答案保密保证。公开合成题与同身份子进程的局限在正文明确列出。
来源：SciPy bootstrap 官方文档。核对有放回重采样、paired 参数、百分位与 basic/BCa 方法的区别，以及退化分布的限制提示。
本实验用标准库实现项目分组的成对百分位重采样，没有调用 SciPy，没有 BCa，也没有证明小样本区间具有良好业务覆盖率。文档本身也不支持用缺乏代表性的合成题推断真实模型总体表现。
lab/evidence/
保存三个受控候选的完整逐样本报告与校准摘要，所有报告标注无真实模型调用。book/lab-results.json
汇总实际运行，发布验证记录保存命令与格式/页面验收范围。真实云模型调用、真实人工一致性研究、供应商排名与生产部署均未执行。
| 术语 | 本书含义 |
|---|---|
| 任务验收 | 根据真实产物、结构化事实与证据判断目标是否满足 |
| 表达合同 | 本实验非空消息和约定数值出现性检查；不是完整语言质量 |
| 校准 | 在有独立参考判断的固定反例上检查评分器误判 |
| 配对 | 同一任务在相同参考与评分条件下保留两边候选结果 |
| 项目组 | 两种任务共享的构造项目，统计时作为重采样单位 |
| 估计对象 | 本书采用项目等权的任务成功差值均值 |
| 百分位区间 | 重采样经验分布的指定分位数；不是模型自评置信度 |
| unknown | 缺少完整输出或用量证据，不能当作正常评分或零消耗 |
| 验收切分 | 程序按清单隔离的任务分组；公开教学题不是秘密基准 |
| 人工裁决 | 对具体报告中表达争议追加决定，不覆盖产物事实 |
| 发布门槛 | 事先规定的完整性、安全、回归与区间检查政策 |
| 指纹 | 字节哈希，用于发现变化；不提供身份认证与签署授权 |
本书没有训练模型、调用真实模型服务或验证其 token 计费，没有 OS 沙箱、生产身份系统、多节点评测服务、不可篡改审计或真实人工研究。实验验证的是公开、受控、合作式实现中的列明行为。
以下文件在构建时从随书实验与测试目录读取，与下载区实验包同源。它们是本书教学实现；实际验证范围以实验章节和验证记录为准。
# 可运行质量交付实验
Python 3.10+，只用标准库，不联网，不需要 API key。从仓库根目录执行：
```sh
python3 -m unittest discover -s tests -v
python3 -m lab.demo
```
本版 17 项测试。demo 实际执行子进程候选、读取文件和回答产物、评分、成对项目重采样、预算与发布门槛，并保存 `lab/evidence/{prompt_v2,retrieval_v2,model_v2,summary}.json`。已记录结果为修复候选通过、检索和生成缺陷候选拒绝。
## 数据合同
`data/samples.json` 包含 26 个合成任务；`data/answers.json` 是独立参考；`data/manifest.json` 记录版本、文件 SHA-256 和切分。训练资料 6 条/3 项目、开发 8 条/4 项目、验收 12 条/6 项目，项目不跨切分。没有训练真实模型；公开小题也不能当作秘密验收集。
文件任务读取修改后的 `service.cfg` 并验证 `policy.lock` 未改。知识回答检查结构化 `days` 和真正支持事实的当前文档 ID。表达合同只检查非空与约定数值出现，不评估任意自然语言的语义、流畅度或文采。`task_pass/quality_pass/safety_pass/review_pending` 分别保存。
数据重建命令 `python3 -m lab.make_data` 用于维护，正常运行无需执行。数据或参考改变时必须更新版本说明并核对清单，不应仅重算哈希以掩盖变化。
## 受控候选
| 模式 | 可执行行为 |
| --- | --- |
| baseline | 朴素数字提取；仅使用当前政策 |
| prompt_v2 | 请求归一化修复数字提取，不代表真实LLM提示效果 |
| retrieval_v2 | 删除文档状态过滤，暴露旧政策回归 |
| model_v2 | 不修改文件却说done，或返回错误引用；不是云模型 |
| unsafe | 修改保护文件 |
| terse | 产物可正确，但消息为空 |
| review | 标记表达待审，阻止发布 |
| timeout | 受控延迟触发子进程超时 |
## 隔离与未实现保证
每次候选有新的临时cwd，只有候选脚本、公开task.json与样本文件。父进程持有参考与评分器，不把它们传给子进程；`-I` 减少解释器导入环境影响。stdout/stderr不进入报告。子进程仍与父进程使用同一OS身份，可以访问其他宿主路径；这不是OS沙箱，不适合运行陌生恶意候选。没有网络/磁盘/内存/后代进程硬隔离。
答案全部公开，公开任务也有明确规律。该数据集只验证管线与已知缺陷，不排名真实模型，不证明泛化、用户收益或验收题从未曝光。
## 报告、统计与预算
```sh
python3 -m lab.pipeline --candidate prompt_v2 --split acceptance \
--output /tmp/quality-report.json --require-pass
```
报告保存原始逐样本对照、分项、原因、产物哈希、候选/流水线/数据指纹与预算。统计对象是项目等权的任务成功率差；按项目保留配对并有放回抽取，seed=17、2000次、95%百分位区间。没有BCa或业务总体覆盖率保证；6组合成项目仅演示方法，退化区间不代表真实世界没有不确定性。
预算单位为公开任务JSON字节数+响应JSON字节数，不是token或货币。每次先预留输入+4096最大输出，完整有效输出按实际结算；超时/非法输出保留unknown预留。预算拒绝样本不删行，完整性门槛阻止发布。wall_ms是本机进程经过时间，不是模型延迟基准。
门槛要求验收切分、完整配对、无安全/未知问题、无待审、新任务/表达回归，并检查演示区间下界不低于-0.05。它是教学政策，不是通用业务非劣检验。带 `--require-pass` 时失败退出2；省略时仍保存失败报告但退出0。demo期待缺陷被拒绝，因此demo整体可以通过。
## 校准与审查
`calibration()` 真正对六个作者构造反例执行简陋文字规则与合同规则。参考由可机械核查的产物和引用确定，不是真实人类标注研究。实跑简陋规则 TP2/FP2/TN2/FN0，合同规则 TP2/FP0/TN4/FN0；不能将六例零错外推为总体零误判。
`python3 -m lab.review --report ... --decisions ... --output ...` 追加表达裁决。decisions必须绑定原报告SHA-256，每项含sample_id/decision/reviewer/rationale；只接受accept_presentation/reject_presentation，不修改任务和安全证据。reviewer是自声明教学身份，无认证系统。完整模拟命令见正文第14章。
没有真实供应商API适配、模型裁判、真实人工一致率、公开服务身份、持久任务调度、多实例账本、签名审计或真实费用对账。这些扩展在正文第15–16章说明，未伪装成已经运行的功能。"""Executable quality-delivery teaching experiments, without real model calls.""""""Cooperative deterministic candidates, copied into a fresh temporary directory.
These are executable baselines and injected defects, not real LLM adapters.
No reference labels, grader or source repository path are supplied here.
"""
from pathlib import Path
import json
import re
import sys
import time
def main():
mode = sys.argv[1]
task = json.loads(Path("task.json").read_text())
if mode == "timeout":
time.sleep(5)
if task["kind"] == "file":
if mode == "prompt_v2":
# A controlled request-normalization change. It does not measure
# how a real language model responds to a different prompt.
match = re.search(r"timeout_ms to (\d+)", task["instruction"])
else:
match = re.search(r"\d+", task["instruction"])
value = int(match[0] if mode != "prompt_v2" else match[1])
if mode != "model_v2":
Path("service.cfg").write_text(f"timeout_ms={value}\nretries=2\n")
if mode == "unsafe":
Path("policy.lock").write_text("overwritten\n")
result = {"message": "done", "needs_review": mode == "review"}
if mode == "terse":
result["message"] = ""
else:
docs = task["documents"]
permitted = docs if mode == "retrieval_v2" else [d for d in docs if d["status"] == "current"]
document = permitted[0]
citations = [document["id"]]
if mode == "model_v2":
citations = [docs[0]["id"]]
result = {"message": f"The refund window is {document['days']} days.",
"days": document["days"], "citations": citations,
"needs_review": mode == "review"}
if mode == "terse":
result["message"] = ""
Path("response.json").write_text(json.dumps(result))
if __name__ == "__main__":
main(){"acceptance-project-0-answer":{"days":60,"supporting_ids":["acceptance-project-0-current"]},"acceptance-project-0-file":{"expected_file":"timeout_ms=60\nretries=2\n","protected":"DO NOT MODIFY\n"},"acceptance-project-1-answer":{"days":61,"supporting_ids":["acceptance-project-1-current"]},"acceptance-project-1-file":{"expected_file":"timeout_ms=61\nretries=2\n","protected":"DO NOT MODIFY\n"},"acceptance-project-2-answer":{"days":62,"supporting_ids":["acceptance-project-2-current"]},"acceptance-project-2-file":{"expected_file":"timeout_ms=62\nretries=2\n","protected":"DO NOT MODIFY\n"},"acceptance-project-3-answer":{"days":63,"supporting_ids":["acceptance-project-3-current"]},"acceptance-project-3-file":{"expected_file":"timeout_ms=63\nretries=2\n","protected":"DO NOT MODIFY\n"},"acceptance-project-4-answer":{"days":64,"supporting_ids":["acceptance-project-4-current"]},"acceptance-project-4-file":{"expected_file":"timeout_ms=64\nretries=2\n","protected":"DO NOT MODIFY\n"},"acceptance-project-5-answer":{"days":65,"supporting_ids":["acceptance-project-5-current"]},"acceptance-project-5-file":{"expected_file":"timeout_ms=65\nretries=2\n","protected":"DO NOT MODIFY\n"},"dev-project-0-answer":{"days":30,"supporting_ids":["dev-project-0-current"]},"dev-project-0-file":{"expected_file":"timeout_ms=30\nretries=2\n","protected":"DO NOT MODIFY\n"},"dev-project-1-answer":{"days":31,"supporting_ids":["dev-project-1-current"]},"dev-project-1-file":{"expected_file":"timeout_ms=31\nretries=2\n","protected":"DO NOT MODIFY\n"},"dev-project-2-answer":{"days":32,"supporting_ids":["dev-project-2-current"]},"dev-project-2-file":{"expected_file":"timeout_ms=32\nretries=2\n","protected":"DO NOT MODIFY\n"},"dev-project-3-answer":{"days":33,"supporting_ids":["dev-project-3-current"]},"dev-project-3-file":{"expected_file":"timeout_ms=33\nretries=2\n","protected":"DO NOT MODIFY\n"},"train-project-0-answer":{"days":10,"supporting_ids":["train-project-0-current"]},"train-project-0-file":{"expected_file":"timeout_ms=10\nretries=2\n","protected":"DO NOT MODIFY\n"},"train-project-1-answer":{"days":11,"supporting_ids":["train-project-1-current"]},"train-project-1-file":{"expected_file":"timeout_ms=11\nretries=2\n","protected":"DO NOT MODIFY\n"},"train-project-2-answer":{"days":12,"supporting_ids":["train-project-2-current"]},"train-project-2-file":{"expected_file":"timeout_ms=12\nretries=2\n","protected":"DO NOT MODIFY\n"}}{
"version": "synthetic-quality-v1",
"synthetic": true,
"purpose": "Test quality pipeline behavior, not model intelligence or vendor ranking",
"splits": {
"train": {
"samples": 6,
"groups": 3
},
"dev": {
"samples": 8,
"groups": 4
},
"acceptance": {
"samples": 12,
"groups": 6
}
},
"sha256": {
"samples.json": "684c829ea8ed4a07c5f47caa60df2a5548b6c7942ec364a30b18257028907727",
"answers.json": "970d44538c5649961c20dcd0e536100020e247ee91d078c68a22b53245141d59"
}
}[{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"train-project-0","id":"train-project-0-file","instruction":"Ticket 9000: Set timeout_ms to 10","kind":"file","origin":"authored-synthetic-v1","project":"train-project-0","split":"train"},{"documents":[{"days":7,"id":"train-project-0-old","status":"superseded","text":"An old policy for train-project-0: refund window is 7 days."},{"days":10,"id":"train-project-0-current","status":"current","text":"The current policy for train-project-0: refund window is 10 days."}],"group":"train-project-0","id":"train-project-0-answer","instruction":"How many days is the current train-project-0 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"train-project-0","split":"train"},{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"train-project-1","id":"train-project-1-file","instruction":"Set timeout_ms to 11","kind":"file","origin":"authored-synthetic-v1","project":"train-project-1","split":"train"},{"documents":[{"days":7,"id":"train-project-1-old","status":"superseded","text":"An old policy for train-project-1: refund window is 7 days."},{"days":11,"id":"train-project-1-current","status":"current","text":"The current policy for train-project-1: refund window is 11 days."}],"group":"train-project-1","id":"train-project-1-answer","instruction":"How many days is the current train-project-1 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"train-project-1","split":"train"},{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"train-project-2","id":"train-project-2-file","instruction":"Ticket 9002: Set timeout_ms to 12","kind":"file","origin":"authored-synthetic-v1","project":"train-project-2","split":"train"},{"documents":[{"days":7,"id":"train-project-2-old","status":"superseded","text":"An old policy for train-project-2: refund window is 7 days."},{"days":12,"id":"train-project-2-current","status":"current","text":"The current policy for train-project-2: refund window is 12 days."}],"group":"train-project-2","id":"train-project-2-answer","instruction":"How many days is the current train-project-2 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"train-project-2","split":"train"},{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"dev-project-0","id":"dev-project-0-file","instruction":"Ticket 9000: Set timeout_ms to 30","kind":"file","origin":"authored-synthetic-v1","project":"dev-project-0","split":"dev"},{"documents":[{"days":7,"id":"dev-project-0-old","status":"superseded","text":"An old policy for dev-project-0: refund window is 7 days."},{"days":30,"id":"dev-project-0-current","status":"current","text":"The current policy for dev-project-0: refund window is 30 days."}],"group":"dev-project-0","id":"dev-project-0-answer","instruction":"How many days is the current dev-project-0 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"dev-project-0","split":"dev"},{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"dev-project-1","id":"dev-project-1-file","instruction":"Set timeout_ms to 31","kind":"file","origin":"authored-synthetic-v1","project":"dev-project-1","split":"dev"},{"documents":[{"days":7,"id":"dev-project-1-old","status":"superseded","text":"An old policy for dev-project-1: refund window is 7 days."},{"days":31,"id":"dev-project-1-current","status":"current","text":"The current policy for dev-project-1: refund window is 31 days."}],"group":"dev-project-1","id":"dev-project-1-answer","instruction":"How many days is the current dev-project-1 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"dev-project-1","split":"dev"},{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"dev-project-2","id":"dev-project-2-file","instruction":"Ticket 9002: Set timeout_ms to 32","kind":"file","origin":"authored-synthetic-v1","project":"dev-project-2","split":"dev"},{"documents":[{"days":7,"id":"dev-project-2-old","status":"superseded","text":"An old policy for dev-project-2: refund window is 7 days."},{"days":32,"id":"dev-project-2-current","status":"current","text":"The current policy for dev-project-2: refund window is 32 days."}],"group":"dev-project-2","id":"dev-project-2-answer","instruction":"How many days is the current dev-project-2 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"dev-project-2","split":"dev"},{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"dev-project-3","id":"dev-project-3-file","instruction":"Set timeout_ms to 33","kind":"file","origin":"authored-synthetic-v1","project":"dev-project-3","split":"dev"},{"documents":[{"days":7,"id":"dev-project-3-old","status":"superseded","text":"An old policy for dev-project-3: refund window is 7 days."},{"days":33,"id":"dev-project-3-current","status":"current","text":"The current policy for dev-project-3: refund window is 33 days."}],"group":"dev-project-3","id":"dev-project-3-answer","instruction":"How many days is the current dev-project-3 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"dev-project-3","split":"dev"},{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"acceptance-project-0","id":"acceptance-project-0-file","instruction":"Ticket 9000: Set timeout_ms to 60","kind":"file","origin":"authored-synthetic-v1","project":"acceptance-project-0","split":"acceptance"},{"documents":[{"days":7,"id":"acceptance-project-0-old","status":"superseded","text":"An old policy for acceptance-project-0: refund window is 7 days."},{"days":60,"id":"acceptance-project-0-current","status":"current","text":"The current policy for acceptance-project-0: refund window is 60 days."}],"group":"acceptance-project-0","id":"acceptance-project-0-answer","instruction":"How many days is the current acceptance-project-0 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"acceptance-project-0","split":"acceptance"},{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"acceptance-project-1","id":"acceptance-project-1-file","instruction":"Set timeout_ms to 61","kind":"file","origin":"authored-synthetic-v1","project":"acceptance-project-1","split":"acceptance"},{"documents":[{"days":7,"id":"acceptance-project-1-old","status":"superseded","text":"An old policy for acceptance-project-1: refund window is 7 days."},{"days":61,"id":"acceptance-project-1-current","status":"current","text":"The current policy for acceptance-project-1: refund window is 61 days."}],"group":"acceptance-project-1","id":"acceptance-project-1-answer","instruction":"How many days is the current acceptance-project-1 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"acceptance-project-1","split":"acceptance"},{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"acceptance-project-2","id":"acceptance-project-2-file","instruction":"Ticket 9002: Set timeout_ms to 62","kind":"file","origin":"authored-synthetic-v1","project":"acceptance-project-2","split":"acceptance"},{"documents":[{"days":7,"id":"acceptance-project-2-old","status":"superseded","text":"An old policy for acceptance-project-2: refund window is 7 days."},{"days":62,"id":"acceptance-project-2-current","status":"current","text":"The current policy for acceptance-project-2: refund window is 62 days."}],"group":"acceptance-project-2","id":"acceptance-project-2-answer","instruction":"How many days is the current acceptance-project-2 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"acceptance-project-2","split":"acceptance"},{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"acceptance-project-3","id":"acceptance-project-3-file","instruction":"Set timeout_ms to 63","kind":"file","origin":"authored-synthetic-v1","project":"acceptance-project-3","split":"acceptance"},{"documents":[{"days":7,"id":"acceptance-project-3-old","status":"superseded","text":"An old policy for acceptance-project-3: refund window is 7 days."},{"days":63,"id":"acceptance-project-3-current","status":"current","text":"The current policy for acceptance-project-3: refund window is 63 days."}],"group":"acceptance-project-3","id":"acceptance-project-3-answer","instruction":"How many days is the current acceptance-project-3 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"acceptance-project-3","split":"acceptance"},{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"acceptance-project-4","id":"acceptance-project-4-file","instruction":"Ticket 9004: Set timeout_ms to 64","kind":"file","origin":"authored-synthetic-v1","project":"acceptance-project-4","split":"acceptance"},{"documents":[{"days":7,"id":"acceptance-project-4-old","status":"superseded","text":"An old policy for acceptance-project-4: refund window is 7 days."},{"days":64,"id":"acceptance-project-4-current","status":"current","text":"The current policy for acceptance-project-4: refund window is 64 days."}],"group":"acceptance-project-4","id":"acceptance-project-4-answer","instruction":"How many days is the current acceptance-project-4 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"acceptance-project-4","split":"acceptance"},{"files":{"policy.lock":"DO NOT MODIFY\n","service.cfg":"timeout_ms=5\nretries=2\n"},"group":"acceptance-project-5","id":"acceptance-project-5-file","instruction":"Set timeout_ms to 65","kind":"file","origin":"authored-synthetic-v1","project":"acceptance-project-5","split":"acceptance"},{"documents":[{"days":7,"id":"acceptance-project-5-old","status":"superseded","text":"An old policy for acceptance-project-5: refund window is 7 days."},{"days":65,"id":"acceptance-project-5-current","status":"current","text":"The current policy for acceptance-project-5: refund window is 65 days."}],"group":"acceptance-project-5","id":"acceptance-project-5-answer","instruction":"How many days is the current acceptance-project-5 refund window?","kind":"answer","origin":"authored-synthetic-v1","project":"acceptance-project-5","split":"acceptance"}]"""Run three controlled comparisons and calibration; verify expected gate results."""
import argparse
import json
from pathlib import Path
from .pipeline import calibration, compare
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--output", type=Path, default=Path("lab/evidence"))
args = parser.parse_args()
args.output.mkdir(parents=True, exist_ok=True)
summary = {"kind": "controlled_fixture_no_real_models", "reports": [], "calibration": calibration()}
for name in ["prompt_v2", "retrieval_v2", "model_v2"]:
report = compare(name)
assert report["gate"]["passed"] == (name == "prompt_v2")
path = args.output / (name + ".json")
path.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n")
summary["reports"].append({"candidate": name, "cases": len(report["rows"]),
"statistics": report["statistics"], "gate": report["gate"],
"failure_count": len(report["failures"]), "budget": report["budget"]})
(args.output / "summary.json").write_text(json.dumps(summary, ensure_ascii=False, indent=2) + "\n")
print(json.dumps(summary, ensure_ascii=False, indent=2))
if __name__ == "__main__":
main(){
"run_kind": "controlled_deterministic_fixture_no_real_model",
"created_at": "2026-10-21T08:26:50.091147+00:00",
"dataset": {
"version": "synthetic-quality-v1",
"synthetic": true,
"purpose": "Test quality pipeline behavior, not model intelligence or vendor ranking",
"splits": {
"train": {
"samples": 6,
"groups": 3
},
"dev": {
"samples": 8,
"groups": 4
},
"acceptance": {
"samples": 12,
"groups": 6
}
},
"sha256": {
"samples.json": "684c829ea8ed4a07c5f47caa60df2a5548b6c7942ec364a30b18257028907727",
"answers.json": "970d44538c5649961c20dcd0e536100020e247ee91d078c68a22b53245141d59"
}
},
"split": "acceptance",
"baseline_id": "baseline",
"candidate_id": "model_v2",
"changes": "controlled_generator_defect",
"provenance": {
"python": "3.14.3",
"candidate_sha256": "43a1b0a1ec7879e87807cfa9a89c1ad67df8f48675337576e62964d2af484169",
"pipeline_sha256": "1c07424fc6740285f52d3f4d23267285a7f7fbd5117f0c8e1a74f0d28f614278",
"answer_file_passed_to_candidate": false,
"isolation": "cooperative subprocess, not an OS sandbox"
},
"budget": {
"limit_fixture_units": 2000000,
"spent_fixture_units": 10518,
"unknown_reservations": 0,
"unit": "public task JSON bytes plus candidate response JSON bytes; not model tokens or money"
},
"rows": [
{
"id": "acceptance-project-0-file",
"group": "acceptance-project-0",
"kind": "file",
"baseline": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "e736f46655d8edc6ecd742c326b1703be20afd3c3bd6601d1e45eddf6e885aad",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 45.106,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "28cf10383595f9a394bc0ff867b5c528ce56811409517db5933e16a30e843201",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 44.199,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-0-answer",
"group": "acceptance-project-0",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 60 days.",
"days": 60,
"citations": [
"acceptance-project-0-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 43.693,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation",
"status": "scored",
"response": {
"message": "The refund window is 60 days.",
"days": 60,
"citations": [
"acceptance-project-0-old"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 122,
"fixture_units": 612,
"usage_state": "known",
"wall_ms": 47.378,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-1-file",
"group": "acceptance-project-1",
"kind": "file",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "bc560491a8815c289e6529249870a0aa4969f7e4f65dbd1d1e7d89ac5c638394",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 47.038,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "28cf10383595f9a394bc0ff867b5c528ce56811409517db5933e16a30e843201",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 44.534,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-1-answer",
"group": "acceptance-project-1",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 61 days.",
"days": 61,
"citations": [
"acceptance-project-1-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 41.001,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation",
"status": "scored",
"response": {
"message": "The refund window is 61 days.",
"days": 61,
"citations": [
"acceptance-project-1-old"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 122,
"fixture_units": 612,
"usage_state": "known",
"wall_ms": 46.685,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-2-file",
"group": "acceptance-project-2",
"kind": "file",
"baseline": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "8464d1799690fcd38320c01ea20b4cac4f171fe286c68627568ead92d119754b",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 40.447,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "28cf10383595f9a394bc0ff867b5c528ce56811409517db5933e16a30e843201",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 39.517,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-2-answer",
"group": "acceptance-project-2",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 62 days.",
"days": 62,
"citations": [
"acceptance-project-2-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 39.35,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation",
"status": "scored",
"response": {
"message": "The refund window is 62 days.",
"days": 62,
"citations": [
"acceptance-project-2-old"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 122,
"fixture_units": 612,
"usage_state": "known",
"wall_ms": 45.09,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-3-file",
"group": "acceptance-project-3",
"kind": "file",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "e4bedec480f05dd6b6112854c6d03ba8fbf14c98570f22dfa24c1914a42839d7",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 39.604,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "28cf10383595f9a394bc0ff867b5c528ce56811409517db5933e16a30e843201",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 42.868,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-3-answer",
"group": "acceptance-project-3",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 63 days.",
"days": 63,
"citations": [
"acceptance-project-3-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 44.694,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation",
"status": "scored",
"response": {
"message": "The refund window is 63 days.",
"days": 63,
"citations": [
"acceptance-project-3-old"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 122,
"fixture_units": 612,
"usage_state": "known",
"wall_ms": 47.933,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-4-file",
"group": "acceptance-project-4",
"kind": "file",
"baseline": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "18e841eee4e9c6e815ff2700d3dd89703d7c73fbaf0944925f4ac11eea820b4a",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 46.802,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "28cf10383595f9a394bc0ff867b5c528ce56811409517db5933e16a30e843201",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 43.588,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-4-answer",
"group": "acceptance-project-4",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 64 days.",
"days": 64,
"citations": [
"acceptance-project-4-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 43.496,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation",
"status": "scored",
"response": {
"message": "The refund window is 64 days.",
"days": 64,
"citations": [
"acceptance-project-4-old"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 122,
"fixture_units": 612,
"usage_state": "known",
"wall_ms": 44.276,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-5-file",
"group": "acceptance-project-5",
"kind": "file",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "668cf02a2eeb5172bbd88dc33f7da86b64e42a2c62eca6bd76787d6eac105db2",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 45.956,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "28cf10383595f9a394bc0ff867b5c528ce56811409517db5933e16a30e843201",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 44.3,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-5-answer",
"group": "acceptance-project-5",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 65 days.",
"days": 65,
"citations": [
"acceptance-project-5-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 42.463,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation",
"status": "scored",
"response": {
"message": "The refund window is 65 days.",
"days": 65,
"citations": [
"acceptance-project-5-old"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 122,
"fixture_units": 612,
"usage_state": "known",
"wall_ms": 45.143,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
}
],
"statistics": {
"estimand": "equal_weight_mean_of_project_task_success_differences",
"paired_groups": 6,
"cases": 12,
"difference": -0.75,
"percentile_interval_95": [
-0.9166666666666666,
-0.5833333333333334
],
"seed": 17,
"resamples": 2000,
"warning": "Synthetic small fixture: method demonstration, not population evidence or a model ranking"
},
"gate": {
"passed": false,
"reasons": [
"new_task_regression",
"paired_interval_below_noninferiority_margin"
],
"policy": "fixture-gate-v1",
"noninferiority_margin": -0.05
},
"failures": [
{
"id": "acceptance-project-0-file",
"kind": "file",
"baseline_reason": "artifact_mismatch",
"candidate_reason": "artifact_mismatch"
},
{
"id": "acceptance-project-0-answer",
"kind": "answer",
"baseline_reason": "supported_fact",
"candidate_reason": "wrong_fact_or_citation"
},
{
"id": "acceptance-project-1-file",
"kind": "file",
"baseline_reason": "artifact_matches",
"candidate_reason": "artifact_mismatch"
},
{
"id": "acceptance-project-1-answer",
"kind": "answer",
"baseline_reason": "supported_fact",
"candidate_reason": "wrong_fact_or_citation"
},
{
"id": "acceptance-project-2-file",
"kind": "file",
"baseline_reason": "artifact_mismatch",
"candidate_reason": "artifact_mismatch"
},
{
"id": "acceptance-project-2-answer",
"kind": "answer",
"baseline_reason": "supported_fact",
"candidate_reason": "wrong_fact_or_citation"
},
{
"id": "acceptance-project-3-file",
"kind": "file",
"baseline_reason": "artifact_matches",
"candidate_reason": "artifact_mismatch"
},
{
"id": "acceptance-project-3-answer",
"kind": "answer",
"baseline_reason": "supported_fact",
"candidate_reason": "wrong_fact_or_citation"
},
{
"id": "acceptance-project-4-file",
"kind": "file",
"baseline_reason": "artifact_mismatch",
"candidate_reason": "artifact_mismatch"
},
{
"id": "acceptance-project-4-answer",
"kind": "answer",
"baseline_reason": "supported_fact",
"candidate_reason": "wrong_fact_or_citation"
},
{
"id": "acceptance-project-5-file",
"kind": "file",
"baseline_reason": "artifact_matches",
"candidate_reason": "artifact_mismatch"
},
{
"id": "acceptance-project-5-answer",
"kind": "answer",
"baseline_reason": "supported_fact",
"candidate_reason": "wrong_fact_or_citation"
}
]
}{
"run_kind": "controlled_deterministic_fixture_no_real_model",
"created_at": "2026-10-21T08:26:47.943428+00:00",
"dataset": {
"version": "synthetic-quality-v1",
"synthetic": true,
"purpose": "Test quality pipeline behavior, not model intelligence or vendor ranking",
"splits": {
"train": {
"samples": 6,
"groups": 3
},
"dev": {
"samples": 8,
"groups": 4
},
"acceptance": {
"samples": 12,
"groups": 6
}
},
"sha256": {
"samples.json": "684c829ea8ed4a07c5f47caa60df2a5548b6c7942ec364a30b18257028907727",
"answers.json": "970d44538c5649961c20dcd0e536100020e247ee91d078c68a22b53245141d59"
}
},
"split": "acceptance",
"baseline_id": "baseline",
"candidate_id": "prompt_v2",
"changes": "request_normalizer",
"provenance": {
"python": "3.14.3",
"candidate_sha256": "43a1b0a1ec7879e87807cfa9a89c1ad67df8f48675337576e62964d2af484169",
"pipeline_sha256": "1c07424fc6740285f52d3f4d23267285a7f7fbd5117f0c8e1a74f0d28f614278",
"answer_file_passed_to_candidate": false,
"isolation": "cooperative subprocess, not an OS sandbox"
},
"budget": {
"limit_fixture_units": 2000000,
"spent_fixture_units": 10542,
"unknown_reservations": 0,
"unit": "public task JSON bytes plus candidate response JSON bytes; not model tokens or money"
},
"rows": [
{
"id": "acceptance-project-0-file",
"group": "acceptance-project-0",
"kind": "file",
"baseline": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "e736f46655d8edc6ecd742c326b1703be20afd3c3bd6601d1e45eddf6e885aad",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 42.69,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "8e321d493d67b90c018d4aa852fa20a1b8028ad944634ebe1cbcffbb582f1e5d",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 46.852,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-0-answer",
"group": "acceptance-project-0",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 60 days.",
"days": 60,
"citations": [
"acceptance-project-0-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 46.023,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 60 days.",
"days": 60,
"citations": [
"acceptance-project-0-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 47.198,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-1-file",
"group": "acceptance-project-1",
"kind": "file",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "bc560491a8815c289e6529249870a0aa4969f7e4f65dbd1d1e7d89ac5c638394",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 45.559,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "bc560491a8815c289e6529249870a0aa4969f7e4f65dbd1d1e7d89ac5c638394",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 46.753,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-1-answer",
"group": "acceptance-project-1",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 61 days.",
"days": 61,
"citations": [
"acceptance-project-1-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 46.666,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 61 days.",
"days": 61,
"citations": [
"acceptance-project-1-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 46.321,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-2-file",
"group": "acceptance-project-2",
"kind": "file",
"baseline": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "8464d1799690fcd38320c01ea20b4cac4f171fe286c68627568ead92d119754b",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 44.094,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "4147488415104aad6ae07aca42f2ddf87bb1e6ca7d1e7020bb780417f03fee1e",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 43.641,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-2-answer",
"group": "acceptance-project-2",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 62 days.",
"days": 62,
"citations": [
"acceptance-project-2-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 41.049,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 62 days.",
"days": 62,
"citations": [
"acceptance-project-2-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 40.484,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-3-file",
"group": "acceptance-project-3",
"kind": "file",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "e4bedec480f05dd6b6112854c6d03ba8fbf14c98570f22dfa24c1914a42839d7",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 42.316,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "e4bedec480f05dd6b6112854c6d03ba8fbf14c98570f22dfa24c1914a42839d7",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 41.407,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-3-answer",
"group": "acceptance-project-3",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 63 days.",
"days": 63,
"citations": [
"acceptance-project-3-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 38.837,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 63 days.",
"days": 63,
"citations": [
"acceptance-project-3-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 47.979,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-4-file",
"group": "acceptance-project-4",
"kind": "file",
"baseline": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "18e841eee4e9c6e815ff2700d3dd89703d7c73fbaf0944925f4ac11eea820b4a",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 36.249,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "a084cced31821aadbd3a94c18d80f20a70631d259b923e834c2734863e1b5c00",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 47.551,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-4-answer",
"group": "acceptance-project-4",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 64 days.",
"days": 64,
"citations": [
"acceptance-project-4-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 43.957,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 64 days.",
"days": 64,
"citations": [
"acceptance-project-4-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 47.863,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-5-file",
"group": "acceptance-project-5",
"kind": "file",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "668cf02a2eeb5172bbd88dc33f7da86b64e42a2c62eca6bd76787d6eac105db2",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 41.148,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "668cf02a2eeb5172bbd88dc33f7da86b64e42a2c62eca6bd76787d6eac105db2",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 43.684,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-5-answer",
"group": "acceptance-project-5",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 65 days.",
"days": 65,
"citations": [
"acceptance-project-5-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 38.168,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 65 days.",
"days": 65,
"citations": [
"acceptance-project-5-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 48.072,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
}
],
"statistics": {
"estimand": "equal_weight_mean_of_project_task_success_differences",
"paired_groups": 6,
"cases": 12,
"difference": 0.25,
"percentile_interval_95": [
0.08333333333333333,
0.4166666666666667
],
"seed": 17,
"resamples": 2000,
"warning": "Synthetic small fixture: method demonstration, not population evidence or a model ranking"
},
"gate": {
"passed": true,
"reasons": [],
"policy": "fixture-gate-v1",
"noninferiority_margin": -0.05
},
"failures": []
}{
"run_kind": "controlled_deterministic_fixture_no_real_model",
"created_at": "2026-10-21T08:26:49.001708+00:00",
"dataset": {
"version": "synthetic-quality-v1",
"synthetic": true,
"purpose": "Test quality pipeline behavior, not model intelligence or vendor ranking",
"splits": {
"train": {
"samples": 6,
"groups": 3
},
"dev": {
"samples": 8,
"groups": 4
},
"acceptance": {
"samples": 12,
"groups": 6
}
},
"sha256": {
"samples.json": "684c829ea8ed4a07c5f47caa60df2a5548b6c7942ec364a30b18257028907727",
"answers.json": "970d44538c5649961c20dcd0e536100020e247ee91d078c68a22b53245141d59"
}
},
"split": "acceptance",
"baseline_id": "baseline",
"candidate_id": "retrieval_v2",
"changes": "status_filter_removed",
"provenance": {
"python": "3.14.3",
"candidate_sha256": "43a1b0a1ec7879e87807cfa9a89c1ad67df8f48675337576e62964d2af484169",
"pipeline_sha256": "1c07424fc6740285f52d3f4d23267285a7f7fbd5117f0c8e1a74f0d28f614278",
"answer_file_passed_to_candidate": false,
"isolation": "cooperative subprocess, not an OS sandbox"
},
"budget": {
"limit_fixture_units": 2000000,
"spent_fixture_units": 10506,
"unknown_reservations": 0,
"unit": "public task JSON bytes plus candidate response JSON bytes; not model tokens or money"
},
"rows": [
{
"id": "acceptance-project-0-file",
"group": "acceptance-project-0",
"kind": "file",
"baseline": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "e736f46655d8edc6ecd742c326b1703be20afd3c3bd6601d1e45eddf6e885aad",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 47.859,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "e736f46655d8edc6ecd742c326b1703be20afd3c3bd6601d1e45eddf6e885aad",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 36.653,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-0-answer",
"group": "acceptance-project-0",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 60 days.",
"days": 60,
"citations": [
"acceptance-project-0-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 23.622,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": false,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation",
"status": "scored",
"response": {
"message": "The refund window is 7 days.",
"days": 7,
"citations": [
"acceptance-project-0-old"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 120,
"fixture_units": 610,
"usage_state": "known",
"wall_ms": 48.031,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-1-file",
"group": "acceptance-project-1",
"kind": "file",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "bc560491a8815c289e6529249870a0aa4969f7e4f65dbd1d1e7d89ac5c638394",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 43.35,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "bc560491a8815c289e6529249870a0aa4969f7e4f65dbd1d1e7d89ac5c638394",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 47.127,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-1-answer",
"group": "acceptance-project-1",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 61 days.",
"days": 61,
"citations": [
"acceptance-project-1-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 48.015,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": false,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation",
"status": "scored",
"response": {
"message": "The refund window is 7 days.",
"days": 7,
"citations": [
"acceptance-project-1-old"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 120,
"fixture_units": 610,
"usage_state": "known",
"wall_ms": 23.428,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-2-file",
"group": "acceptance-project-2",
"kind": "file",
"baseline": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "8464d1799690fcd38320c01ea20b4cac4f171fe286c68627568ead92d119754b",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 40.1,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "8464d1799690fcd38320c01ea20b4cac4f171fe286c68627568ead92d119754b",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 46.84,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-2-answer",
"group": "acceptance-project-2",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 62 days.",
"days": 62,
"citations": [
"acceptance-project-2-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 45.031,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": false,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation",
"status": "scored",
"response": {
"message": "The refund window is 7 days.",
"days": 7,
"citations": [
"acceptance-project-2-old"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 120,
"fixture_units": 610,
"usage_state": "known",
"wall_ms": 38.416,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-3-file",
"group": "acceptance-project-3",
"kind": "file",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "e4bedec480f05dd6b6112854c6d03ba8fbf14c98570f22dfa24c1914a42839d7",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 39.08,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "e4bedec480f05dd6b6112854c6d03ba8fbf14c98570f22dfa24c1914a42839d7",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 47.52,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-3-answer",
"group": "acceptance-project-3",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 63 days.",
"days": 63,
"citations": [
"acceptance-project-3-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 48.145,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": false,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation",
"status": "scored",
"response": {
"message": "The refund window is 7 days.",
"days": 7,
"citations": [
"acceptance-project-3-old"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 120,
"fixture_units": 610,
"usage_state": "known",
"wall_ms": 45.675,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-4-file",
"group": "acceptance-project-4",
"kind": "file",
"baseline": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "18e841eee4e9c6e815ff2700d3dd89703d7c73fbaf0944925f4ac11eea820b4a",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 41.294,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "18e841eee4e9c6e815ff2700d3dd89703d7c73fbaf0944925f4ac11eea820b4a",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 227,
"output_bytes": 42,
"fixture_units": 269,
"usage_state": "known",
"wall_ms": 44.582,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-4-answer",
"group": "acceptance-project-4",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 64 days.",
"days": 64,
"citations": [
"acceptance-project-4-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 42.507,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": false,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation",
"status": "scored",
"response": {
"message": "The refund window is 7 days.",
"days": 7,
"citations": [
"acceptance-project-4-old"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 120,
"fixture_units": 610,
"usage_state": "known",
"wall_ms": 41.333,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
},
{
"id": "acceptance-project-5-file",
"group": "acceptance-project-5",
"kind": "file",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "668cf02a2eeb5172bbd88dc33f7da86b64e42a2c62eca6bd76787d6eac105db2",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 47.886,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
},
"candidate": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches",
"status": "scored",
"response": {
"message": "done",
"needs_review": false
},
"artifact_sha256": {
"service.cfg": "668cf02a2eeb5172bbd88dc33f7da86b64e42a2c62eca6bd76787d6eac105db2",
"policy.lock": "08e3dfc089e66c44ae8abd3476b0f810f11f4cfc2ab925f1607c7df21345d639"
},
"input_bytes": 214,
"output_bytes": 42,
"fixture_units": 256,
"usage_state": "known",
"wall_ms": 45.435,
"candidate_visible_files": [
"candidate.py",
"policy.lock",
"service.cfg",
"task.json"
]
}
},
{
"id": "acceptance-project-5-answer",
"group": "acceptance-project-5",
"kind": "answer",
"baseline": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact",
"status": "scored",
"response": {
"message": "The refund window is 65 days.",
"days": 65,
"citations": [
"acceptance-project-5-current"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 126,
"fixture_units": 616,
"usage_state": "known",
"wall_ms": 46.98,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
},
"candidate": {
"task_pass": false,
"quality_pass": false,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation",
"status": "scored",
"response": {
"message": "The refund window is 7 days.",
"days": 7,
"citations": [
"acceptance-project-5-old"
],
"needs_review": false
},
"artifact_sha256": {
"service.cfg": null,
"policy.lock": null
},
"input_bytes": 490,
"output_bytes": 120,
"fixture_units": 610,
"usage_state": "known",
"wall_ms": 43.097,
"candidate_visible_files": [
"candidate.py",
"task.json"
]
}
}
],
"statistics": {
"estimand": "equal_weight_mean_of_project_task_success_differences",
"paired_groups": 6,
"cases": 12,
"difference": -0.5,
"percentile_interval_95": [
-0.5,
-0.5
],
"seed": 17,
"resamples": 2000,
"warning": "Synthetic small fixture: method demonstration, not population evidence or a model ranking"
},
"gate": {
"passed": false,
"reasons": [
"new_task_regression",
"new_answer_quality_regression",
"paired_interval_below_noninferiority_margin"
],
"policy": "fixture-gate-v1",
"noninferiority_margin": -0.05
},
"failures": [
{
"id": "acceptance-project-0-file",
"kind": "file",
"baseline_reason": "artifact_mismatch",
"candidate_reason": "artifact_mismatch"
},
{
"id": "acceptance-project-0-answer",
"kind": "answer",
"baseline_reason": "supported_fact",
"candidate_reason": "wrong_fact_or_citation"
},
{
"id": "acceptance-project-1-answer",
"kind": "answer",
"baseline_reason": "supported_fact",
"candidate_reason": "wrong_fact_or_citation"
},
{
"id": "acceptance-project-2-file",
"kind": "file",
"baseline_reason": "artifact_mismatch",
"candidate_reason": "artifact_mismatch"
},
{
"id": "acceptance-project-2-answer",
"kind": "answer",
"baseline_reason": "supported_fact",
"candidate_reason": "wrong_fact_or_citation"
},
{
"id": "acceptance-project-3-answer",
"kind": "answer",
"baseline_reason": "supported_fact",
"candidate_reason": "wrong_fact_or_citation"
},
{
"id": "acceptance-project-4-file",
"kind": "file",
"baseline_reason": "artifact_mismatch",
"candidate_reason": "artifact_mismatch"
},
{
"id": "acceptance-project-4-answer",
"kind": "answer",
"baseline_reason": "supported_fact",
"candidate_reason": "wrong_fact_or_citation"
},
{
"id": "acceptance-project-5-answer",
"kind": "answer",
"baseline_reason": "supported_fact",
"candidate_reason": "wrong_fact_or_citation"
}
]
}{
"kind": "controlled_fixture_no_real_models",
"reports": [
{
"candidate": "prompt_v2",
"cases": 12,
"statistics": {
"estimand": "equal_weight_mean_of_project_task_success_differences",
"paired_groups": 6,
"cases": 12,
"difference": 0.25,
"percentile_interval_95": [
0.08333333333333333,
0.4166666666666667
],
"seed": 17,
"resamples": 2000,
"warning": "Synthetic small fixture: method demonstration, not population evidence or a model ranking"
},
"gate": {
"passed": true,
"reasons": [],
"policy": "fixture-gate-v1",
"noninferiority_margin": -0.05
},
"failure_count": 0,
"budget": {
"limit_fixture_units": 2000000,
"spent_fixture_units": 10542,
"unknown_reservations": 0,
"unit": "public task JSON bytes plus candidate response JSON bytes; not model tokens or money"
}
},
{
"candidate": "retrieval_v2",
"cases": 12,
"statistics": {
"estimand": "equal_weight_mean_of_project_task_success_differences",
"paired_groups": 6,
"cases": 12,
"difference": -0.5,
"percentile_interval_95": [
-0.5,
-0.5
],
"seed": 17,
"resamples": 2000,
"warning": "Synthetic small fixture: method demonstration, not population evidence or a model ranking"
},
"gate": {
"passed": false,
"reasons": [
"new_task_regression",
"new_answer_quality_regression",
"paired_interval_below_noninferiority_margin"
],
"policy": "fixture-gate-v1",
"noninferiority_margin": -0.05
},
"failure_count": 9,
"budget": {
"limit_fixture_units": 2000000,
"spent_fixture_units": 10506,
"unknown_reservations": 0,
"unit": "public task JSON bytes plus candidate response JSON bytes; not model tokens or money"
}
},
{
"candidate": "model_v2",
"cases": 12,
"statistics": {
"estimand": "equal_weight_mean_of_project_task_success_differences",
"paired_groups": 6,
"cases": 12,
"difference": -0.75,
"percentile_interval_95": [
-0.9166666666666666,
-0.5833333333333334
],
"seed": 17,
"resamples": 2000,
"warning": "Synthetic small fixture: method demonstration, not population evidence or a model ranking"
},
"gate": {
"passed": false,
"reasons": [
"new_task_regression",
"paired_interval_below_noninferiority_margin"
],
"policy": "fixture-gate-v1",
"noninferiority_margin": -0.05
},
"failure_count": 12,
"budget": {
"limit_fixture_units": 2000000,
"spent_fixture_units": 10518,
"unknown_reservations": 0,
"unit": "public task JSON bytes plus candidate response JSON bytes; not model tokens or money"
}
}
],
"calibration": {
"kind": "authored_counterexamples_not_human_or_model_judge_measurements",
"confusion": {
"naive": {
"tp": 2,
"fp": 2,
"tn": 2,
"fn": 0
},
"contract": {
"tp": 2,
"fp": 0,
"tn": 4,
"fn": 0
}
},
"naive_false_positive_rate": 0.5,
"cases": [
{
"id": "file-written",
"naive_pass": true,
"contract_pass": true,
"reference_pass": true,
"score_detail": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_matches"
}
},
{
"id": "claims-done-file-unchanged",
"naive_pass": true,
"contract_pass": false,
"reference_pass": false,
"score_detail": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch"
}
},
{
"id": "correct-value-wrong-citation",
"naive_pass": true,
"contract_pass": false,
"reference_pass": false,
"score_detail": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "wrong_fact_or_citation"
}
},
{
"id": "empty-answer",
"naive_pass": false,
"contract_pass": false,
"reference_pass": false,
"score_detail": {
"task_pass": true,
"quality_pass": false,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact"
}
},
{
"id": "supported-answer",
"naive_pass": true,
"contract_pass": true,
"reference_pass": true,
"score_detail": {
"task_pass": true,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "supported_fact"
}
},
{
"id": "explicit-failure",
"naive_pass": false,
"contract_pass": false,
"reference_pass": false,
"score_detail": {
"task_pass": false,
"quality_pass": true,
"safety_pass": true,
"review_pending": false,
"reason": "artifact_mismatch"
}
}
]
}
}"""Create the checked-in synthetic dataset, private labels and version manifest."""
from pathlib import Path
import hashlib
import json
ROOT = Path(__file__).parent / "data"
def canonical(value):
return json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode()
def main():
ROOT.mkdir(exist_ok=True)
public, answers = [], {}
for split, count, offset in [("train", 3, 10), ("dev", 4, 30), ("acceptance", 6, 60)]:
for i in range(count):
group = f"{split}-project-{i}"
target = offset + i
for kind in ["file", "answer"]:
sid = f"{group}-{kind}"
record = {"id": sid, "group": group, "split": split, "kind": kind,
"origin": "authored-synthetic-v1", "project": group}
if kind == "file":
# Some instructions contain an unrelated issue number. A
# naive candidate parses it as the requested timeout.
instruction = f"Set timeout_ms to {target}"
if i % 2 == 0:
instruction = f"Ticket 900{i}: " + instruction
initial = "timeout_ms=5\nretries=2\n"
record.update(instruction=instruction, files={"service.cfg": initial,
"policy.lock": "DO NOT MODIFY\n"})
answers[sid] = {"expected_file": f"timeout_ms={target}\nretries=2\n",
"protected": "DO NOT MODIFY\n"}
else:
current, old = f"{group}-current", f"{group}-old"
record.update(instruction=f"How many days is the current {group} refund window?",
documents=[{"id": old, "status": "superseded", "days": 7,
"text": f"An old policy for {group}: refund window is 7 days."},
{"id": current, "status": "current", "days": target,
"text": f"The current policy for {group}: refund window is {target} days."}])
answers[sid] = {"days": target, "supporting_ids": [current]}
public.append(record)
(ROOT / "samples.json").write_bytes(canonical(public) + b"\n")
(ROOT / "answers.json").write_bytes(canonical(answers) + b"\n")
manifest = {"version": "synthetic-quality-v1", "synthetic": True,
"purpose": "Test quality pipeline behavior, not model intelligence or vendor ranking",
"splits": {s: {"samples": sum(r["split"] == s for r in public),
"groups": len({r["group"] for r in public if r["split"] == s})}
for s in ["train", "dev", "acceptance"]},
"sha256": {name: hashlib.sha256((ROOT / name).read_bytes()).hexdigest()
for name in ["samples.json", "answers.json"]}}
(ROOT / "manifest.json").write_text(json.dumps(manifest, ensure_ascii=False, indent=2) + "\n")
print(json.dumps(manifest, ensure_ascii=False))
if __name__ == "__main__":
main()"""Versioned paired evaluation, subprocess candidates, reports and release gate."""
from collections import defaultdict
from datetime import datetime, timezone
import hashlib
import json
from pathlib import Path
import random
import shutil
import statistics
import subprocess
import sys
import tempfile
import time
ROOT = Path(__file__).resolve().parent
MODES = {"baseline", "prompt_v2", "retrieval_v2", "model_v2", "unsafe", "terse", "review", "timeout"}
def sha(data):
return hashlib.sha256(data).hexdigest()
def load_data():
manifest = json.loads((ROOT / "data/manifest.json").read_text())
for name, expected in manifest["sha256"].items():
if sha((ROOT / "data" / name).read_bytes()) != expected:
raise ValueError("dataset_hash_mismatch:" + name)
samples = json.loads((ROOT / "data/samples.json").read_text())
answers = json.loads((ROOT / "data/answers.json").read_text())
seen, groups = set(), defaultdict(set)
for sample in samples:
if sample["id"] in seen or sample["id"] not in answers:
raise ValueError("duplicate_or_unlabelled_sample")
seen.add(sample["id"])
groups[sample["split"]].add(sample["group"])
for a in groups:
for b in groups:
if a != b and groups[a] & groups[b]:
raise ValueError("group_leaks_between_splits")
if seen != set(answers):
raise ValueError("orphan_label")
return manifest, samples, answers
def score(sample, truth, response, artifacts):
"""Separate task evidence from presentation. Not a semantic LLM judge."""
quality = isinstance(response.get("message"), str) and bool(response["message"].strip())
if sample["kind"] == "file":
task = artifacts.get("service.cfg") == truth["expected_file"]
safe = artifacts.get("policy.lock") == truth["protected"]
evidence = "artifact_matches" if task else "artifact_mismatch"
else:
# An exact structured-fact oracle, suitable only for this constrained
# dataset; it does not establish arbitrary text entailment or fluency.
fact = type(response.get("days")) is int and response["days"] == truth["days"]
citations = response.get("citations")
cited = (isinstance(citations, list) and bool(citations) and
all(isinstance(c, str) and c in truth["supporting_ids"] for c in citations))
task, safe = fact and cited, True
quality = quality and str(truth["days"]) in response.get("message", "")
evidence = "supported_fact" if task else "wrong_fact_or_citation"
review = response.get("needs_review") is True
return {"task_pass": task, "quality_pass": quality, "safety_pass": safe,
"review_pending": review, "reason": evidence}
def run_candidate(sample, truth, mode, timeout=1.0, maximum_response=4096):
public = {key: value for key, value in sample.items() if key not in {"split", "origin", "group"}}
public_bytes = json.dumps(public, sort_keys=True).encode()
with tempfile.TemporaryDirectory(prefix="quality-candidate-") as directory:
work = Path(directory)
shutil.copyfile(ROOT / "candidate.py", work / "candidate.py")
(work / "task.json").write_bytes(public_bytes)
for name, text in sample.get("files", {}).items():
if name not in {"service.cfg", "policy.lock"}:
raise ValueError("unapproved_fixture_path")
(work / name).write_text(text)
initial_names = sorted(p.name for p in work.iterdir())
start = time.monotonic()
try:
process = subprocess.run([sys.executable, "-I", str(work / "candidate.py"), mode],
cwd=work, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL, timeout=timeout, check=False,
env={"LANG": "C.UTF-8"})
elapsed = round((time.monotonic() - start) * 1000, 3)
if process.returncode != 0:
raise ValueError("candidate_process_failed")
path = work / "response.json"
if not path.is_file() or path.stat().st_size > maximum_response:
raise ValueError("missing_or_oversized_response")
raw = path.read_bytes()
response = json.loads(raw)
if not isinstance(response, dict):
raise ValueError("invalid_response")
artifacts = {name: (work / name).read_text() if (work / name).is_file() else None
for name in ["service.cfg", "policy.lock"]}
result = score(sample, truth, response, artifacts)
return {**result, "status": "scored", "response": response,
"artifact_sha256": {name: sha(text.encode()) if text is not None else None
for name, text in artifacts.items()},
"input_bytes": len(public_bytes), "output_bytes": len(raw),
"fixture_units": len(public_bytes) + len(raw), "usage_state": "known",
"wall_ms": elapsed, "candidate_visible_files": initial_names}
except subprocess.TimeoutExpired:
reason, usage_state = "timeout", "unknown"
except (ValueError, OSError, UnicodeError):
reason, usage_state = "invalid_candidate_output", "unknown"
return {"task_pass": False, "quality_pass": False, "safety_pass": False,
"review_pending": False, "reason": reason, "status": "unscored",
"response": None, "artifact_sha256": {}, "input_bytes": len(public_bytes),
"output_bytes": None, "fixture_units": None, "usage_state": usage_state,
"wall_ms": round((time.monotonic() - start) * 1000, 3),
"candidate_visible_files": initial_names}
def quantile(values, p):
ordered = sorted(values)
position = (len(ordered) - 1) * p
low = int(position)
high = min(low + 1, len(ordered) - 1)
return ordered[low] + (ordered[high] - ordered[low]) * (position - low)
def paired_statistics(rows, seed=17, resamples=2000):
"""Equal-weight group means, paired cluster percentile bootstrap.
A group is resampled as a unit with both candidates and all its cases.
The tiny authored fixture is not a representative business population.
"""
by_group = defaultdict(list)
for row in rows:
by_group[row["group"]].append(int(row["candidate"]["task_pass"]) - int(row["baseline"]["task_pass"]))
effects = [statistics.mean(values) for _, values in sorted(by_group.items())]
if not effects or resamples < 1:
raise ValueError("empty_pair_set_or_resamples")
rng = random.Random(seed)
distribution = [statistics.mean(rng.choices(effects, k=len(effects))) for _ in range(resamples)]
return {"estimand": "equal_weight_mean_of_project_task_success_differences",
"paired_groups": len(effects), "cases": len(rows), "difference": statistics.mean(effects),
"percentile_interval_95": [quantile(distribution, 0.025), quantile(distribution, 0.975)],
"seed": seed, "resamples": resamples,
"warning": "Synthetic small fixture: method demonstration, not population evidence or a model ranking"}
def release_gate(report):
reasons = []
rows = report["rows"]
if report["split"] != "acceptance":
reasons.append("not_acceptance_split")
if any(side["status"] != "scored" for r in rows for side in [r["baseline"], r["candidate"]]):
reasons.append("incomplete_or_unscored_pairs")
if any(not r["candidate"]["safety_pass"] for r in rows):
reasons.append("safety_or_unknown_failure")
if any(r["candidate"]["review_pending"] for r in rows):
reasons.append("unresolved_review")
if any(r["baseline"]["task_pass"] and not r["candidate"]["task_pass"] for r in rows):
reasons.append("new_task_regression")
if any(r["baseline"]["quality_pass"] and not r["candidate"]["quality_pass"] for r in rows):
reasons.append("new_answer_quality_regression")
if report["statistics"]["percentile_interval_95"][0] < -0.05:
reasons.append("paired_interval_below_noninferiority_margin")
# This gate accepts bounded equivalence; it does not require every release
# to be better and is not a claim of a business-powered noninferiority test.
return {"passed": not reasons, "reasons": reasons,
"policy": "fixture-gate-v1", "noninferiority_margin": -0.05}
def compare(candidate="prompt_v2", split="acceptance", budget=2_000_000, seed=17, timeout=1.0):
if candidate not in MODES or split not in {"train", "dev", "acceptance"}:
raise ValueError("unknown_candidate_or_split")
manifest, samples, answers = load_data()
cases = [s for s in samples if s["split"] == split]
rows, spent, held = [], 0, 0
for sample in cases:
pair = {"id": sample["id"], "group": sample["group"], "kind": sample["kind"]}
for label, mode in [("baseline", "baseline"), ("candidate", candidate)]:
public = {k: v for k, v in sample.items() if k not in {"split", "origin", "group"}}
reserve = len(json.dumps(public, sort_keys=True).encode()) + 4096
if spent + held + reserve > budget:
result = {"status": "budget_denied", "task_pass": False, "quality_pass": False,
"safety_pass": False, "review_pending": False, "reason": "budget_exhausted",
"fixture_units": 0, "usage_state": "not_started"}
else:
held += reserve
result = run_candidate(sample, answers[sample["id"]], mode, timeout=timeout)
if result["usage_state"] == "known":
held -= reserve
spent += result["fixture_units"]
pair[label] = result
rows.append(pair)
report = {"run_kind": "controlled_deterministic_fixture_no_real_model",
"created_at": datetime.now(timezone.utc).isoformat(), "dataset": manifest,
"split": split, "baseline_id": "baseline", "candidate_id": candidate,
"changes": {"prompt_v2": "request_normalizer", "retrieval_v2": "status_filter_removed",
"model_v2": "controlled_generator_defect"}.get(candidate, candidate),
"provenance": {"python": sys.version.split()[0], "candidate_sha256": sha((ROOT / "candidate.py").read_bytes()),
"pipeline_sha256": sha(Path(__file__).read_bytes()),
"answer_file_passed_to_candidate": False,
"isolation": "cooperative subprocess, not an OS sandbox"},
"budget": {"limit_fixture_units": budget, "spent_fixture_units": spent,
"unknown_reservations": held,
"unit": "public task JSON bytes plus candidate response JSON bytes; not model tokens or money"},
"rows": rows, "statistics": paired_statistics(rows, seed)}
report["gate"] = release_gate(report)
report["failures"] = [{"id": r["id"], "kind": r["kind"], "baseline_reason": r["baseline"]["reason"],
"candidate_reason": r["candidate"]["reason"]}
for r in rows if not r["candidate"]["task_pass"] or not r["candidate"]["quality_pass"]]
return report
def calibration():
# Execute both graders on authored counterexamples. Labels describe these
# mechanically inspectable examples, not a human rater agreement study.
_, samples, answers = load_data()
file = next(s for s in samples if s["split"] == "dev" and s["kind"] == "file")
answer = next(s for s in samples if s["split"] == "dev" and s["kind"] == "answer")
ft, at = answers[file["id"]], answers[answer["id"]]
good = {"service.cfg": ft["expected_file"], "policy.lock": ft["protected"]}
unchanged = file["files"]
supported = {"message": f"The window is {at['days']} days.", "days": at["days"], "citations": at["supporting_ids"]}
examples = [("file-written", file, {"message": "done"}, good, True),
("claims-done-file-unchanged", file, {"message": "done"}, unchanged, False),
("correct-value-wrong-citation", answer, {**supported, "citations": ["wrong-doc"]}, {}, False),
("empty-answer", answer, {**supported, "message": ""}, {}, False),
("supported-answer", answer, supported, {}, True),
("explicit-failure", file, {"message": "failed"}, unchanged, False)]
matrices = {name: {"tp": 0, "fp": 0, "tn": 0, "fn": 0} for name in ["naive", "contract"]}
cases = []
for sid, sample, response, artifacts, reference in examples:
truth = answers[sample["id"]]
naive_pass = ("done" in response["message"] if sample["kind"] == "file"
else str(truth["days"]) in response["message"])
actual = score(sample, truth, response, artifacts)
contract_pass = all(actual[k] for k in ["task_pass", "quality_pass", "safety_pass"])
for name, predicted in [("naive", naive_pass), ("contract", contract_pass)]:
matrices[name][("t" if predicted == reference else "f") + ("p" if predicted else "n")] += 1
cases.append({"id": sid, "naive_pass": naive_pass, "contract_pass": contract_pass,
"reference_pass": reference, "score_detail": actual})
return {"kind": "authored_counterexamples_not_human_or_model_judge_measurements",
"confusion": matrices,
"naive_false_positive_rate": matrices["naive"]["fp"] / (matrices["naive"]["fp"] + matrices["naive"]["tn"]),
"cases": cases}
def main():
import argparse
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--candidate", choices=sorted(MODES), default="prompt_v2")
parser.add_argument("--split", choices=["train", "dev", "acceptance"], default="acceptance")
parser.add_argument("--budget", type=int, default=2_000_000)
parser.add_argument("--output", type=Path, default=Path("lab/evidence/report.json"))
parser.add_argument("--require-pass", action="store_true")
args = parser.parse_args()
report = compare(args.candidate, args.split, args.budget)
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n")
print(json.dumps({"candidate": args.candidate, "cases": len(report["rows"]),
"statistics": report["statistics"], "gate": report["gate"],
"failures": report["failures"], "output": str(args.output)}, ensure_ascii=False))
if args.require_pass and not report["gate"]["passed"]:
raise SystemExit(2)
if __name__ == "__main__":
main()"""Apply explicit presentation-review decisions to a fingerprinted saved report.
Self-asserted reviewer IDs are for teaching audit shape, not authenticated human
identity. No review decision can rewrite artifact or safety acceptance results.
"""
import argparse
import copy
import json
from pathlib import Path
from .pipeline import release_gate, sha
def apply(report, report_bytes, decisions):
if json.loads(report_bytes) != report:
raise ValueError("report_object_does_not_match_fingerprint_bytes")
if decisions.get("report_sha256") != sha(report_bytes):
raise ValueError("review_targets_different_report")
result = copy.deepcopy(report)
rows = {r["id"]: r for r in result["rows"]}
seen = set()
for decision in decisions.get("decisions", []):
sid = decision["sample_id"]
if sid not in rows or sid in seen:
raise ValueError("unknown_or_duplicate_review_id")
seen.add(sid)
if decision["decision"] not in {"accept_presentation", "reject_presentation"}:
raise ValueError("review_cannot_override_task_or_safety")
if not decision.get("reviewer") or not decision.get("rationale"):
raise ValueError("review_needs_identity_and_rationale")
candidate = rows[sid]["candidate"]
if not candidate["review_pending"]:
raise ValueError("sample_not_pending_review")
candidate["review_pending"] = False
if decision["decision"] == "reject_presentation":
candidate["quality_pass"] = False
result["review_audit"] = {"source_report_sha256": sha(report_bytes), "decisions": decisions["decisions"],
"identity_assurance": "self-asserted teaching identity, not authenticated"}
result["gate"] = release_gate(result)
return result
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--report", required=True, type=Path)
parser.add_argument("--decisions", required=True, type=Path)
parser.add_argument("--output", required=True, type=Path)
args = parser.parse_args()
data = args.report.read_bytes()
reviewed = apply(json.loads(data), data, json.loads(args.decisions.read_text()))
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(json.dumps(reviewed, ensure_ascii=False, indent=2) + "\n")
print(json.dumps(reviewed["gate"], ensure_ascii=False))
if __name__ == "__main__":
main()import copy
import json
from pathlib import Path
import subprocess
import sys
import tempfile
import unittest
from lab.pipeline import ROOT, calibration, compare, load_data, paired_statistics, run_candidate, score, sha
from lab.review import apply
class PipelineTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.manifest, cls.samples, cls.answers = load_data()
cls.good = compare("prompt_v2")
cls.bad = compare("model_v2")
def test_dataset_fingerprints_and_group_isolation(self):
self.assertEqual(self.manifest["splits"]["acceptance"], {"samples": 12, "groups": 6})
groups = {s: {r["group"] for r in self.samples if r["split"] == s} for s in ["train", "dev", "acceptance"]}
self.assertFalse(groups["train"] & groups["dev"])
self.assertFalse(groups["dev"] & groups["acceptance"])
def test_candidate_workdir_has_no_reference_answers_or_grader(self):
for row in self.good["rows"]:
for side in [row["baseline"], row["candidate"]]:
visible = side["candidate_visible_files"]
self.assertNotIn("answers.json", visible)
self.assertNotIn("pipeline.py", visible)
self.assertIn("candidate.py", visible)
self.assertIn("task.json", visible)
self.assertEqual(self.good["provenance"]["isolation"], "cooperative subprocess, not an OS sandbox")
def test_claimed_completion_does_not_override_file_evidence(self):
failed = [r for r in self.bad["rows"] if r["kind"] == "file"]
self.assertTrue(all(r["candidate"]["response"]["message"] == "done" for r in failed))
self.assertTrue(all(not r["candidate"]["task_pass"] for r in failed))
self.assertTrue(all(r["candidate"]["quality_pass"] for r in failed))
def test_citation_identity_is_separate_from_correct_value(self):
rows = [r for r in self.bad["rows"] if r["kind"] == "answer"]
self.assertTrue(all(r["candidate"]["quality_pass"] for r in rows))
self.assertTrue(all(not r["candidate"]["task_pass"] for r in rows))
def test_controlled_request_fix_passes_fixture_gate(self):
self.assertTrue(self.good["gate"]["passed"])
self.assertEqual(self.good["statistics"]["difference"], 0.25)
self.assertEqual(self.good["statistics"]["paired_groups"], 6)
def test_generator_defect_fails_gate_and_keeps_case_failures(self):
self.assertFalse(self.bad["gate"]["passed"])
self.assertIn("new_task_regression", self.bad["gate"]["reasons"])
self.assertEqual(len(self.bad["failures"]), 12)
def test_retrieval_change_attribution_is_recorded(self):
result = compare("retrieval_v2")
self.assertEqual(result["changes"], "status_filter_removed")
self.assertEqual(result["statistics"]["difference"], -0.5)
self.assertEqual(sum(r["kind"] == "answer" for r in result["failures"]), 6)
def test_budget_denials_remain_in_denominator_and_gate(self):
report = compare("prompt_v2", budget=1)
self.assertEqual(len(report["rows"]), 12)
self.assertEqual(report["budget"]["spent_fixture_units"], 0)
self.assertFalse(report["gate"]["passed"])
self.assertIn("incomplete_or_unscored_pairs", report["gate"]["reasons"])
def test_timeout_keeps_unknown_reservation(self):
sample = self.samples[0]
result = run_candidate(sample, self.answers[sample["id"]], "timeout", timeout=0.05)
self.assertEqual((result["reason"], result["usage_state"]), ("timeout", "unknown"))
report = compare("timeout", budget=15000, timeout=0.04)
self.assertGreater(report["budget"]["unknown_reservations"], 0)
self.assertFalse(report["gate"]["passed"])
def test_task_success_and_presentation_quality_can_diverge(self):
sample = next(s for s in self.samples if s["kind"] == "answer")
result = run_candidate(sample, self.answers[sample["id"]], "terse")
self.assertTrue(result["task_pass"])
self.assertFalse(result["quality_pass"])
def test_protected_artifact_change_is_independent_safety_failure(self):
sample = next(s for s in self.samples if s["kind"] == "file")
result = run_candidate(sample, self.answers[sample["id"]], "unsafe")
self.assertFalse(result["safety_pass"])
def test_calibration_executes_both_graders_and_finds_false_positives(self):
result = calibration()
self.assertEqual(result["confusion"]["naive"], {"tp": 2, "fp": 2, "tn": 2, "fn": 0})
self.assertEqual(result["confusion"]["contract"], {"tp": 2, "fp": 0, "tn": 4, "fn": 0})
self.assertEqual(result["naive_false_positive_rate"], 0.5)
def test_bootstrap_preserves_pair_and_project_unit(self):
rows = copy.deepcopy(self.good["rows"])
result = paired_statistics(rows, seed=22)
self.assertEqual(result, paired_statistics(list(reversed(rows)), seed=22))
self.assertEqual(result["paired_groups"], 6)
self.assertLessEqual(result["percentile_interval_95"][0], result["difference"])
self.assertGreaterEqual(result["percentile_interval_95"][1], result["difference"])
def test_development_split_cannot_be_published_by_gate(self):
result = compare("prompt_v2", split="dev")
self.assertIn("not_acceptance_split", result["gate"]["reasons"])
def test_pending_review_blocks_release_and_wrong_report_is_rejected(self):
report = compare("review")
self.assertIn("unresolved_review", report["gate"]["reasons"])
raw = json.dumps(report).encode()
decisions = {"report_sha256": "wrong", "decisions": []}
with self.assertRaises(ValueError):
apply(report, raw, decisions)
with self.assertRaises(ValueError):
apply({**report, "candidate_id": "different"}, raw,
{"report_sha256": sha(raw), "decisions": []})
def test_review_cannot_erase_artifact_failure(self):
report = compare("review")
raw = json.dumps(report).encode()
decisions = {"report_sha256": sha(raw), "decisions": [{"sample_id": r["id"],
"decision": "accept_presentation", "reviewer": "teaching-reviewer",
"rationale": "Authored workflow demonstration, not authenticated human judgment"}
for r in report["rows"]]}
reviewed = apply(report, raw, decisions)
self.assertNotIn("unresolved_review", reviewed["gate"]["reasons"])
self.assertEqual([r["candidate"]["task_pass"] for r in report["rows"]],
[r["candidate"]["task_pass"] for r in reviewed["rows"]])
def test_cli_returns_nonzero_for_blocked_release(self):
with tempfile.TemporaryDirectory() as directory:
process = subprocess.run([sys.executable, "-m", "lab.pipeline", "--candidate", "retrieval_v2",
"--require-pass", "--output", str(Path(directory)/"report.json")],
cwd=ROOT.parent, capture_output=True, text=True)
self.assertEqual(process.returncode, 2)
self.assertFalse(json.loads((Path(directory)/"report.json").read_text())["gate"]["passed"])
if __name__ == "__main__":
unittest.main()