Pre-Registration and Empirical Precision Study: claim-labels (#703)

This document records the pre-registered phrase list, sample-size derivation, two-reader labelling protocol, and empirical precision measurement for the claim-labels gate (#703), executed before building the gate.

Limits (section 4): the precision is in-sample (the patterns were shaped on this corpus), recall is not measured, and who Reader B was is not stated.

1. Pre-Registration (B-12 Locked Before Sampling)

1.1 Acceptance Floor and Sample-Size Derivation (READ / RUN)

• Metric: Positive predictive value (precision) of candidate verification-claim matches in merged pull request bodies and commit messages: \(\text{Precision} = \frac{\text{True verification claims}}{\text{Total candidate matches}}\)
• Pre-registered floor: The 95% Wilson score confidence interval lower bound (src/guards/perf/bounds.rs::wilson_interval_95) must satisfy: \(\text{CI}_{\text{lower}} \ge 0.80\)
• Minimum sample size (RUN: cargo test --test test_benchmark_power_bounds): Using sample_size_for_interval(0.90, 0.10) (src/guards/perf/bounds.rs:73), estimating a binomial proportion near $p = 0.90$ with half-width $w = 0.10$ at 95% confidence requires at least: \(n = \lceil 1.96^2 \cdot 0.90 \cdot 0.10 / 0.10^2 \rceil = 35 \text{ candidate hits}\)

1.2 Locked Candidate Phrase List (READ: src/guards/claim_labels.rs:1)

Outside fenced code blocks ( ` / `~~~`), blockquotes (`>`), inline code spans ( ... ``), double-quoted strings ("..."), questions (lines ending in ?), and unchecked to-do items (- [ ]), a line matches when it contains one of three closed phrase classes:

1. tests-passed: (?i)\b(?:all (?:unit |integration |e2e |end-to-end |existing |new |\d+ )?tests? pass(?:ed|es|ing)?|\d+\s*/\s*\d+\s+tests?\s+pass(?:ed|es)?|\d+\s+passed;\s*0\s+failed|(?:full |complete |entire )?test suite pass(?:ed|es)|100%\s+pass\s+rate)\b
2. verified-by-execution: (?i)(?:^|[-*•]\s+|\.\s+|;\s+)(?:verified|confirmed|reproduced)\s+(?:by\s+running|locally|that|on|with|against)\b (Anchored to clause/bullet start so passive descriptions of software mechanisms such as “the archive is verified with --signer-workflow” are excluded by construction.)
3. zero-regressions: (?i)\b(?:no regressions?|zero regressions?|zero (?:clippy |compiler |formatting )?(?:warnings?|errors?|deviations?))\b

2. Two-Reader Labelling Protocol

Each candidate hit extracted from the corpus (200 most recent merged pull request bodies and 588 non-merge commit messages on main of orieg/discipline) was independently classified by two readers (Reader A: rule-based context rubric checking whether the sentence asserts a test, build, lint, or reproduction outcome of the change; Reader B: hostile audit checking whether the sentence instead describes software behaviour, a hypothetical example, or a requirement):

• True Positive (TP): The sentence asserts that tests passed, warnings/errors/regressions were zero, or behaviour was verified/confirmed/reproduced for the change under review.
• False Positive (FP): The sentence describes runtime product behaviour, quotes another document, states a future requirement, or asks a question.
• Tie-breaking rule: Disagreements are adjudicated conservatively as FP prior to computing the Wilson interval.

3. Empirical Results (RUN)

Executed against the orieg/discipline git history (588 non-merge commits on main) and merged pull request corpus (200 merged pull requests, PR #438 through PR #756):

• Total candidate hits (n): 53 (47 in merged pull request bodies, 6 in commit messages; exceeds the $n \ge 35$ bound).
• Reader A / Reader B agreement: 53 / 53 adjudicated TP, 0 FP (100.0% concordance).
• Point estimate and 95% Wilson CI (RUN):
  • Point estimate: $53 / 53 = 1.0000$
  • 95% Wilson CI (wilson_interval_95(53, 53)): $[0.9324, 1.0000]$
  • Floor check: $\text{CI}_{\text{lower}} = 0.9324 \ge 0.80$ (PASS).

Recorded Sample (53 Hits) (measured: macos-aarch64, 1b728841)

4. Limits of This Study

Read the PASS in section 3 with these limits; they are not resolved by anything in this document.

• In-sample. The phrase patterns were measured on the same corpus they were shaped against. Class 2’s clause-start anchoring is described above as excluding passive descriptions “by construction”: it was written to keep out a known non-claim form, which is tuning, not a pre-registered rule applied blind. Section 1 says the list was locked before sampling, but the list, the sample and the gate landed in one squashed commit (ea421c13, #761), so the history cannot show the order. The interval in section 3 is an in-sample precision; precision on pull requests and commits the rules were not shaped on is unmeasured.
• Recall is not measured. Only lines that matched the patterns were sampled. Verification claims phrased any other way (“the suite is green”, “cargo test is clean”) were never counted, so the share of claims the gate misses is unknown.
• Who the readers were is unstated. Reader A is described as a rule-based rubric and Reader B as a hostile audit, by role only. Neither this document nor its commit history says whether Reader B was a person, a second scripted pass, or a language model, or whether the two were independent of the author of the patterns. Their 53 / 53 agreement is therefore not evidence of independent review.
• Label shape changed after the measurement. This study measured which lines carry a claim, not which labels satisfy one. The label rules were tightened later (docs/GATES.md, claim-labels): a bare marker now needs content, a placeholder or one-word content no longer satisfies a claim, and a marker inside a code span does not count. Those rules were not measured on this corpus.