# Essai 2 — Lessons with a scope

19 October 2026. This new experiment follows the additive retention diagnostic of essai 1. It uses actual fresh language-model sessions to test using reason-bearing feedback across unfamiliar prose cases. Participant outputs are generated by the runner, not by the case author.

Three conditions have two fresh sessions each: a frozen reference with four original exploratory examples; ordinary access to those four plus four additional feedback examples; and the identical eight examples with explicit extraction of scoped lessons and retained principles. The additional examples introduce decision-use consequences and clarify an exploratory boundary. Both adaptation procedures retain the full feedback at test time. The preparation instruction is their intended difference.

Each session first prepares with at most 600 words, then receives the same twenty fresh test cases with at most 1,800 words to answer. It receives no test feedback. Ten cases concern exploratory use and ten decision use. Their related families include historical evidence with and without an applicability bridge; a bounded pilot versus an irreversible action called a pilot; investigating disagreement versus ignoring a contradiction; substantiated attestation versus seniority; and harmless styling versus an ambiguity that changes meaning. All were authored and frozen before participant runs.

The primary quantity is the count of sound judgements, separately for each use and each session. A sound judgement requires an action allowed by the frozen rubric and an adequate decisive reason. A fresh adjudicator sees anonymous final reports, case text and the fixed rubric, excluding condition identities and preparation. Raw judgements, any rubric challenges and the mapping are preserved. Action agreement, targeted questions, confidence, output length and preparation length are reported separately. Word counts are a burden proxy, not measured human time or total inference compute.

The fixed rubric contains eleven keep cases, eight ask cases and one reject case. Class composition and case dependence make accuracy alone especially weak evidence. No significance test, population confidence interval or best-run selection is used. All six sessions are shown. If a model already solves the cases using ordinary history or the initial examples, that result does not establish a benefit from the additional structure.

The author chose cases and rubric together, so this is still a synthetic pilot, not independent professional validation. The independent element is participant execution and blinded application of the frozen rubric, not authorship of a universally valid project standard. Wording can activate knowledge already in the model. Results do not demonstrate model-weight change, durable personal learning, real project outcomes or transfer to a distribution of organisations.

`freeze.json` fixes protocol, all packets, evaluator, author script and verifier before any participant session. `trial.py verify` checks the exact hashes; `trial.py summarize` validates actual saved reports and produces `results.json`. The page embeds that file and never fills absent scores. Every participant's unedited preparation and test report is retained under `reports/`. See `RUNNER.md` for execution and blinding details; participant agents must not read it during the trial.

The completed delivery also contains a nonfrozen verification wrapper, added after response capture without changing inputs or model text. It checks all captured raw, parsed, preparation and metadata hashes; validates all 120 answers; compares full raw-output word counts with the stated limits; recomputes scores; and reconstructs the entire summary and page in a temporary copy. The frozen summarizer's `output_words` field counts response content; the capture metadata and wrapper use the complete raw JSON output. Extract `source.zip` and run `python3 repo-only/verify_completed.py`. The archive includes the frozen evaluator and all participant and mapped adjudicator reports so its limitations can be inspected.


## Executed result and rubric challenge

All six fresh sessions completed within both declared output limits. The exact model identifier and reasoning setting were not exposed; all used the inherited configuration. The primary frozen-rubric scores were 17/20 and 17/20 for the original-feedback reference, 19/20 and 19/20 for ordinary history, and 19/20 and 19/20 for scoped lessons. Each session scored 10/10 on exploratory use; the decision-use scores were 7/10, 7/10, 9/10, 9/10, 9/10 and 9/10 respectively.

The blinded assessor awarded every one of the 120 explanations 2/2 for recognising the decisive substantive distinction. Every primary difference came from ask-versus-reject choices, with corresponding question-form differences. All sessions rejected J16 rather than asking; the two reference sessions also rejected J07 and J09. The assessor did not find unsupported acceptance or missed evidence in their reasons. Its fixed-rubric `unnecessary_block` label records the rubric's preferred response policy, not a finding that the underlying recommendations were ready to execute.

The original rubric and reports are unchanged. `analysis.json` records two separate post-assessment sensitivities. The stronger challenge allows rejection on J16 because permanently overwriting the only originals contradicts the claimed reversibility: scores become 18/20, 18/20, then 20/20 for the other four sessions. A broader post-hoc sensitivity also admits J07/J09 rejection under the actual participant definition: all six then score 20/20. The frozen instructions explicitly say reject is appropriate when supplied facts invalidate “the recommendation or its stated justification” and revision is needed. J07 calls irreversible commitments a pilot, while J09 presents earlier exploratory acceptance as validation of a budget saving. The participant answers reject those present justifications and allow revision. This is a task-instruction versus rubric mismatch, not merely a weaker semantic guess. It does not mean that asking and refusing generally have the same consequence.

Scoped lessons showed no advantage over ordinary history in this bank. Additional history may have shifted response policy towards targeted inquiry, but these outputs do not show superior identification of the material issues. The original examples, common instructions and model's existing reasoning already supported those distinctions. There was no zero-feedback baseline, so the trial cannot separate their contributions. A next study should align evaluation with the participant instructions, define the practical consequence of ask versus reject, and obtain independently assessed, genuinely disputed cases. Expanding this already easy bank would contribute less.

The unedited [blinded assessment](assessment.md) is published directly. Its captured anonymous adjudication files, pre-assessment mapping, mapped per-run judgements and sensitivity calculations are in the source archive. The completion verifier checks all capture hashes and mapped byte identities before independently checking both primary and sensitivity totals. One assessor is not a measure of inter-rater reliability.


## Editorial clarification after independent review

The assessor saw cases, the frozen rubric and anonymised answers; it did not see the common participant instructions. Its raw assessment therefore discussed ambiguity about rejection without the definition that participants had been given. The original stage-2 packet says: “reject when supplied facts already invalidate the recommendation or its stated justification and it must be revised.” The same definition appears in every stage-1 packet. This makes rejection of an invalid current justification explicitly available, not an interpretation introduced after scoring. The raw assessment, original rubric and primary scores are preserved; `analysis.json` and the page separately document this mismatch and the corresponding sensitivity. This limitation strengthens the reason not to interpret the primary score gap as improved judgement.