Original examples only
Four exploratory judgements with reasons. The model prepares ordinary working notes, then faces both uses.
Intentionally less information. This separates extra feedback from the method used to prepare with it.
The comparison is ready.
Twenty fresh prose cases test both exploratory and decision uses. Each participant receives actual reasons for earlier judgements. The trial compares ordinary access to those examples with an explicit attempt to extract scoped lessons.
The first essai’s additive replay found that new-use correction damaged an old evidence check. This trial changes the mechanism: a language model interprets prose, reasons and purpose.
One difference to inspect
Same feedback. Different preparation.
Read the additional feedback
The same original four, plus four corrections and explanations. The model can use the history as an ordinary reviewer would.
A capable comparator, with all reasons available throughout the test.
Explain the lesson’s boundary
The identical eight examples. Preparation explicitly states principles, applicability, what remains valid and what would revise a lesson.
No extra facts or test answers. The change is the preparation instruction.
All sessions: at most 600 preparation words and 1,800 test words; two calls, twenty cases, no evaluation feedback between cases. Equal visible budgets do not imply equal hidden inference compute.
All runs, both uses
Preservation counts alongside improvement.
The primary score requires both the action specified in the frozen rubric and the decisive reason. One fresh assessor applied that rubric with condition identities and preparation withheld. All 120 explanations received full reason credit; every lost primary point concerns ask versus reject.
| Session | Fixed rubric /20 | Explore /10 | Decide /10 | Reason points /40 | Prep words |
|---|
After assessment · primary scores preserved
When a defensible refusal looks like failure.
All sessions rejected J16: a supposedly reversible trial would permanently overwrite the only originals. The frozen rubric required a question about recovery. The assessor found rejection defensible because the supplied facts already contradict the claim of reversibility. This is the strongest challenge to the scoring rule.
The two original-feedback sessions also rejected J07 and J09, while the other sessions asked targeted questions. Their reasons identified the missing migration case and unsupported transfer of a saving into a budget. The frozen participant task explicitly permitted rejection when supplied facts invalidate “the recommendation or its stated justification” and it must be revised. J07 understates irreversible commitments as a pilot; J09 uses exploratory acceptance as validation for a budget saving. Allowing rejection of those current justifications therefore follows a meaning already supplied to participants.
| Condition · two sessions | Fixed rubric | Allow reject on J16 | Allow rejecting the current justification |
|---|
Every value is a session score out of 20, not a pooled estimate. Both alternative readings are post-assessment sensitivities; neither replaces the frozen primary result. The broader sensitivity addresses a mismatch between participant instructions and the ask-only rubric; it does not establish that asking and refusing are generally equivalent.
Editorial clarification: the blinded assessor saw the cases, rubric and anonymous answers, but not the common participant instructions. Its raw report called the meaning of rejection ambiguous without seeing that definition. The report is preserved unchanged; this clarification follows an independent review of the frozen task instructions.
Read the assessor’s unedited assessment · Inspect the separate sensitivity calculations
Inspect the actual response
The distinction is in the reason.
Start with J16, then compare J07 and J09 against the task’s explicit definition of rejection.
What the models could learn from
Eight examples, with their reasons.
The frozen reference receives F01–F04. Both adaptation procedures receive all eight, in the same order. The test cases were withheld during preparation.
A bounded test of using feedback.
These are authored project scenarios, not client records or observed outcomes. Case families are related. Two fresh sessions per condition make response differences visible; they do not estimate a population or establish durable model learning.
Success can reflect better use of knowledge the model already has. Scoped lessons matched ordinary history in both sessions. The original-feedback sessions also identified every decisive substantive distinction according to this assessor. That is compatible with the earlier examples and the model’s existing reasoning already being sufficient for this bank; there was no zero-feedback condition. The extra history may have influenced response policy. A more demanding next test should align the scorer with the participant instructions, clarify the practical consequence of asking versus rejecting, then use genuinely disputed cases assessed independently. The rubric was fixed before participant outputs, but authored within the same experiment.
Protocol, execution and integrity
Extract the complete source archive, then run python3 repo-only/verify_completed.py to verify capture hashes, raw-to-parsed equality, complete-output word limits and exact reconstruction of the saved scores and page. The unchanged trial.py checks frozen inputs and summarises reports. Neither script calls a model or fabricates missing reports. Preparation and test outputs are retained separately.