# A review desk that learns

Foray 238 · first essai · 18 October 2026

An executed synthetic mechanism test of learning from review feedback. It does not test a language model, human judgement, real project outcomes or durable personal adaptation.

## Question and constructed world

Can a feedback-induced rule classify unfamiliar cases, survive a change of use, and distinguish reviewer preference from an explicitly stated adequacy standard?

The author specified all combinations of six categorical fields: evidence (absent/cited/checked), currency (old/current), scope coverage (partial/complete), contradiction (clear/unresolved), accountable attestation (absent/present), and presentation (plain/polished). Their 96 combinations occur under two uses, exploratory briefing and decision recommendation: **192 cases**, with no private records, collected human feedback or real outcomes. The prose on the page is narrative framing, not model input.

The original adequacy rubric rejects an unresolved contradictory recommendation, asks for absent evidence or incomplete scope, and otherwise keeps the brief. Decision use additionally requires current evidence and either checked evidence or an accountable attestation of cited evidence. Attestation concerns the evidential claim, not permission to commit project resources. The decision class is therefore sparse: 6 of 96 cases are adequate, compared with 16 in exploratory use. The class counts and full confusion matrices must accompany accuracy.

The simulated biased reviewer queries adequate plain briefs for polish and keeps polished, complete, uncontradictory briefs despite absent evidence. These are deliberately authored biases, not findings about real reviewers. The separate adequacy rubric remains fixed. All reasons are displayed for inspection; the algorithms receive only field values and feedback labels, not textual reasons.

## Frozen comparison

The complete data, exact splits, protocol and executable were hashed at **2026-10-18T16:11:07.207039+00:00**, before any model fitting or scoring in this run. See `freeze.json`. This is local inspectable provenance, not independent preregistration. No results-driven tuning, alternative seed selection or post-freeze code amendment was performed.

Forty-eight substantive bundles exclude presentation and use. A deterministic SHA-256 ordering assigns them evenly to three folds. Each fold trains on 32 bundles (64 exploratory records) and evaluates 16 untouched bundles (32 cases per use). All presentation variants and uses of an underlying bundle remain together. Thus the same substantive bundle cannot occur on both sides of a fold. Other feature combinations remain similar in this deliberately small factorial world; this is not a claim of semantic distance between real documents.

A further 16 training bundles are selected by a separate, fixed hash for decision-use correction: 32 records per fold. Their labels are added to the original 64, then the tree is refitted. The unchanged decision test cases are never selected as feedback. The three folds are pooled, so every case is evaluated once per condition. Folds are not independent trials; no population confidence interval or significance claim is made.

Procedures:

- **Frozen checklist:** exactly states the old exploratory standard. It is a deliberately credible ordinary baseline, supplied with that rule rather than induced from examples.
- **Nearest history:** unweighted categorical Hamming distance over all seven fields, including use and presentation. All nearest-distance ties vote. Tied labels prefer Ask, Reject, then Keep. It receives precisely the tree's feedback records; it is not semantic retrieval or an LLM with history.
- **Learned tree:** an ID3-inspired, full-data categorical information-gain tree. It greedily splits on the highest-gain field, using the fixed feature order for equal gain, and stops at purity, no remaining fields or gain ≤ 1e-12. Majority ties use the same fixed order; unseen categories use node majority. There is no pruning, original-ID3 windowing, LLM or access to the rubric inside fitting or prediction. The rule is actually induced; it is not supplied as a learned answer.
- **Purpose-aware reference:** directly implements the full rubric and is perfect by construction. It has privileged rule information and is not an equally informed learned competitor.

## Executed results

Each cell is correct / 96; parentheses are false accepts.

| Condition | Frozen checklist | Nearest history | Learned tree |
|---|---:|---:|---:|
| Same use | 96 (0) | 90 (0) | 96 (0) |
| Changed use, old feedback | 86 (10) | 80 (10) | 86 (10) |
| Changed use, added feedback | 86 (10) | 70 (4) | 88 (8) |
| Biased reviewer, scored for adequacy | 96 (0) | 81 (3) | 84 (4) |

The correction batch repairs six tree errors and creates four others. The net improvement is two cases, not a clean repair. Its decision-use adequate retention is 6/6; nearest history retains only 2/6 after correction. Under biased feedback, the tree matches 96/96 reviewer labels but retains only 8/16 adequate briefs and makes four false accepts. This separates imitation from adequacy in the constructed world.

The unchanged ordinary checklist is equally strong in the original use. An explicit purpose-aware checklist resolves all cases by construction. The trial supports no benefit from preferring learned machinery where the agreed standard is already known.

All primary metrics were declared before scoring: three-way accuracy, macro recall, false accepts, adequate retention, acceptance precision and confusion matrices. Repair/regression counts and the default illustrative docket are post-scoring explanatory selections; every case remains inspectable. No claimed score depends on selecting that example. `results.json` includes fold metrics, predictions, tree routes, nearest examples and fitted trees.

## Limits and reproduction

The author co-designed features, rule and task; the rule fits the learner's hypothesis class. The evaluator is separate from imitation labels but not independently authored. Synthetic standards carry no authority in a live project. The change-of-use category has no semantic meaning to the tree; the learner cannot infer unstated policy from its name. The modest correction set is one prespecified curriculum; other batches, algorithms, metrics or field weights may behave differently.

Feedback costs are counts, not measured human time: 64 initial and 32 additional labels per fold. Initial trees have 10/8/8 nodes; corrected trees 16/12/12. Retrieval compares 64 or 96 records per prediction. No review-effort saving is established. Support in a tree leaf is not calibrated confidence. No treatment of disputed real outcomes, free-text explanations, active questioning or long-term forgetting is tested.

Place `experiment.py`, `data.json`, `protocol.json` and `freeze.json` together, then run:

```sh
python3 experiment.py verify
python3 experiment.py score
```

Python 3 standard library only. Scoring refuses modified frozen inputs and writes `results.json`. Repeated runs reproduce the predictions, tree structure and scores; the scoring timestamp changes. Python 3.12's changed floating-point summation can alter the last digits of diagnostic information gains. The original Python 3.9 run remains frozen: a Python 3.12 rerun differed only in one gain by 2.22 × 10⁻¹⁶. Verification allows an absolute difference of at most 10⁻¹² in tree `gain` metadata only; every prediction, route, split, count and score must match exactly. This comparison tolerance does not change the learner, its tie handling or the saved result.

To regenerate a fresh receipt, use the unchanged script's `prepare` action in a fresh directory; it refuses to overwrite an existing freeze. The interactive page embeds the executed outputs and works without a network connection.