Learning from what we keep and reject
Does feedback teach a useful judgement—or only the last reviewer’s habits?
Executed synthetic experiment · 18 October 2026A deterministic decision tree learns from authored review labels. The features and adequacy rubric were designed together. This is an inspectable mechanism test; it measures neither ChatGPT adaptation nor professional or human benefit.
Three grouped folds. Every case tested once per condition.
01 · Change what the desk is doing
The team recommends keeping its delivery sequence. Should the desk keep the brief, ask for more, or reject the recommendation as submitted?
The purpose-aware checklist is perfect by construction: its author receives the adequacy rule. This is a useful reference, not an equally informed learning competitor. Where a standard can be agreed and stated, the experiment gives no reason to prefer learned inference.
“False accept” means keeping a brief the declared standard says to query or reject. Accuracy alone can conceal this.
02 · Open a brief
Change a field or select a square below. Each configuration replays a recorded held-out case; clicks do not retrain the learner.
A brief recommends continuing the plan. These fields are its entire machine-readable record; no prose or documents were interpreted.
Distance counts mismatched fields equally. All cases tied at the nearest distance vote. An exact vote tie prefers Ask, then Reject, then Keep.
Each square is one of 96 held-out briefs. Select it to see the facts and the fitted rule behind the outcome.
The evaluator is part of the lesson
The simulated reviewer asks adequate plain briefs to be polished, but keeps polished, complete briefs with no evidence. The learner reproduces that preference on every unseen exploratory case. The separate fixed scoring rule still asks whether the brief is adequate.
03 · Sources became a test
Primary methods were read before construction. These are adaptations and distinctions, rather than replications of the papers.
Concept → construction
Attribute tests selected by information gain become a visible categorical tree fitted from feedback labels.
Limit
Our unpruned, full-data variant omits ID3’s original windowing. The attributes already make the authored rule expressible; general project understanding is untested. Read: §§3–4.
Concept → construction
Classifying by nearby stored examples supplies an equivalent-history comparator with an explicit mismatch distance.
Limit
Equal field weights and tie voting are authored choices. The paper’s asymptotic results do not validate this tiny, categorical desk. Read: introduction and §§II–III.
Concept → construction
Separating action, evaluation and retained feedback helps distinguish the learner from the standard used to judge it.
Limit
This desk induces rules from labels; it does not generate verbal reflections. Reflexion’s evaluator choices and WebShop failure caution against treating any feedback loop as sufficient. Read: §3, Algorithm 1 and Appendix B.1.
Concept → construction
Its feedback-and-revision loop sharpens our question: does a lesson help a different case, beyond revising the same answer?
Limit
No self-feedback or language model runs here. The changed-use test concerns transfer between cases, not Self-Refine’s within-answer process. Read: §2, Algorithm 1 and §3.
What this first essai establishes
Within this constructed world, labels were enough to recover the old rule. They were not enough to determine when that rule should stop governing. Additional examples partly repaired the transfer and introduced new mistakes.
The wider foray remains open: can feedback improve useful project judgement across unfamiliar circumstances? A next trial needs independently assessed cases, interpretable reasons for disagreement and a real cost of collecting and maintaining feedback.
All 192 records are synthetic. Three folds cover the finite world, not a population of projects. Both presentation variants of a substantive bundle stay together. The standards, features and feedback were co-designed; held-out labels are withheld from fitting, not unknown to the experiment’s author.
The tree learns no language, asks no new question and estimates no calibrated confidence. The retrieval baseline uses a simple distance, not an LLM. Review time, adoption, outcomes and lasting model adaptation were not measured. The small correction batch is a single frozen choice, not a search for the best teaching curriculum.
The local hash receipt was saved before fitting and scoring. It is inspectable provenance, not independent preregistration. Results below pool all three prespecified folds; the complete per-case output and trees are downloadable.
Download the script, data, protocol and freeze receipt into one directory. Run python3 experiment.py verify, then python3 experiment.py score. Python 3’s standard library is sufficient. Scoring preserves the frozen inputs and replaces the result file; timestamps change, predictions do not.