Essai 3 · 18 November 2026

What feedback actually identifies

A correction can make a fitted reviewer worse on something it previously understood. Keeping several possible rules alive changes the kind of learning we can promise.

In the first essay's synthetic experiment, adding stricter decision feedback improved the fitted tree from 86 to 88 correct decisions out of 96. But performance on its original exploratory work fell from 96 to 92. Four new false acceptances appeared. The new attempt asks which judgments the feedback actually determines, before choosing any single reviewer.

Follow one correction

This is a new analysis of the first essay's published 192-case fixture, using its original three grouped folds and feedback schedule. It is not a new language-model trial or a measurement of professional benefit.

Show the feedback that fixes these bounds

The assumptions do real work

For this authored world, judgments are ordered reject → ask → accept. Current evidence, wider coverage, cleared conflict and an attestation may make a judgment more permissive; none may make it less permissive. Evidence moves from absent to cited to checked. A case good enough for a decision must also be good enough to explore. Finish is initially irrelevant. These are explicit modelling choices, not universal properties of review.

With no ordinal relationships, exact matching identifies none of these held-out cases. The abstraction creates the transfer. When the presentation-biased feedback contradicts the chosen relationships, the method stops: the assumptions and feedback cannot jointly hold. It does not quietly discard the inconvenient example or declare a human mistaken.

Replace a selected rule with an identified interval

Every feedback item below a case in this order supplies a lower bound on its permissible judgment. Every item above it supplies an upper bound. Take the strongest of each. The resulting interval contains exactly the labels attainable by monotone rules consistent with that feedback.

lower(x) = max of observed lower labels below x
upper(x) = min of observed upper labels above x

The defaults are reject and accept. A singleton is identified under the assumptions. A wider interval means several rules remain possible. Unidentified is not the review action “ask”. It describes our knowledge of the rule; asking about the underlying case is a separate decision.

Why are the bounds tight? Assign any label inside the interval to the new case. No ordered pair now has a lower bound exceeding the other's upper bound. The lower-envelope rule—at each point, take the greatest lower label below it—is a monotone extension satisfying every interval. Thus every displayed label has a compatible rule. This also explains why adding consistent feedback can only narrow an interval. An already identified judgment cannot change.

What the attempt earned

Default assumptions; 96 held-out judgments per stage
StageIdentifiedStill openOld fitted tree correct
Original purpose, before742296
Stricter purpose, before445286
Stricter purpose, after643288
Original purpose, after742292

All 384 displayed intervals contain the fixture's authored answer, and every identified label is correct in this finite world. The 96 rows comprise 48 underlying briefs shown in plain and polished form. In the stricter setting, unique three-way decisions rise from 22/48 to 32/48. The two accepted rows are the two renderings of one brief. For the narrower decision “May this proceed?”, settled answers rise from 35/48 to 39/48. After correction 38 cannot proceed and one can. The remaining nine may or may not proceed. Under the published comparison, the old tree still makes eight false acceptances. The cost is visible: this method refuses to select one answer for 32 stricter-purpose cases. The old tree's perfect first-stage score shows that conservative identification is not automatically better predictive accuracy.

The improvement is a different learning contract: preserve justified conclusions, expose the assumptions enabling transfer, and use fresh feedback to narrow the unsettled region. It is not a claim that professionals should accept this ordering. If ask and reject are incomparable, evidence can be actively misleading, or one purpose changes what “better” means, a different partial order or a set-valued model is required.

Disagreement and the source

Aroyo and Welty argue that disagreement in annotation can represent ambiguity and differing interpretations, rather than defective workers. That motivates retaining disputed ask/reject feedback as an interval and showing incompatible assumptions. Their paper does not supply the ordinal model or prove its bounds; those are constructions for this essay. Truth Is a Lie (2015), pp. 15–16 and the seven myths.

The preceding small language-model study also found that an ask/reject distinction changed the apparent result. This analysis does not reuse its outcomes as new training evidence. It supplies an inspectable way to represent that kind of ambiguity before claiming a rule has been learned.

Checks and reproduction

Independent Python and JavaScript implementations agree on all 384 bounds. Tests check all 192 points in each fold before and after correction, complete enumeration of small monotone worlds, disputed-label widening, conflicting feedback, and withdrawal of transfer assumptions. These verify the finite construction, not the professional appropriateness of the order.

Frozen inputs and historical trees · Independent Python results · Python calculation · Browser model · Reproduction package

This reader keeps no record. Selections affect only this page; reload restores the opening state.

Return to the foray