Essai 4 · Benchmark-method study · 18 November 2026
Test the change, not one preferred answer
A believable revised total can conceal a vanished request. A perfectly feasible alternative can disagree with the benchmark's preferred answer. The checker needs to recognise both.
The preceding work showed that attractive browser answers and high average scores did not settle whether an update preserved the underlying job. In a later four-case local pilot, an allocation changed from 23 to 20 without making the three unserved requests explicit. This attempt tests a checking method on a completely enumerable fictional problem. No new language model was run, and no model ranking is updated. The twenty-case benchmark and earlier answers remain intact.
Which opening schedule would you commit to?
All four openings give A through E a place. M is morning, L is late and R is remote. Choose a schedule before finding out what changes. For each possible disruption the table searches every maximum-service repair and counts how many people would have to change assignment. A deferral counts as a change.
| What changes | People served | Fewest moves | Closest repairs, A–E |
|---|
Giving C the late slot makes the permission withdrawal harmless. The two openings with C on L require zero, one and one moves across the three shocks. The other two require two, one and two. A receiving planner can promise all five places today and at most one reassignment under these named changes. The data support that choice without assigning probabilities or deciding whose deferral is fairest.
Five workshop requests, two changed facts
A–E request workshop places. M has two morning places, L has two late places, and R has one remote place. Each request has its own permitted options. The stated objective is to serve as many requests as possible; no priority between people is specified. An unserved request must remain explicitly deferred.
| Request | Options |
|---|---|
| A | M or L |
| B | M only |
| C | L or R |
| D | M or L |
| E | L or R |
| Request | Outcome |
|---|
| Check | Result |
|---|
Check every candidate, not a handful of convenient failures
Five requests with four outcomes each give 4⁵ = 1,024 complete candidate vectors. We checked every one in each of the four factual worlds: 4,096 verdicts. An independent Python generator builds only permitted plans while decrementing remaining places; the browser separately generates all vectors and rejects violations. They agree on every maximum-service plan.
| Facts | Legitimate maximum-service plans | Bad plans accepted by totals-only check | Good plans rejected by one-reference check |
|---|---|---|---|
| Original | 4 | 26 | 3 |
| Lost L place | 10 | 50 | 9 |
| C withdraws R | 2 | 28 | 1 |
| Both changes | 5 | 55 | 4 |
The totals check is given the correct achievable service count and checks resource capacities. It still misses forbidden allocations. Separate identity mutations show why five rows and the correct number served cannot establish that all five original requests survived. The complete constraint check has no false verdicts within the enumerated language. This is a finite method check, not a measured error rate for AI outputs or all business consulting work.
A tempting update rule fails
One might require an update to leave every unaffected person in place. Start with A→M, B→M, C→R, D→L and E→L. C now withdraws R. Keeping everyone else fixed leaves no place for C, so only four are served. Yet moving E to R frees L for C and serves all five. Every maximum-service repair moves E, even though E's own facts did not change.
The correct invariant is on the feasible set: withdrawing an option cannot create a newly feasible allocation. It is not an invariant requiring the selected plan to preserve all unaffected assignments. The reader reports each changed or missing request, so legitimate rearrangement remains visible rather than forbidden.
With the L outage as well, five requests compete for four places. There are five maximum-service plans, deferring different people; some defer the same person while allocating others differently. The constraints do not supply a fairness policy. A benchmark should not invent one by treating its first valid plan as the sole answer.
What should enter the next benchmark run
For an allocation-style case, freeze the request identities, permitted options, capacity facts and explicit service objective. Ask the updated answer to preserve each original request with a served or deferred outcome, and expose changes from the prior plan. Check feasibility and the stated objective, while allowing the legitimate alternatives. Then apply small fact changes and inspect the relationships between outputs.
The tested relations are: a simultaneous renaming of identities preserves validity; reordering rows preserves validity; removing a permission or capacity shrinks the feasible set; and these two independent fact changes commute when interpreted as facts. A more general claim that all narrative edits commute would be false. The independent Python generator checks feasible and maximum-service allocations. Separate JavaScript tests exercise row order, the stated renaming and the two changes to the facts. These checks do not simulate committing two changes in sequence.
This gate could reject a polished but invalid answer before style scores inflate the result. It cannot assess negotiation, stakeholder legitimacy, writing quality or whether throughput is the right objective. Nor does it prove a generative model will emit a usable record. Applying it to the full twenty-case suite requires a justified case-specific contract and a fresh, correctly identified model run.
The methodological source
Chen, Cheung and Yiu propose deriving follow-up tests from relationships expected between inputs and outputs when an individual answer is difficult to judge. Here that operation becomes a permission withdrawal and a capacity change with explicit set relationships. Their method still depends on correct domain knowledge. The complete finite enumeration and counterexample to “leave unaffected people fixed” are this essay's checks, not findings from their paper. Metamorphic Testing: A New Approach for Generating Next Test Cases (1998), §§1–3.
Independent exhaustive results · Inspectable browser model · Reproduction package
This is an ephemeral fictional checking exercise. No booking, request or file is saved, sent or changed outside this page. Reload restores the original facts and plan.