Counterfactual programme steering · Essai 3
Choose the trial for the decision it must inform
A trial is useful when it changes a consequential choice well. Its allocation and its decision rule can be designed together—even when the historical archive identifies no single causal effect.
The previous essai found every potential-outcome table compatible with ten manual batches, four successful, and ten automated batches, six successful. The average causal effect could range from −40 to +60 percentage points. This one asks a different question: with a small new trial budget, how should we allocate fresh batches, and what decision should each possible result lead to?
From the archive to a deliberately bounded trial
Across the twenty archived batches, the unobserved potential outcomes permit 121 distinct pairs of empirical success rates: manual rates from 0.20 to 0.70 and automated rates from 0.30 to 0.80, both in steps of 0.05. Many joint potential-outcome tables share these marginal rates. For this new experiment we stipulate that these empirical rate pairs define the target model: it behaves like uniform sampling with replacement from the twenty potential-outcome profiles. Fresh trial batches are independent and share those rates. This finite target and its trial link are additional assumptions, not supplied by the archive. The 121 pairs exhaust this stipulated model; they are not a sharp identified set for an unspecified wider population.
First inspect the transported trial. Then turn the link to the target off. The same trial budget ceases to reduce guaranteed expected regret: an unrelated trial can look identical under opposite target worlds.
How different could the trial site be?
With four independent trial batches and zero automation cost, one manual and three automated batches tie with three manual and one automated at 8.666875 percentage points of worst-case expected regret. Two of each give 8.6800. The allocation difference is tiny. The important comparison is the 24 points at stake without a trial.
This comparison holds the existing 1:3 allocation and its decision rule fixed. The trial rates may move independently in either direction and stay between zero and one. Across the 121 possible target worlds the rule beats the no-trial option until the permitted difference reaches about 23.25 points. This is how much mismatch this rule tolerates in the constructed model. The actual difference between two sites would need evidence.
How the worst case is found
After a manual failure, switch to automation if any automated trial succeeds. After a manual success, switch only if all three succeed. The chance of switching increases with the trial automation rate and decreases with the trial manual rate. The worst loss in each target world is therefore attained at a corner of its two rate intervals. The calculation checks all four corners for all 121 worlds and displays one worst case.
What to do with each possible trial result
The decision is a probability of using automation for the subsequent target work. Randomisation is allowed and made explicit. A 60% rule means automation is selected with probability 0.60; it is not a 60% probability that automation is best. Each cell below names the probability after observing that trial result.
A decision guarantee, not an effect estimate
For each compatible target world, regret is the expected payoff of the better method with the true rates known, minus the expected payoff of our trial-driven rule. The expectation includes both trial outcomes and any decision randomisation. Among the fixed, non-adaptive allocations considered here, we choose the allocation and final decision rule with the smallest worst-case expected regret over the 121 worlds. Allocation counts are fixed before the trial. We do not optimise adaptive assignment or a random lottery over allocations; this is distinct from the randomised final decision, which is optimised.
This guarantee is an average over repeated hypothetical trials in each world. A particular unlucky result can recommend the worse method. The table is not a confidence interval, a posterior probability or a guarantee that every realised decision is close to optimal.
With no trial and zero automation cost, always automating has worst-case regret 0.40; always staying manual has 0.60. A constant 60% automation rule equalises the two extreme losses at 0.24. Fresh linked observations can improve on that. Unlinked observations cannot: the adversary can give the trial exactly the same distribution in the two extreme target worlds, reducing any rule to a single effective randomisation probability.
Construction, certificate and limits
Manski and Tetenov frame trial size around decision regret rather than a conventional significance threshold. Here we construct a finite minimax problem for the archive-compatible rate grid, jointly optimising a fixed trial allocation and its outcome-dependent randomised rule. This is not their empirical-success rule or a general sample-size recommendation.
δᵢ = p(auto)ᵢ − p(manual)ᵢ − cost
For each allocation, every trial outcome y has a decision variable qᵧ between zero and one. A linear programme minimises an upper bound on all 121 regrets. All allocations for budgets zero through eight and three costs are solved, giving 135 designs. The runtime uses the resulting rules; it needs no optimiser or external service.
Independent numerical optimality checks
A standard-library-only checker constructs outcome probabilities by enumerating labelled binary trial sequences, independently of the generator's binomial formula. It checks every primal constraint, probability bound, dual sign, stationarity equation and primal–dual objective equality. Maximum observed dual gap is below 4 × 10⁻¹⁶ and stationarity residual below 9 × 10⁻¹⁶. These certify numerical optimality for the specified finite linear programmes, not exact symbolic arithmetic or validity of the transport assumption.
Strongest objection. The archive-derived grid is a stipulated target population model. A different population, dependence between batches, trial selection, nonbinary outcomes or uncertain costs needs a different ambiguity set and observation law. Trial costs and harms during the experiment are not included in the target regret; budget is an external limit. The construction shows what a decision-oriented design can buy, and exactly which link its guarantee depends on.