Foray 247 · BPB2 v0.3-pilot
Useful work, with inspectable scores.
This essai asks twenty different project questions. The test is whether the answer supports a real decision, survives checking and works in the browser.
What changed for this trial
GPT 6 Pro developed complete synthetic case packs from the twenty-scenario design brief. A separate Codex review retained all twenty, narrowed compulsory browser controls in cases 12, 17 and 18, clarified case 02’s sensitivity grid, and made case 15’s retesting penalty proportionate to its consequence. The target of a deliberately low success rate was removed.
This is a development trial of BPB2 v0.3-pilot, not a frozen benchmark or a claim that a newer model outperformed earlier essais. Essais 1 and 2 used twenty questions about one repair-and-reuse project; their results remain available under that original suite.
Recorded development-run outcome
The separate checked first-attempt score was 98.8/100, with 19 of 20 ready. Case 11 then received one assisted repair to its funding-summary calculation, taking 153 seconds. The checked current score is 99.9/100, with 20 of 20 ready. These are same-family AI grades supported by the recorded checks, with no Claude or human practitioner assessment and no adaptation exercises run. These near-ceiling same-family grades call for calibration with independent reviewers and contrasting models before freezing a comparative benchmark; they do not show that project-work benchmarking is solved. Inspect the recorded results and limits.
Project-work score: 0–100
Each dimension receives an integer 0–4. The weighted score is (30C + 25D + 20E + 15F + 10M) ÷ 4. The overall model score is the equal-weight mean of the twenty case scores. We show coverage, and leave the overall score pending until every case has been assessed. A retained failed or absent answer can score zero; an unreviewed answer is pending.
| Dimension | Weight | 3: usable | 4: strong |
|---|---|---|---|
| C · Correctness and constraints | 30% | Required arithmetic and logic correct; only minor defects. | Correct across required scenarios, with useful reconciliation and constraint checks. |
| D · Decision usefulness | 25% | Usable decision, owner, conditions and consequences. | Prioritised action, defensible alternatives and clear decision-switch conditions. |
| E · Evidence and uncertainty | 20% | Material claims sourced; facts, forecasts and proposals separated. | Contradictions resolved transparently; uncertainty and authority carried through the recommendation. |
| F · Completeness | 15% | All required completion conditions covered. | A coherent work product with boundaries, dependencies and follow-through. |
| M · Communication | 10% | Clear for its user; units, dates and qualifications close to claims. | Concise priorities and easy access to supporting detail; likely misreadings prevented. |
Across these dimensions: 0 means absent or fundamentally wrong; 1 means major defects; 2 means a plausible result with an important defect. A defensible alternative recommendation is acceptable. Scores are judgement on an ordinal scale; decimals do not imply scientific precision.
Browser function is scored separately
0: no usable browser answer, including download-only delivery. 1: critical content/navigation unusable. 2: readable but a required interaction or evidence path materially fails. 3: required interactions and navigation work with minor defects. 4: specified user actions work reliably, changes propagate, invalid states are handled and original assumptions can be recovered.
A readable decision brief with useful navigation and evidence disclosure can earn 4. Extra sliders, animations or a general optimiser do not earn credit by themselves.
Ready after checking requires every PM dimension at least 3, browser at least 3, and no open material failure. A strong initial result additionally needs PM at least 90/100. We do not tune the tasks to achieve a predetermined pass rate.
Material failures include a decision-changing calculation error, infeasible commitment, invented/erased evidence, invented approval or consent, or a broken required interaction. They stay visible alongside the score. Unnecessary retesting is normally an efficiency issue unless it creates a consequential error or falsely invalidates evidence.
Who scored the answer?
- Self-score: the answering model’s own ratings and reasons, recorded before feedback. These are not independent validation.
- Checked score: a separate Codex review using the case, candidate checks and observed interactions. The scorer and method are named. This remains AI assessment, not a practitioner panel or a Claude review.
- Your assessment: a named human or other model review you enter, with reasons. It is a separate local record and does not replace either AI track.
Do not average self-scores and checked scores together. Compare the same track, same suite and same answer stage across models. A future Claude or practitioner review should identify its actual model/person, date, method and evidence; the label alone is not verification.
The gallery switches between preserved first attempts and current answers. Corrections remain separate. A repaired answer cannot retrospectively improve the first-attempt score. Identical unchanged answers may share the same assessment with their provenance recorded.
How the attempts are made
Each case is assigned to a fresh gpt-6-astra agent with ultra reasoning, its individual model-visible pack and a neutral presentation shell. Case-specific solutions, evaluator notes, previous answers and the design-review history are not supplied. Agents are instructed not to inspect other case folders or reference material. They share a host, so this is procedural separation rather than an enforced security boundary.
The initial maximum is 45 minutes wall time per case, including calculations and self-testing. Tool/runtime clarification is recorded separately from substantive help. Unknown token usage or monetary cost stays unknown. First artifacts are retained before review or repair. The first development batch is 02, 03, 08, 12, 16 and 20; all twenty belong to the same essai.
Case pages contain the analysis and useful interactions, with the full supplied case available in a disclosure. Assessment widgets and source disclosure are common packaging. Their addition does not silently change the case analysis. The run record distinguishes original artifact and packaged page hashes where relevant.
The candidate evaluator keys were generated during design and checked selectively. They are evidence to inspect, not unquestionable truth. No score establishes universal project-management competence, observed organisational consent, professional certification or measured time savings. Actual practitioner checking and repair time need a separate measured trial.
Five later-evidence exercises are retained for a separately labelled adaptation stage. They are not mixed into the initial /100. Any adaptation results must show their own inputs, context, score and assistance.
Give the same job to Claude or another reviewer
The brief below can be used with a model that can operate a browser, or by a practitioner. Enter its actual name as the reviewer. A separate browser has its own local score history; use the assessment backup to bring its saved reviews into this browser.
Open the twenty-case review brief
Review all twenty first-attempt answers in this Essai 3 gallery under BPB2 v0.3-pilot. You are assessing the named answering model; do not claim to be that model or to have performed checks you did not run.
- Read the scoring method and supplied case records. For each case, establish the required decision, constraints and evidence before examining the answer. Avoid reading its self-score until your own ratings are recorded.
- On the assessment page choose the case and First attempt, then use the link marked ‘prior scores hidden’. This hides the displayed self and checked ratings for your review. Open each preserved first attempt this way. Recalculate decision-critical quantities from the supplied records and accept defensible alternative recommendations. Do not use external facts to override this synthetic case.
- Use the visible controls. Change a consequential input or option, inspect all affected outputs, follow evidence, and reset or reload. Test invalid values where relevant. Record exactly what worked or failed. A source-code inspection alone is not a browser test.
- Rate C, D, E, F and M from 0 to 4 using the published anchors. Rate browser function B separately. Explain each rating, cite the specific source or observed behaviour, and mark consequential calculation, feasibility, evidence, authority or interaction failures. Do not infer correctness from polish or from another score.
- On the assessment page, choose the case and First attempt, use your actual reviewer/model name, enter the six ratings and evidence, then save. Read the receipt, find it in History and reopen it. Corrections must create a new revision; retain the first assessment.
- Complete all twenty, including failed cases. Report the equal-weight PM score out of 100, coverage, readiness count and material failures. Do not average browser function into PM or mix your scores with the model's self-scores.
- Report actual checking time only if measured. State any inaccessible page, untested condition or uncertainty. Do not repair answers during this assessment; propose repairs separately. Export a score backup only after the saved history contains the intended twenty assessments.
No model or human assessment is assumed to be professional approval. Saving a score changes only browser-local review history; it does not approve the underlying project decision.
Saving your own scores
The assessment page saves in this browser only. The receipt identifies the case, reviewer, version, score and revision. History lets you find, reopen and correct it; corrections retain earlier revisions. No file is created unless you request a backup, and nothing is uploaded. A backup can be restored without duplicating identical receipts. Saves and restores use a shared browser lock; browsers without safe locking cannot write, and keep the entry visible. Clearing site data removes browser-local records. New assessments record the exact raw answer fingerprint from the run record, separately for first and current stages. Earlier unbound assessments and ratings of a changed answer stay in History but do not count towards a current whole-run score. To reassess, open the saved assessment, inspect the selected answer, explicitly confirm its version and save a new revision. Opening History retains displaced unsaved entries with a restore control; these drafts last only until the tab is reloaded.
Answer what-if controls are separate from assessment storage. Unless a case explicitly says otherwise, changing an assumption is ephemeral and reset/reload returns its supplied baseline. No control creates a real project instruction, contract, acceptance or patient agreement.