Repeat and inspect

Same questions. Later answers.

Keep a fixed project case, ask the next model the same twenty questions, and inspect what changes.

The first set is a developed, model-assisted baseline dated 21 October 2026, using gpt-6-astra, ultra reasoning. It was iterated, tested and repaired with parallel assistance. It is not a blind first attempt. No human review scores, comparative improvement or client outcomes have been inferred.

The next run

  1. Download the version 1.0.0 question pack. It contains the full fictional case, task records and twenty prompts, without these developed answers. Keep the pack unchanged and retain its recorded hash.
  2. Select the latest suitable model available at the time. Record its exact identifier, date, reasoning setting, tools, model access route, time and assistance budget. “Latest” is a run-time choice, not a permanent claim for this page.
  3. Give each question a fresh context containing the fixed inputs and its prompt. Use the same instructions, tool access and budget across compared runs. Return an interactive HTML/JavaScript answer or JSX with a working renderer; tables and briefings remain inspectable in the browser.
  4. Retain all twenty first answers before repairs. Log corrections, assistance and final answers separately. Record actual time, cost and failed attempts if available; leave them unknown otherwise. Do not silently replace a past run.
  5. Test calculations against independent expectations and try the controls. Review each answer against the rubric below. Where practical, conceal model names and alternate the order during human comparison.
  6. Keep the answers, input pack, execution record and review together as a dated run. Add its twenty tiles as a new run in this foray. The gallery can import a review locally for comparison; it does not execute or publish imported HTML and does not call a model.

What to compare

Rate correctness, usefulness, evidence, clarity and interaction separately: 0 fails, 1 needs substantial repair, 2 useful with limits, 3 strong within the stated scope. Record the reason and any serious defect. A polished answer cannot compensate for a wrong calculation.

The gallery compares only dimensions rated in both runs. Matching version labels and assistance classes are necessary but not sufficient: check the actual inputs, settings, tools, budgets and reviewer before interpreting a difference. An assisted run and an unassisted run should be shown separately. Repeated runs help distinguish improvement from sampling variation; twenty selected questions cannot represent all consulting work.

Scope of these first answers

Every record and economic figure is fictional. The models demonstrate specified rules and assumptions. The exception-radar labels are supplied fixture judgements, meeting delays use a stated arrival model, posterior probabilities depend on supplied likelihoods, and plans use simplified dependencies and costs.

Question 14 uses fixed case evidence rather than a live regulatory review. Question 17 is an inert screening exercise, not a security evaluation. Question 18 compares eight authored policies, not eight independent agents. Question 19 uses fictional tool archetypes, not current vendor advice. Question 20 replays synthetic records; it connects to no live service. These choices make the first pack repeatable and are visible in the answers.

The original App007 proposed targets such as faster recognition and fewer errors. Those remain proposed targets, not observed achievements. All twenty original prompts, source HTML fragments and tag definitions are retained. The original spreadsheet and slide requests are now interactive browser tables and briefings.

Research options

HELM suggests broad, explicit evaluation dimensions and transparent adaptation conditions. AgentBench is a reference for evaluating interactive tasks. Neither validates this question set. Practitioner review, repeated trials, counterexamples, reproducibility and effort to repair are additional possible methods.

The emblem borrows the idea of a survey benchmark: a fixed reference across different routes. The paths do not assert an improving score.