The Panel Loop

Why this shape

The loop is built around two specific failures. Both belong to goal-driven agent testing — the default way people point personas at a product — and both are invisible to it by construction.

Failure one: a goal-driven agent pushes through anything

Give an agent a persona and an objective and it will pursue the objective. That is the point, and it is also the problem: it walks straight past a wrong icon, a broken hierarchy, a cheap-looking panel, a contradiction between two numbers — because none of those stop it reaching the goal. A related loop on another project ran 142 times with 35 personas and never once mentioned that the header rendered a sun icon in dark mode, on every page. A human opened the site and found it in under a minute.

So the first thing a persona does is not the goal. It is a blind reaction pass: what is this, where does my eye go, does anything look wrong, would I trust this with money. Reaction before intent.

Failure two: perception is not a goal property

One landing page measured 21.6 screens, 3,536 words, about sixteen minutes of reading. Two personas drove it and neither mentioned the length — because a persona pursuing an outcome scrolls past twenty-one screens without ever experiencing twenty-one screens, and an agent scrolling by script feels nothing at all. The owner opened it once and said it felt "long and way too long and uncomfortable. Not well proportioned at all."

The fix is not to ask an agent to have feelings. It is to make the property measurable, and to measure it before the page has been absorbed — then ask the question in character while the number is still surprising. That is why the console measurement lives inside the blind pass and not in the goal pass.

What makes it generative

A defect loop asks is this broken? A generative loop also asks what should this be? — and treats the answer as a first-class output rather than a comment. Three mechanisms do the work.

1. Proposals are structured, and the cost field is mandatory

Every proposal carries four fields. Three are easy. The fourth is the entire mechanism.

costs is what stops the loop converging on mush. Without it, every persona asks for the thing that would make the product easier for them, each request sounds reasonable in isolation, and forty runs later you have a beige average of eight incompatible preferences. Forcing the cost makes the conflict explicit at file time, so the fixer chooses between named trade-offs instead of accumulating unnamed ones.

The most valuable proposal in the set is one filed against its author's own interest — the technical persona arguing for something that helps the nervous beginner at his own expense. Say out loud that this is allowed, or you will not get it.

2. Personas carry taste, not just goals

A goal makes an agent a tester. A taste field makes it a critic. Name real references — "Stripe docs, Linear, the Wizard of Odds appendices; likes density when it is earned; distrusts anything that looks designed to convert" — because a persona that can point at a real artifact produces usable proposals, and one that cannot produces "make it more modern."

3. Every persona has a stated reason to exist

why_this_persona_exists is one line on what this panelist catches that no other one does. It has two jobs: it stops the roster drifting into variations of the same agreeable person, and it tells you when to delete one. A persona whose stated job is already covered is costing you a run slot.

Two sessions, one branch

The panel and the fixer are different sessions, and this is not ceremony. A session that can fix what it finds starts justifying instead of reporting — it reads the source, decides the thing is fine, and files nothing. A session that has just built something cannot evaluate it blind.

Run the panel against the deployed URL, not a dev server. Half the real findings in the first run existed only in production: a CORS origin still pointing at the old domain, a sign-in returning a bare 500 because an API key was unset, a cached DNS failure. None of those exist on localhost.

The four files

Everything lives in a panel/ directory. Three are authored once and edited rarely; the fourth is append-only state.

4.1  personas.json

Six to ten people. Fewer than six and you are testing one taste; more than ten and most never run.

{
  "version": 1,
  "site": "https://example.com",
  "personas": [{
    "id": "dev",
    "name": "Dev, 31, backend engineer",
    "account": "signed-out",
    "situation": "Denver, ~$180k, reads HN every morning. Vegas four times a
                  year with college friends, $25 tables. Read the strategy
                  page twice and never drilled it.",
    "goal": "Decide in ninety seconds whether this was built by someone
             competent or by a growth team.",
    "surfaces": ["/", "/train", "/table"],
    "taste": "Stripe docs, Linear, the Wizard of Odds appendices. Likes
              density when it is earned. Distrusts anything designed
              to convert.",
    "verify": [
      "Does the hero prose agree with the panel beside it?",
      "Change the payout to 6:5. Do chips, headline, verdict and chart
       all move together?",
      "Play twenty hands. Does correct + missed equal 20?"
    ],
    "leaves_if": "A single number contradicts another number on the
                  same screen.",
    "shares_if": "It is boring the way good tools are boring, and one
                  number surprises him.",
    "why_this_persona_exists": "The only panelist who will open the
                                engine and check the arithmetic.",
    "runs": 0,
    "last_used": null
  }],
  "account_fixtures": { "signed-out": null, "free": "...", "paid": "..." }
}

Field notes that matter:

• verify is per-persona and concrete — the cross-surface checks only this person would think to run. Most wrong findings come from here.
• account maps to a fixture. Two personas per run share fixtures, so parallel runs corrupt each other. Widen coverage by adding accounts, not by adding agents.
• leaves_if / shares_if are the persona's two thresholds. They turn a vague reaction into a binary the run can report.
• runs and last_used drive selection, and are the only fields the panel writes back.

4.2  invariants.md

The short list of things the panel does not get to vote on. Everything else — colour, type, layout, copy, flow, what gets built, what gets cut — is theirs, and when a persona says a section is cold, that wins by default.

Each entry has three parts: the rule, why it is not a vote, and what the panel may still do. The third is the one people skip, and the one that keeps the list from becoming a wall. An invariant almost never forbids an entire need — it forbids one expression of it. "Every figure carries its rule set" forbids removing the placard; it permits any redesign of the placard whatsoever.

A refusal is recorded like any other verdict — it names the invariant, quotes the reason, and offers the nearest available thing:

Refused under invariant 1. Kyle asked for "what could I realistically make in a weekend." Not available at any wording. Nearest available: make the prize at true count +2 more prominent, which is what he is actually reaching for. Filed as proposal P-041.

A persona who keeps asking for the same refused thing is itself a finding. It means the legitimate need behind the request is not being met anywhere, and the ledger should say so.

4.3  panel-prompt.md

The run instructions — §5 is a generic version of this file. It is written at the panel session in the second person, and it opens by saying what that session is not: not a developer on this project, may not read source to decide whether something is good, may not fix anything.

4.4  ledger.json, and where it lives

Append-only state: findings, proposals, landed verdicts, wishes, settled questions, per-persona counters, and last_shipped.

Keep it on an orphan branch — qa-state — not in the working tree. It changes every run, it is generated, and in the tree it means every panel run dirties the working branch and every merge conflicts in a file nobody reads.

git checkout --orphan qa-state
git rm -rf . && mkdir panel
echo '{"runs":[],"settled":[],"last_shipped":[]}' > panel/ledger.json
git add panel/ledger.json && git commit -m "panel state"

Two arrays carry more weight than they look like they do:

• settled — questions already answered. Re-raising one without new evidence is noise, and a loop with no settled list re-litigates the same three arguments forever.
• last_shipped — what the fixer changed since the previous run. The panel must not read this before step 7. Knowing it early contaminates the blind pass, which is the only instrument that can tell you whether the change worked.

The run, step by step

Preamble: check out the state branch, read the ledger, read the invariants, read settled. Note that last_shipped exists and do not open it.

Measuring perception

Run this in the console during the blind pass, before scrolling. Deliberately three numbers — a persona can hold three numbers and react to them.

const vh = innerHeight, h = document.documentElement.scrollHeight;
const words = document.body.innerText.trim().split(/\s+/).length;
({ screens: +(h/vh).toFixed(1), words, readMins: +(words/220).toFixed(1) })

Reference points, so the number means something:

Then the question, asked in character and before any scrolling:

Knowing this page is N screens and M minutes of reading, would you read it, skim it, or leave?

A number alone is not a finding. The finding is the persona's honest answer. Report the measurement every run even when the answer is "I would read all of it" — one of your personas will say that, and for him it will be true.

Budgets

Set explicit thresholds so a breach is mechanical rather than a matter of opinion:

• The landing page is over 9 screens
• Any single section is over 2.5 screens
• More than three consecutive prose sections run with no table, no control, no break

Mobile is a different measurement, not a smaller one

Re-running the same script at a phone viewport is necessary and not sufficient, because two defect classes exist only there. Check both explicitly:

• Horizontal page scroll. documentElement.scrollWidth > innerWidth means something is pushing the page sideways. Note the distinction that matters: a wide table inside its own overflow-x:auto container is correct and should not be filed — walk the parent chain and confirm which one is scrolling before calling it a defect.
• Tap-target size. Anything under 44px (Apple) or 48dp (Android) is a defect on a touch device and invisible on a desktop where every control is hit with a 1px cursor.

// mobile-only defect classes -- run at the smallest supported width
const de = document.documentElement;
({
  horizontalScroll: de.scrollWidth > innerWidth + 1,
  smallTaps: [...document.querySelectorAll('button,a.btn,[role=radio],[role=button]')]
    .map(el => ({ txt: el.innerText.trim().slice(0,30),
                  h: Math.round(el.getBoundingClientRect().height) }))
    .filter(x => x.h > 0 && x.h < 44),
})

Density, past length

Length was the first thing measured and it was not the real problem. "A lot of information dumped at you at once" survived the page getting shorter, so the measurement was widened. These are worth counting per section rather than per page:

// figures, eyebrow labels, controls, and how long a read-run gets
const S = [...document.querySelectorAll('section, .block')];
S.map((el, i) => ({
  i,
  screens : +(el.offsetHeight / innerHeight).toFixed(1),
  words   : el.innerText.trim().split(/\s+/).length,
  figures : (el.innerText.match(/[-+−]?\d[\d,.]*\s*%?/g) || []).length,
  labels  : el.querySelectorAll('.label').length,
  controls: el.querySelectorAll('button,input,select,[role=radio]').length,
}));

// typographic variance -- run once on the whole page
const seen = new Set();
document.querySelectorAll('*').forEach(e => {
  const s = getComputedStyle(e);
  if (e.innerText && e.innerText.trim())
    seen.add([s.fontFamily.split(',')[0], s.fontSize, s.fontWeight].join(' | '));
});
[...seen].sort();

That second snippet settled an argument. The owner said there were "too many fonts." There were two families the whole time — but twelve sizes, four weights, twenty combinations. Wrong about the cause, right about the symptom, which is the normal case and the reason to measure rather than litigate. It went to two families, eight sizes, three weights, fourteen combinations.

It also caught something no persona would have: a mono face on 139 elements against 96 for the body face. That looked like mono creeping into body copy. It was not — every mono rule was legitimate. The page simply had fifty uppercase eyebrow labels. At fifty, a label stops being navigation and becomes texture, and the fix belonged in the density budget rather than the type scale.

Buckets and the gate

The discipline is at the boundaries. Never promote a confusing for emphasis — the temptation is real, because a proportion breach can feel worse than a genuine defect and there is an obvious way to make it land harder. Doing it corrupts the two buckets that actually gate, and then nothing gates meaningfully. Never soften a real wrong because the run otherwise went well. The test is evidence — can you quote both sides — not how it felt.

mine exists so the loop can be honest about its own failures without contaminating the product signal. A loop with no mine bucket files harness bugs as product bugs, and the fixer spends a day on a race in the test driver.

A dedicated stop-the-run bucket is worth having if your product has one class of failure categorically worse than the rest — a regulated claim, a security leak, a privacy exposure. It changes the run's behaviour: report it alone, do not dilute it with the eleven other things you noticed.

Did the last change land

After the findings are filed and before the commit, the panel reads last_shipped for the first time. For each change the fixer made since the previous run, it answers one question against its own step-2 notes: did I notice this before I knew about it?

This is the loop's only honest measure of whether a change was an improvement, and it works only because the blind pass was recorded before the answer was available. Nearly everything else in agentic QA measures whether a change was made.

Running it on design, not just pages

The same loop works on an artifact that does not exist yet — a logo, a palette, a naming shortlist, an empty-state illustration. Generate several candidates, then put them through the panel. Four changes to the standard run:

1. Blind first, informed second, and record both. Show every candidate with no names and no rationales: what is this, what industry, what does it remind you of, would you trust it. Only then reveal the reasoning and ask what changed. The gap between the two passes is the finding — a mark that only works once explained does not work.
2. Add adversarial stress lenses that are not people. They read the artifact's actual data rather than its story: does it survive at 16px, in one flat colour, under a circular crop, when the font fails to load. In practice these caught more than the personas did.
3. Never average. The reconciler weights segments by how the business works — who pays, who vetoes, who is the largest headcount — and applies one rule: a candidate one segment loves and another finds embarrassing loses to one everybody finds solid. Shipping two winners "as alternates" is averaging by another name.
4. Check the artifact against its own rationale. The winning logo's write-up insisted three times on "the dead space on the right" while its coordinates gave equal margins both sides. Same class of defect as a hard-coded number in the UI, and worth catching the same way.

A worked run

Run 1 on the blackjack project, to show what the output looks like at each severity.

And two proposals, to show the form — including one filed against its author's interest:

That last cost was the load-bearing part. This project had already shipped a marketing page duplicating the app's numbers and then contradicting them — the hero said 0.41% while the panel beside it said 1.99%. The proposal names the failure mode it would reintroduce and constrains the implementation to avoid it. That is what costs is for.

Pointing it at a new project

The loop is portable. The contents of the four files are not — copying another project's personas or invariants is the main way this gets implemented badly.

Deriving personas

1. List who actually shows up, including the ones you would rather not sell to. The panel is worthless if it is eight versions of your ideal customer.
2. For each, write why_this_persona_exists first. If you cannot say what they catch that nobody else catches, do not add them.
3. Include one veto — someone burned by the category, looking for a reason to leave. The only one who will catch what reads as dishonest.
4. Include one who will not read. Most of your traffic is this person, and no articulate persona represents them.
5. Give every one a taste with real references, and a verify list only they would think to run.

Deriving invariants

Ask what would remain true even if every user asked for the opposite. Usually: legal or regulatory exposure, arithmetic or data integrity, a semantic system that only works if it means one thing, and the honest bad news that makes the rest credible. Write the "what the panel may still do" clause for every entry, and stop at five.

Bootstrapping

1. Deploy something real at a stable URL.
2. Write the four files. Invariants first, because they constrain the persona briefs; personas last.
3. Create the orphan state branch.
4. Run once with two personas. Expect the first run to be mostly confusing and mostly proportion. That is normal, not a sign the loop is weak.
5. Fix, record last_shipped, run again with one new persona and one veteran.
6. From run 3, step 7 starts producing signal. Before that there is no baseline to notice against.

What it is bad at

This panel is not real users. It is a cheap prior — articulate, patient, and systematically wrong in ways real traffic is not. It over-weights reasoning and under-weights the first two seconds. When a proposal only makes sense if the reader is paying close attention, flag it, because most will not be.

It cannot tell you what converts. It tells you what is broken, what is contradictory, what is unreadable, and what a thoughtful person would want. Those correlate with conversion loosely, and are not a substitute for traffic.

It is weak on aesthetics in the small. It catches "this panel looks cheap" and misses four pixels of misalignment. Perception measurement helps at the page scale and does nothing at the component scale.

It drifts agreeable. Personas asked to react to a thing tend to like the thing. leaves_if, the veto persona and the mandatory costs field are all counterweights, and they are partial. If three consecutive runs produce no confusing findings, suspect the loop before congratulating the product.

An inconclusive result is a result. A run that quietly skips a check and reports success is worse than one that reports nothing. If something could not be verified, name the thing and say why.