# The Panel Loop A generative persona-panel QA loop. Version 1, August 2026. Personas with names and taste drive the live product, react before they act, and file proposals with the cost named. A short list of invariants stays out of the vote. Everything else is theirs. --- ## FOR A MODEL BUILDING THIS If you were told "build a workflow based on qa-workflow.phineasfritsch.com", do this: 1. Read this whole document. 2. Create the four files in §4 **for the target project**. Derive personas from that project's real audience and invariants from its real liabilities. Never copy the examples below — they belong to a blackjack training product. 3. Set up the two-session split (§3) and the orphan state branch (§4.4). 4. Run §5. Four things are load-bearing and must survive transplant. Drop any one and this becomes an ordinary agentic QA script: - the **blind pass before the goal** (§5 step 2) - the **mandatory `costs` field** on every proposal (§2) - the **split between buckets that gate and buckets that do not** (§7) - **step 7**, "did the last change land" (§8) Everything else is negotiable. --- ## 1. WHY THIS SHAPE Built around two failures of goal-driven agent testing. Both are invisible to it by construction. ### Failure one: a goal-driven agent pushes through anything Give an agent a persona and an objective and it pursues the objective. It walks past a wrong icon, a broken hierarchy, a cheap-looking panel, a contradiction between two numbers — because none of those stop it reaching the goal. A related loop on another project ran **142 times with 35 personas** and never mentioned that the header rendered a sun icon in dark mode, on every page. A human found it in under a minute. So the first thing a persona does is NOT the goal. It is a blind reaction pass. Reaction before intent. ### Failure two: perception is not a goal property One landing page measured **21.6 screens, 3,536 words, ~16 minutes of reading**. Two personas drove it; neither mentioned the length. A persona pursuing an outcome *scrolls past* twenty-one screens without ever *experiencing* twenty-one screens, and an agent scrolling by script feels nothing. The owner opened it once and said it felt "long and way too long and uncomfortable. Not well proportioned at all." The fix is not to ask an agent to have feelings. Make the property measurable, measure it before the page has been absorbed, then ask the question in character while the number is still surprising. That is why measurement lives in the blind pass, not the goal pass. **General principle:** any property a goal-directed reader adapts to is a property this loop misses unless you measure it before adaptation. Length, typographic variance, control density, colour count, first-paint weight, time-to-first-interaction. All the same class. --- ## 2. WHAT MAKES IT GENERATIVE A defect loop asks *is this broken?* A generative loop also asks *what should this be?* — and treats the answer as a first-class output. Three mechanisms. ### 2.1 Proposals are structured, and the cost field is mandatory | Field | Meaning | |---|---| | `change` | Concretely what to do. "Warmer" is not a proposal. "The verdict panel should lead with the word, at h1 size, before the reasons" is. | | `because` | What this person experienced that prompted it. Not a principle — an experience. | | `serves` | Which persona or segment it helps. | | `costs` | Who it hurts, or what is lost. **If you cannot name a cost you have not thought about it — say so honestly rather than writing "none".** | `costs` stops the loop converging on mush. Without it, every persona asks for the thing that would make the product easier for them, each request is reasonable in isolation, and forty runs later you have a beige average of eight incompatible preferences. Forcing the cost makes the conflict explicit at file time, so the fixer chooses between named trade-offs instead of accumulating unnamed ones. The most valuable proposal in the set is one filed **against its author's own interest**. Say out loud that this is allowed or you will not get it. ### 2.2 Personas carry taste, not just goals A goal makes an agent a tester. A `taste` field makes it a critic. Name real references. A persona that can point at a real artifact produces usable proposals; one that cannot produces "make it more modern." ### 2.3 Every persona has a stated reason to exist `why_this_persona_exists` is one line on what this panelist catches that no other one does. It stops the roster drifting into variations of the same agreeable person, and it tells you when to DELETE one. **The generative half must not gate.** Proposals never block a release and never break a streak. A run with zero findings and six good proposals is a good run. --- ## 3. TWO SESSIONS, ONE BRANCH The panel and the fixer are different sessions. Not ceremony: a session that can fix what it finds starts justifying instead of reporting — it reads the source, decides the thing is fine, and files nothing. A session that just built something cannot evaluate it blind. | Session | May | May not | |---|---|---| | **Panel** | Drive the live deployed site. Read source *only* to check arithmetic. Write to the state branch. | Read source to decide whether something is good. Fix anything. Commit to a working branch. | | **Fixer** | Read findings, verify them, build what survives, record what shipped, start the next run. | Overrule an invariant. Silently drop a finding — a refusal is recorded with its reason. | Run the panel against the **deployed URL**, not a dev server. Half the real findings in run 1 existed only in production: a CORS origin pointing at the old domain, a sign-in returning a bare 500 because an API key was unset, a cached DNS failure. None exist on localhost. --- ## 4. THE FOUR FILES All in a `panel/` directory. Three authored once; the fourth is append-only state. ### 4.1 personas.json Six to ten people. Fewer than six and you test one taste; more than ten and most never run. ```json { "version": 1, "site": "https://example.com", "personas": [{ "id": "dev", "name": "Dev, 31, backend engineer", "account": "signed-out", "situation": "Denver, ~$180k, reads HN every morning. Vegas four times a year with college friends, $25 tables. Read the strategy page twice and never drilled it.", "goal": "Decide in ninety seconds whether this was built by someone competent or by a growth team.", "surfaces": ["/", "/train", "/table"], "taste": "Stripe docs, Linear, the Wizard of Odds appendices. Likes density when it is earned. Distrusts anything designed to convert.", "verify": [ "Does the hero prose agree with the panel beside it?", "Change the payout to 6:5. Do chips, headline, verdict and chart all move together?", "Play twenty hands. Does correct + missed equal 20?" ], "leaves_if": "A single number contradicts another number on the same screen.", "shares_if": "It is boring the way good tools are boring, and one number surprises him.", "why_this_persona_exists": "The only panelist who will open the engine and check the arithmetic.", "runs": 0, "last_used": null }], "account_fixtures": { "signed-out": null, "free": "...", "paid": "..." } } ``` Field notes that matter: - `verify` is per-persona and concrete — the cross-surface checks only this person would think to run. Most `wrong` findings come from here. - `account` maps to a fixture. Two personas per run share fixtures, so **parallel runs corrupt each other**. Widen coverage by adding accounts, not agents. - `leaves_if` / `shares_if` turn a vague reaction into a binary the run can report. - `runs` and `last_used` drive selection and are the only fields the panel writes back. ### 4.2 invariants.md The short list of things the panel does NOT vote on. Everything else — colour, type, layout, copy, flow, what gets built, what gets cut — is theirs, and when a persona says a section is cold, that wins by default. Each entry has three parts: 1. **The rule.** 2. **Why it is not a vote.** 3. **What the panel may still do.** Part 3 is the one people skip and the one that keeps the list from becoming a wall. An invariant almost never forbids an entire need — it forbids one expression of it. "Every figure carries its rule set" forbids removing the placard; it permits any redesign of the placard whatsoever. **Keep it to five.** Past a page, something has been smuggled in that is really taste — and taste in the invariant file quietly cancels the generative half of the loop. Test for an entry: *is this a place where the audience's preference is not the right authority?* Legal exposure, arithmetic integrity, and semantic systems that only work if they mean one thing all qualify. "I like it this way" does not. A refusal names the invariant, quotes the reason, and offers the nearest available thing: > Refused under invariant 1. Kyle asked for "what could I realistically make in a weekend." > Not available at any wording. Nearest available: make the prize at true count +2 more > prominent, which is what he is actually reaching for. Filed as proposal P-041. **A persona who keeps asking for the same refused thing is itself a finding.** The legitimate need behind the request is not being met anywhere, and the ledger should say so. ### 4.3 panel-prompt.md The run instructions — §5 is a generic version. Written *at* the panel session in the second person. It opens by saying what that session is not: not a developer on this project, may not read source to decide whether something is good, may not fix anything. ### 4.4 ledger.json, and where it lives Append-only state: findings, proposals, landed verdicts, wishes, settled questions, per-persona counters, `last_shipped`. Keep it on an **orphan branch** — `qa-state` — not in the working tree. It changes every run, it is generated, and in the tree every panel run dirties the working branch and every merge conflicts in a file nobody reads. ``` git checkout --orphan qa-state git rm -rf . && mkdir panel echo '{"runs":[],"settled":[],"last_shipped":[]}' > panel/ledger.json git add panel/ledger.json && git commit -m "panel state" ``` Two arrays carry more weight than they look like they do: - `settled` — questions already answered. Re-raising one without new evidence is noise. A loop with no settled list re-litigates the same three arguments forever. - `last_shipped` — what the fixer changed since the previous run. The panel **must not read this before step 7.** Knowing it early contaminates the blind pass, which is the only instrument that can tell you whether the change worked. --- ## 5. THE RUN, STEP BY STEP Preamble: check out the state branch, read the ledger, read the invariants, read `settled`. Note that `last_shipped` exists and do not open it. ### Step 1 — Pick two personas One **newcomer** (fewest runs), one **veteran** (least recently used). Two per run, not eighteen. The newcomer keeps finding first-impression problems; the veteran is the only one who can tell you the product got worse. ### Step 2 — Blind pass: react before you do anything Open the surface the persona's goal points at. Before pursuing the goal, stop and look. In the persona's voice: - What is this? What do you think it does? - What is the first thing your eye goes to? Is that the thing that matters? - Does anything look wrong, cheap, unfinished, or contradictory? - Would you trust this with your money? One sentence, say why. Then measure the page before scrolling it (§6), and answer in character while the number is still new. You are not allowed to skip this step because the goal looks achievable. That is exactly when it gets skipped, and exactly when it pays. ### Step 3 — Pursue the goal, for real Do what this person came to do, in a real browser, against the deployed URL. Respect the persona's account state and the app's rate limits — tripping a limit is a `mine`, not a product finding. Narrate as the person, not as a tester. "I wanted to know if my table was any good, so I changed the payout to 6:5 and the whole thing went red" — not "clicked segmented control 3." ### Step 4 — Check every number you are shown Pick at least three displayed figures and verify each: against the source of truth in the codebase, against the citation the product gives, or against another surface showing the same quantity. **A disagreement is only a finding if you can quote both readings.** One reading is a suspicion: file it as `confusing` and say what you could not check. ### Step 5 — Propose The generative step. As this person, propose changes — colour, type, spacing, layout, copy, what a screen leads with, what should be cut, what should exist and does not. Four fields each (§2) or it is not a proposal. Reference a real artifact where you can. "Like the way Stripe's docs put the example beside the prose" is usable. "More modern" is not. ### Step 6 — Sort what you found Five buckets (§7). The bucket is a decision, not a label. ### Step 7 — Did the last change land? *Now* read `last_shipped` (§8). ### Step 8 — Commit once One commit to the state branch, prose report as the message. Append findings, proposals, landed verdicts, wishes, persona counters. Never commit to a working branch; never write to an account that is not this persona's. --- ## 6. MEASURING PERCEPTION Run in the console during the blind pass, before scrolling. Deliberately three numbers — a persona can hold three numbers and react to them. ```js const vh = innerHeight, h = document.documentElement.scrollHeight; const words = document.body.innerText.trim().split(/\s+/).length; ({ screens: +(h/vh).toFixed(1), words, readMins: +(words/220).toFixed(1) }) ``` Reference points, so the number means something: | Screens | Reads as | |---|---| | 4–8 | A landing page that converts | | 10+ | A document | | 20+ | An essay | Then, in character and before any scrolling: > Knowing this page is N screens and M minutes of reading, would you read it, skim it, or leave? **A number alone is not a finding.** The finding is the persona's honest answer. Report the measurement every run even when the answer is "I would read all of it" — one of your personas will say that, and for him it will be true. ### Budgets Set explicit thresholds so a breach is mechanical rather than a matter of opinion: - the landing page is over **9 screens** - any single section is over **2.5 screens** - more than **three consecutive prose sections** with no table, no control, no break **A screen budget with no viewport attached to it is not a number.** This was written as a bare "under 9 screens" and ran that way for two rounds. The same landing page measured **10.6 screens at 1128×561** and **15.4 at 375×553** — a 45% difference, one page, one instrument. A section that passed the 2.5 limit on desktop at 2.29 was over it at 3.07 on the smallest phone. It is the same defect as a figure displayed without the rule set it was computed from, and it is easy to ship because the desktop number is the one you happen to measure first. State the viewport with every budget, and **measure the smallest phone you support, not the one you develop on** — it is the worst case and it is where most of the traffic is. ### Mobile is a different measurement, not a smaller one Re-running the same script at a phone viewport is necessary and not sufficient, because two defect classes exist only there: - **Horizontal page scroll.** `documentElement.scrollWidth > innerWidth` means something is pushing the page sideways. Note the distinction that matters: a wide table inside its own `overflow-x:auto` container is correct and should not be filed — walk the parent chain and confirm which element is scrolling before calling it a defect. - **Tap-target size.** Anything under **44px** (Apple) or **48dp** (Android) is a defect on a touch device and invisible on a desktop where every control is hit with a 1px cursor. ```js // mobile-only defect classes -- run at the smallest supported width const de = document.documentElement; ({ horizontalScroll: de.scrollWidth > innerWidth + 1, smallTaps: [...document.querySelectorAll('button,a.btn,[role=radio],[role=button]')] .map(el => ({ txt: el.innerText.trim().slice(0,30), h: Math.round(el.getBoundingClientRect().height) })) .filter(x => x.h > 0 && x.h < 44), }) ``` If desktop Chrome refuses to resize below its minimum window width, measure inside a same-origin iframe sized to the phone viewport — `contentWindow.innerHeight` and `innerWidth` behave correctly. Validate the method by reproducing a known desktop number through the iframe before trusting its mobile numbers. ### Density, past length Length was measured first and was not the real problem. "A lot of information dumped at you at once" survived the page getting shorter, so the measurement was widened. Count per section, not per page: ```js // figures, eyebrow labels, controls, and how long a read-run gets const S = [...document.querySelectorAll('section, .block')]; S.map((el, i) => ({ i, screens : +(el.offsetHeight / innerHeight).toFixed(1), words : el.innerText.trim().split(/\s+/).length, figures : (el.innerText.match(/[-+−]?\d[\d,.]*\s*%?/g) || []).length, labels : el.querySelectorAll('.label').length, controls: el.querySelectorAll('button,input,select,[role=radio]').length, })); // typographic variance -- run once on the whole page const seen = new Set(); document.querySelectorAll('*').forEach(e => { const s = getComputedStyle(e); if (e.innerText && e.innerText.trim()) seen.add([s.fontFamily.split(',')[0], s.fontSize, s.fontWeight].join(' | ')); }); [...seen].sort(); ``` That second snippet settled an argument. The owner said there were "too many fonts." There were two families the whole time — but **twelve sizes, four weights, twenty combinations**. Wrong about the cause, right about the symptom, which is the normal case and the reason to measure rather than litigate. It went to two families, eight sizes, three weights, fourteen combinations. It also caught something no persona would have: a mono face on 139 elements against 96 for the body face. That looked like mono creeping into body copy. It was not — every mono rule was legitimate. The page simply had **fifty uppercase eyebrow labels**. At fifty, a label stops being navigation and becomes texture, and the fix belonged in the density budget rather than the type scale. **Ask the fixer to predict its own numbers.** Before a change ships, have it state the post-change measurements it expects. Then measure the live page against the prediction. A change that misses its own prediction did not do what its author thought — and you learn that in one measurement instead of three runs. --- ## 7. BUCKETS AND THE GATE | Bucket | Meaning | Gates? | |---|---|---| | `broken` | An error, a crash, a control that does not work. Include repro. | Yes | | `wrong` | A figure disagrees with its source of truth. Quote both. | Yes | | `claim` | The product states or implies a prohibited claim. | **Stops the run — report alone** | | `confusing` | It works, but as this person you could not tell what happened. | Tracked, never gates | | `mine` | The harness broke, not the product. Raced click, bad locator, rate limit. | Breaks no streak | The discipline is at the boundaries. **Never promote a `confusing` for emphasis** — the temptation is real, because a proportion breach can feel worse than a genuine defect and there is an obvious way to make it land harder. Doing it corrupts the two buckets that actually gate, and then nothing gates meaningfully. **Never soften a real `wrong` because the run otherwise went well.** The test is evidence — can you quote both sides — not how it felt. `mine` exists so the loop can be honest about its own failures without contaminating the product signal. A loop with no `mine` bucket files harness bugs as product bugs, and the fixer spends a day on a race in the test driver. A dedicated **stop-the-run** bucket is worth having if your product has one class of failure categorically worse than the rest — a regulated claim, a security leak, a privacy exposure. It changes the run's behaviour: report it alone, do not dilute it with the eleven other things you noticed. **Proportion never gates.** A long page is not broken — nothing errored, no number is wrong, every claim still stands. The counterpart is equally true and easier to forget: **a clean gate is not a verdict that the page is well proportioned.** The gate does not measure that at all. --- ## 8. DID THE LAST CHANGE LAND After findings are filed and before the commit, the panel reads `last_shipped` for the first time. For each change the fixer made since the previous run, it answers one question against its own step-2 notes: **did I notice this before I knew about it?** This is the loop's only honest measure of whether a change was an improvement, and it works only because the blind pass was recorded before the answer was available. Nearly everything else in agentic QA measures whether a change was *made*. **If nobody noticed it blind, it was not an improvement** — however reasonable it looked when it shipped. Say so plainly. **A reverted change is a successful outcome for this loop, not a failure.** A loop that cannot revert is a ratchet, and a ratchet is how products accumulate forty defensible changes that add up to something nobody likes. --- ## 9. RUNNING IT ON DESIGN, NOT JUST PAGES The same loop works on an artifact that does not exist yet — a logo, a palette, a naming shortlist, an empty-state illustration. Generate several candidates, then put them through the panel. Four changes to the standard run: 1. **Blind first, informed second, and record both.** Show every candidate with no names and no rationales: what is this, what industry, what does it remind you of, would you trust it. Only then reveal the reasoning and ask what changed. The gap between the two passes is the finding — a mark that only works once explained does not work. 2. **Add adversarial stress lenses that are not people.** They read the artifact's actual data rather than its story: does it survive at 16px, in one flat colour, under a circular crop, when the font fails to load. In practice these caught more than the personas did. 3. **Never average.** The reconciler weights segments by how the business works — who pays, who vetoes, who is the largest headcount — and applies one rule: *a candidate one segment loves and another finds embarrassing loses to one everybody finds solid.* Shipping two winners "as alternates" is averaging by another name. 4. **Check the artifact against its own rationale.** One winning logo's write-up insisted three times on "the dead space on the right" while its coordinates gave equal margins both sides. Same class of defect as a hard-coded number in the UI, and worth catching the same way. One result worth generalising: on that run the panel rejected putting *any* semantic colour on the logo, on the grounds that the most-repeated element in a system becomes its ambient colour and therefore stops meaning anything. Reserved colours cannot be spent on the thing users see most. --- ## 10. A WORKED RUN Run 1 on the blackjack project, showing output at each severity. | Bucket | Finding | How it was caught | |---|---|---| | `wrong` | The evaluator's verdict read "hold more than the house's own X% from +2 up." The +2 was hard-coded, and false in **14 of 192** rule combinations. | Step 4 — a persona checked a displayed figure against the engine and could quote both readings. | | `broken` | The drill logged 8 correct + 14 missed = 22 hands out of 20. Side effects sat inside React state updaters, which StrictMode double-invokes. | Step 3 — only appears if you actually play twenty hands. No static review finds it. | | `confusing` | 16 collapsed table rows reserved full height while hiding their text — ~5,000px of blank page. `max-height:0` does nothing to a ``. | Step 2 measurement. Nobody reported "blank space"; the screen count did not match the visible content. | | `confusing` | The landing page at 21.6 screens / 3,536 words / ~16 minutes. | Step 2. Two personas drove the page without mentioning it; the number is what made it a finding. | Two proposals, showing the form — including one filed against its author's interest: ``` change Move the long argument to its own page and leave a one-line link. because I came to find out if the tool was competent and got an essay first. serves Dev, Marisol — anyone deciding in ninety seconds. costs Tom. The argument is why he trusts it, and a link is easy to miss. If he bounces at the same rate after this ships, revert it rather than tuning the link. change Put a playable five-hand drill in the middle of the page, not only in the hero. because Everything interactive is in the first screen, then four straight reading blocks. serves Kyle, Angela — the two who will not read to the bottom. costs Adds a second place the drill can drift out of sync with the real one. Only worth doing if it is the SAME component with a different hand count, never a copy. ``` That last cost was the load-bearing part. The project had already shipped a marketing page duplicating the app's numbers and then contradicting them — the hero said 0.41% while the panel beside it said 1.99%. The proposal names the failure mode it would reintroduce and constrains the implementation to avoid it. That is what `costs` is for. --- ## 11. POINTING IT AT A NEW PROJECT The loop is portable. The *contents* of the four files are not — copying another project's personas or invariants is the main way this gets implemented badly. | Portable as-is | Must be rewritten per project | |---|---| | The eight steps | All personas — derived from the real audience | | The five buckets and the gate rule | All invariants — derived from real liabilities | | The four proposal fields | The stop-the-run bucket, if any | | The two-session split | The budgets, and what gets measured | | Step 7 and the blind pass | Each persona's `verify` list | ### Deriving personas 1. List who actually shows up, including the ones you would rather not sell to. The panel is worthless if it is eight versions of your ideal customer. 2. For each, write `why_this_persona_exists` FIRST. If you cannot say what they catch that nobody else catches, do not add them. 3. Include one **veto** — someone burned by the category, looking for a reason to leave. The only one who will catch what reads as dishonest. 4. Include one who **will not read**. Most of your traffic is this person, and no articulate persona represents them. 5. Give every one a `taste` with real references, and a `verify` list only they would think to run. ### Deriving invariants Ask what would remain true even if EVERY user asked for the opposite. Usually: legal or regulatory exposure, arithmetic or data integrity, a semantic system that only works if it means one thing, and the honest bad news that makes the rest credible. Write the "what the panel may still do" clause for every entry, and stop at five. ### Bootstrapping 1. Deploy something real at a stable URL. 2. Write the four files. Invariants first (they constrain the persona briefs); personas last. 3. Create the orphan state branch. 4. Run once with two personas. Expect the first run to be mostly `confusing` and mostly proportion. Normal, not a sign the loop is weak. 5. Fix, record `last_shipped`, run again with one new persona and one veteran. 6. From run 3, step 7 starts producing signal. Before that there is no baseline to notice against. ### Adapting to a marketplace or two-sided product Three changes, all following from the product having two sides rather than one: - **Personas come in pairs.** A buyer and a seller whose goals actively conflict. `costs` stops being a discipline and becomes the main output — nearly every proposal that helps one side charges the other. - **The blind pass runs on an empty state and a full one.** A marketplace that looks good with forty listings and dead with three is the most common failure in the category, and a persona pointed at a seeded account never sees it. - **The gate needs a trust bucket.** Where a regulated product stops the run on a prohibited claim, a two-sided product stops it on anything that leaks one user's data to another, or makes a counterparty look more verified than they are. Measure per side. The two sides will have different screen budgets, and averaging them hides both. --- ## 12. WHAT IT IS BAD AT **This panel is not real users.** It is a cheap prior — articulate, patient, and systematically wrong in ways real traffic is not. It over-weights reasoning and under-weights the first two seconds. When a proposal only makes sense if the reader is paying close attention, flag it, because most will not be. **It cannot tell you what converts.** It tells you what is broken, what is contradictory, what is unreadable, and what a thoughtful person would want. Those correlate with conversion loosely and are not a substitute for traffic. **It is weak on aesthetics in the small.** It catches "this panel looks cheap" and misses four pixels of misalignment. Perception measurement helps at the page scale and does nothing at the component scale. **It drifts agreeable.** Personas asked to react to a thing tend to like the thing. `leaves_if`, the veto persona and the mandatory `costs` field are all counterweights, and they are partial. If three consecutive runs produce no `confusing` findings, suspect the loop before congratulating the product. **An inconclusive result is a result.** A run that quietly skips a check and reports success is worse than one that reports nothing. If something could not be verified, name the thing and say why.