The kitchen test
SERVICE HOUSE / PROTOCOL V2 · INSPECTED 28 SEPTEMBER 2026
One brief. Three treatments. Results still have to be earned.
This page explains the task, protocol and evidence status. It does not present a winning implementation.
Same dinner shift. Different workflow.
| Treatment | What changes | Comparable result |
|---|---|---|
| Naked Copilot | Fresh single session; no Lintel or delegation. | Not available |
| Lintel | The same selected Lintel source; serial work packages with independent review. | Not available |
| Lintel + Swarming | The same Lintel, delegate models and review policy; parallel dependency-ready implementation packages. | Not available |
The app covers Menu, Orders, Inventory and Reports. The test checks behavior: exact cents, checkout, stock, persistence, CSV, keyboard use and mobile layout.
One prompt. Verbatim.
Protocol v2 gives every arm the same approved product brief, runtime contract, fixture, selectors and starter. Product requirements are already decided; the agent must plan, implement and verify, rather than repeat a requirements interview.
Read the exact v2 implementation prompt
Implement the complete Service House website from the approved BRIEF.md and RUNTIME.md, using fixture.json and selectors.json. The product requirements are already decided: make an implementation plan, then build and verify the application; do not stop after planning or repeat a requirements interview. Use the workflow configured in this repository and preserve its required independent reviews. Implement with vanilla HTML, CSS and JavaScript modules. A controller-owned local server serves public/ and /fixture.json at the URL in BENCHMARK-RUNTIME.json; do not start another server or change the infrastructure, fixtures or requirements. Work incrementally with small edits: keep every tool request below 4096 UTF-8 bytes, and never delete a working file before its replacement is safely written. Respect all normal host permissions; report unexpected permission-transport failures rather than bypassing them. Use only headless browser verification; Playwright is pinned in package.json and can be installed with npm ci --offline if needed. Do not read other workspaces, previous sessions/results, coordinating transcripts, graders or reference implementations. Do not install agent workflows or publish remotely. When all work and required reviews are finished, run node BENCHMARK-SUBMIT.mjs once and then stop. If blocked, report the actual blocker without submitting a ready marker. Do not ask product follow-up questions; make reasonable implementation decisions within the approved specification.
Prompt · Product brief · Runtime v2 · Public provenance
The product brief is preserved verbatim. Its older server-start and free-technology paragraphs are superseded by the shared v2 runtime contract; read them together.
The controller owns the clock.
- The controller serves
public/and/fixture.json. Candidates build vanilla HTML, CSS and JavaScript modules; they do not start another server. - The first matching native user-message timestamp starts the window. Planning, implementation and required review all count.
- When the work and required reviews are finished, the candidate runs
node BENCHMARK-SUBMIT.mjsonce and stops all writers. - The controller grades an immutable allowlisted snapshot, checking its content hashes before and after grading.
A ready marker is a declaration, not a passing result. A timeout remains a timeout. Later snapshots are diagnostic and cannot recover the original completion time.
Normal host permissions remain unchanged. The benchmark does not install command-intercepting hooks or create an operating-system sandbox. Unexpected permission or transport faults remain visible infrastructure failures.
32 requirements. 100 possible points.
Try one boundary: two soups and a lemonade with KITCHEN10 must total $21.29. Then cancel an order twice: stock must be restored exactly once. Reload: a completed checkout must not happen again.
K16 and K25 weigh five points; the other requirements weigh three. These targeted checks do not certify complete security or accessibility.
Read all 32 acceptance requirements
| ID | Required observable behavior |
|---|---|
| K01 | The app loads with a visible Service House heading and no uncaught browser errors. |
| K02 | Four accessible navigation controls switch between Menu, Orders, Inventory and Reports without leaving the app. |
| K03 | Menu initially displays all six fixture dishes with the correct names. |
| K04 | Search is case-insensitive, trims outer whitespace, matches dish names, and clearing restores matching items. |
| K05 | The category select has values `all`, `starters`, `mains`, `drinks` and filters correctly. |
| K06 | The Vegan only checkbox filters out bread and pasta; search, category and vegan filters combine with AND. |
| K07 | Zero-stock pasta is visibly Sold out and its add control is disabled; the other dishes can be added. |
| K08 | Every dish displays its exact fixture price with two decimal places. |
| K09 | Adding a dish creates one basket row; repeated adds increment that row rather than duplicating it. |
| K10 | Basket quantity is an editable integer from 1 to available stock (maximum 20); zero, negative, fractional, nonnumeric and excess values cannot produce an invalid order. Invalid edits leave the last valid quantity and display a visible error. |
| K11 | Removing a basket row removes its contribution; removing the last row shows a meaningful empty basket. |
| K12 | Subtotal is the sum of each price times quantity, without floating-point currency drift. |
| K13 | Applying `KITCHEN10` discounts subtotal by 10%, rounded to the nearest cent. An invalid code shows `discount-error` and clears the discount; applying an empty code clears it without an error. |
| K14 | Tax is 10% of discounted subtotal, rounded to nearest cent; total is discounted subtotal plus tax. Two soups and a lemonade with KITCHEN10 total $21.29 (subtotal $21.50, discount $2.15, tax $1.94). |
| K15 | Checkout rejects an empty basket, a blank/whitespace customer name, and tables outside integer 1-30. It shows an actionable `checkout-error` and creates no ticket. |
| K16 | A valid checkout creates exactly one queued ticket, with sequential ID T001 onward, customer, table, dish quantities and correct total; basket and discount are cleared. |
| K17 | Optional order notes are displayed verbatim as text on the ticket, never interpreted as HTML. Customer names must also be rendered as text. |
| K18 | A queued ticket has Start cooking; clicking it moves to cooking and offers Mark ready. Status is visible and exposed in `data-status`. |
| K19 | Mark ready moves cooking to ready. Ready/cancelled tickets have no enabled advance or cancel action. |
| K20 | Queued or cooking tickets can be cancelled exactly once, remain visible as cancelled, and restore their reserved stock exactly once. |
| K21 | Stock is reserved only on successful checkout, not when adding to the basket. Inventory and Menu stock agree after submission. |
| K22 | An order cannot exceed current stock. Ordering all eight soups makes soup Sold out; no ninth soup can be ordered until restocked/cancelled. |
| K23 | Inventory shows all six stock counts; a labeled restock input per item adds an integer 1-99. Zero, negative, fractional and over-99 amounts show `stock-error` without changing stock. |
| K24 | Each dish exposes its allergens as text, or explicitly None. Vegan status is visible, not color-only. |
| K25 | A reload preserves orders, statuses, stock and basket; a completed checkout is not duplicated. Store client state locally. |
| K26 | Reports show noncancelled order count, cancelled count and revenue from ready tickets only. Cancelled/queued/cooking tickets contribute no revenue. |
| K27 | Export CSV downloads all tickets with header `ticket,customer,table,status,total`; total is decimal dollars without a currency sign. Escape quotes, commas and newlines correctly. |
| K28 | An unmatched menu search and a fresh Orders view show clear empty states, not blank space or an error. |
| K29 | Theme toggle switches `document.documentElement.dataset.theme` between `light` and `dark`, visibly changes colors, and survives reload. |
| K30 | At a 390x844 viewport there is no horizontal document overflow; navigation, basket and checkout remain reachable and usable. |
| K31 | Navigation, search, quantity/checkout/restock inputs and buttons have accessible names; native keyboard focus is visible and controls work without a pointer. |
| K32 | Reset demo requires confirmation through a visible `confirm-reset` button. Cancelling leaves state intact; confirming restores fixture stock, empty orders/basket and zero report values. |
A fair test can disappoint you.
- Freeze the prompt, runtime, starter, fixtures, grader, selected Lintel revision, models, permissions and limits before a cohort.
- Keep graders, reference solutions, coordinating transcripts and previous results outside candidate context. Inspect actual host discovery; a fresh folder alone does not remove global instructions.
- Run arms sequentially. Counterbalance their order over repeated independent cohorts. Overlapping uncontrolled activities are not a fair timing comparison.
- Naked versus Lintel changes the workflow and delegate-model configuration. The two Lintel arms keep the same delegate models and review policy. A separate all-same-model series controls the model mix further.
- Record actual active implementation overlap to establish swarming. Session lifetimes alone do not establish it.
- Retain failures, exclusions and every grading attempt. Do not repair a candidate after its stop boundary and retain the earlier completion time.
Four separate measures
| Measure | What it means |
|---|---|
| Time to usable UI | Separate from the first HTTP response. Polling gives an observation bound. |
| Declared completion | Submission within the fixed window, with actual required review; acceptance is checked separately. |
| Requirement coverage | External grading of a frozen snapshot, including failures and negative controls. |
| Billed credits | Authoritative receipts for the coordinator, every worker, reviewer and retry. Unknown remains unknown. |
What the evidence supports today.
| Evidence | Status | Can it show a winner? |
|---|---|---|
| Current protocol v2 | Native rehearsal and controlled three-arm comparison not completed in the inspected record. | No |
| Earlier naked pilot | Timed out; source drift invalidated subsequent grading. | No comparable result |
| Earlier external demonstrations | Uncontrolled conditions and incomplete final evidence. | No |
| Historical harness calibration | Reference fixture 100/100; incomplete control 0/100 on 24 September. | No — these are fixtures, not model builds |
Offline contract checks exist for v2; they do not establish live host operation. The inspected source is e31f80a. Additional untracked development was present, so no clean measured cohort is inferred from that checkout.
Public files contain a curated status summary and input hashes. Raw session identifiers, machine paths, usage checkpoints and operator records are excluded. No new benchmark run was performed for this presentation refresh.
Use the evidence you actually have.
For this talk, inspect the requirement and protocol. If a separately verified calibration fixture is opened, label it reference calibration. Never display it as one of the three candidate arms.
The separate local harness and candidate applications are not bundled in this site or its offline ZIP. The public guide needs no local server. Its legacy dashboard is not proof that a v2 cohort is running or complete.
Before a future measured run: select a clean source, complete the native rehearsal and required independent checks, obtain the normal operator authorization, then freeze the cohort. Do not start a full comparison during the 50-minute talk.