Ship a customer AI pilot behind an eval gate

Tool stack

Steps

1. Write a one-page charter before any code: the team whose daily work changes, the sponsor who can unblock access, today's number for one outcome, the decisions you can make alone, an out-of-scope list, and the name of whoever runs it after you leave. A blank handoff owner on day one predicts a dead pilot.
2. Collect 30-50 real inputs from the customer's workflow (tickets, documents, records) with the expected output for each. Freeze this as the eval set in LangSmith before building. Include the ugly cases: missing fields, conflicting sources, records the user should not be able to see.
3. Build one thin vertical slice with Claude Code: real data in, one model step, a human review screen, and a write back into the system the team already uses. Skip the dashboard. A slice that writes nowhere is a demo.
4. Run the eval set on every change. Gate expansion on the failures that matter to the customer (wrong account, leaked permission, silent no-op write), not on the average score. A 94% pass rate can hide the one failure that ends the project.
5. Route production traffic through Helicone and report cost per successful task, including retries and human corrections. Lower token cost can raise cost per success if a cheaper model needs more review.
6. Pilot with 3-5 named users for two weeks. Measure adoption weekly: how many runs, how many outputs accepted unedited, how many minutes of review per output, and whether the charter's outcome number moved.
7. Write the handoff: runbook, eval set, known failure modes, owner, and the one constraint that repeated across customers. Send that constraint to product; it is the part worth productizing.

Prompts

You are a forward deployed engineer starting an AI pilot. From the discovery notes, call transcripts, and SOW below, write a one-page pilot charter.

## OUTCOME
One measurable outcome, the team whose work it changes, and today's baseline number. If no baseline exists, say how to collect it this week.

## PEOPLE
Sponsor (unblocks access and scope). Daily users (3-5 names). Handoff owner after the pilot. Mark any blank as RISK.

## DECISION RIGHTS
What the FDE can change without approval (schemas written to, prompts, pilot users) and what needs sign-off.

## THIN SLICE
Input source -> model step -> human review -> system of record write. One path only.

## EVAL GATE
The 3 failures that would end the project if they reached a user. These become blocking eval cases.

## OUT OF SCOPE
At least five requests heard during discovery that this pilot will not do.

Quote the notes for every claim about the customer. Do not invent numbers. Max 500 words.

Notes:
{NOTES}

Given the pilot charter and these 40 real workflow inputs with expected outputs, propose eval cases.

For each case output: input id, what must be true of the output, severity (blocking / major / minor), and which charter failure it guards.

Prioritize: permission boundaries (user sees only records they are allowed to), wrong-entity matches, partial or duplicate writes, and inputs with missing or conflicting fields. Mark any case you had to imagine rather than take from the inputs as SYNTHETIC.

Charter:
{CHARTER}

Inputs:
{INPUTS}

Pitfalls

• Accepting every request in week one to show momentum. Three half-built tools leave you unable to say which one mattered. Point at the out-of-scope list.
• Changing the eval set after seeing results. Add new cases to a separate set; keep the frozen one comparable across versions.
• Treating a timeout as a failed write. If the downstream system may have committed, reconcile before retrying or you will create duplicates.
• Declaring success on technical quality alone. A pilot can pass every eval and still be abandoned because it added a step to someone's day.