How to Build an AI Copilot: A Field-Tested PM Playbook
Your best judgment is trapped in your calendar. If the same two or three questions keep landing on you, from students, clients, or your own product teams, you are the bottleneck, and hiring more of yourself is off the table. I talked with a product leader, ex-big-tech, who now teaches product managers how to interview, and he escaped that trap the way a PM should: he decided to build an AI copilot on Claude Projects that answers those repeat questions the way he would. The build took him roughly two weeks of focused tinkering, a few hundred iterations, and zero engineers. Today that copilot sells at $299 a seat and justified raising his course price from $600 to $1,000.
The playbook below is what I extracted from that conversation. The tool is the least interesting part. The product discipline behind it is the part worth stealing.
Why build an AI copilot instead of custom software?
Because consumer AI workspaces subsidize your tokens at a scale you cannot match. The product leader I sat down with spent four to five months building his own API-based copilot platform with an engineer friend, then killed it once he saw the raw token bill next to the price of a consumer subscription.
The economics get worse as the product gets better. A copilot people love gets heavy use, and heavy use burns tokens. During his own tuning sessions he regularly hit usage limits mid-session and had to wait hours to continue. Later, his students asked how to get around those same limits. He read that correctly: users complaining about usage caps are telling you the tool matters to them.
So start where the subsidies are: Claude Projects, custom GPTs, or Gemini's Gems. Graduate to your own software only when you have proof the workflow earns its token bill. If your plan for an expertise tool starts with an API key and a wrapper, you are paying enterprise prices to learn something a consumer subscription would teach you faster.
How do you start building an AI copilot?
Start with a failing baseline: open a fresh Claude Project with no instructions and no files, ask it one real question from your domain, and watch where it goes wrong. Then add one instruction at a time and re-test, so you always know which sentence changed the output.
That is exactly how he began. He asked the untuned model a classic product-sense interview question, the kind candidates face at big-tech companies, and it jumped straight into solution mode, the same mistake weak PM candidates make. That failing baseline defined his first instruction: frame the context and the audience before proposing anything. He then used the retry button, which re-runs the answer with the project's latest instructions, to watch what each added line did, and opened a fresh chat whenever a thread's accumulated context started polluting the results. A few hundred iterations of this, spread over roughly two weeks, produced the first working version.
What surprised him was which part worked first. Content quality took tuning, but structural adherence came almost immediately: the model set expectations up front, laid out a game plan, and checked in at each step, the exact behavior that separates strong interview candidates from weak ones. Structure was the hard thing he needed the tool to teach, and it turned out to be the thing the model does best.
Every PM preaches measuring the control before shipping the variant. Apply that to prompts. Whoever writes a full page of instructions on day one can never trace an output change back to its cause.
How do you scope an AI copilot so it stays reliable?
Give each copilot one workflow and one evaluation rubric; two conflicting rubrics in the same project degrade both. Copilots work best on repeatable, linear workflows where every step builds on the one before it.
He learned this by getting it wrong. His first version bundled two interview types, product sense and analytical thinking, into a single project. The interviews look similar from a distance, but each is graded against a different rubric, and the model kept blending them. Splitting them into two separate projects fixed it. He called this the modern version of scoping: you are drawing boundaries around the environments where things happen nondeterministically.
Two more scoping calls are worth copying. First, he had the AI play the candidate, not the interviewer, because years of interviewing candidates and running mock interviews had calibrated him to judge candidate performance, and because students learn fastest by watching what a strong answer looks like. Point the copilot at the role you are best equipped to evaluate. Second, he only built a tool because the task recurs. His rule holds for any team: a one-off task deserves an answer; only a repeatable, convergent process deserves a copilot.
How do you validate an AI copilot before charging for it?
Climb a validation ladder before money changes hands: calibrated experts first, real users second, and ship only when the tool clears the name-on-it bar, meaning zero contradictions between what it says and what you teach.
His ladder had three rungs. First, he sent the copilot to former colleagues who had been trained interviewers at his old big-tech employer and asked them to grill it like a real candidate. One of them asked whether he could hire it, saying its answers would have earned a passing score. Second, he gave it free to alumni of his previous course cohorts. They surfaced real defects: shallow segmentation, a skipped analysis step he always insists on, verbose framing. Then came the feedback that mattered most. Some alumni said they were now confused about whose answer to trust, the coach's or the copilot's.
That is the one unacceptable outcome. A copilot that contradicts the expert does worse than nothing; it converts your accumulated trust into doubt. So he went back into the instructions section by section, question by question, until the tool and the teacher agreed everywhere. He shipped only past what I now call the name-on-it bar: if you would hesitate to put your name on an output, it does not go to users. Turning private judgment into a product without burning your credibility is exactly the kind of zero-to-one decision I work through with founders and independent experts in mentoring sessions.
Only after clearing that bar did he touch pricing. The product leader I sat down with sold roughly $10,000 of copilot access in a single month, the month an article he wrote for a major product newsletter went live, at $299 per seat. Across hundreds of copilot sales, the same product leader saw just two refunds, on a platform where instructors typically see 5 to 10 percent refund rates on cohort courses. Premium pricing followed validation. It never led.
The five-step copilot playbook
The whole build reduces to five steps, in order:
- Pick judgment you can grade. Choose expertise where you can instantly tell a strong output from a weak one. Every later step depends on that evaluation ability.
- Map where it fits. Locate the exact moment in the user's workflow the copilot serves: for his students, after learning the material and before mock interviews.
- Baseline, then one instruction at a time. Watch the untuned model fail, then tune with single, attributable changes in clean chats.
- Scope one workflow, one rubric per project. Split anything with conflicting evaluation criteria.
- Codify the rubric and climb the ladder. Writing down what good and bad look like improved his main use case, and the tool now grades its own runs; AI teams call this discipline writing evals. Then validate with experts, then users, up to the name-on-it bar.
Key takeaways
- Consumer AI workspaces subsidize tokens heavily; prove the workflow there before building custom software.
- Start from a failing baseline and add one instruction at a time, in clean chats, so every output change has a cause.
- Scope one workflow and one evaluation rubric per project; split anything that conflicts.
- Validate with calibrated experts, then real users; a tool that contradicts its expert erodes trust in both.
- Ship only past the name-on-it bar, and let pricing follow validation.
The test you can run this week: pick the one question people ask you most, open a blank Claude Project, and ask it that question with no instructions. The gap between its answer and yours is your build list. If you are deciding whether your expertise, or your team's, should become a product, book a mentoring call and we will pressure-test the idea against this playbook, or start with the frameworks in the resources hub.