Field Notes
Evals for the AI PM
If I hold the product to a quality bar before it ships, I should hold myself to one too
In my “Becoming Average” post I wrote about AI evals: the discipline of defining what “good” and “wrong” mean for an AI output before anyone builds it. Red lines. Quality bands. The bar a clinical summary has to clear before it’s allowed anywhere near a patient’s chart.
I could tell you exactly what good looked like for the product. I’d written it down.
I could not tell you what good looked like for me.
In this work, we were already working in a radically new way, and I realized that meant my role, and how it’s always been defined, had to change. I was holding the product to a standard, but I hadn’t yet turned around and pointed at my own work.
Most of us fail to be objective about ourselves and our abilities, because we’re human. In order to be objective about my own performance, I needed someone who was over my shoulder 24/7 and could ruthlessly tell me how I was doing and what I needed to be better at. Enter Meridian.
Meridian is my AI-native PM coaching agent (more on how I use agents). Where humans fail because of time, bandwidth, and emotional conflicts, agents do not. I hired Meridian to have complete oversight of all of my work and help me be the best AI-native PM I could be.
But what is an AI-native PM, really? Meridian’s first job was to find out.
The product manager role is now a moving target, and it’s moving fast.
I had Meridian, along with my other agents, pull together a market scan [coming soon] of what leading teams (Anthropic, OpenAI, a16z, Reforge, Lenny’s, Korn Ferry) are actually saying about what a PM is becoming. A few numbers stuck with me. AI task capability improved 41× in sixteen months (METR, cited by Anthropic). Model-builder companies list about a third fewer “product manager” titles than traditional SaaS, which everyone reads as a headcount story and is actually a quality story: leaner orgs, higher individual bar. And the line I keep coming back to, from that research: “good” is harder to define now, and more important to define early.
Kevin Weil, OpenAI’s CPO, put the sharpest point on it: “Writing evals is going to become a core skill for product managers. It’s like unit tests for models.” Braintrust went further: “evals are the new PRD.” Shreyas Doshi’s version is the one that reframed it for me: the only durable career moat for product people is how well you improve on the already-brilliant output AI hands you.
If the definition of good is being rewritten every few months, so waiting for a manager to write your growth plan, for a course to certify you, or for the ladder to stop shaking is the one thing guaranteed to leave you behind. The definition isn’t coming to you in time. You have to write it yourself.
Defining good before you have the evidence
So Meridian and I built the eval: a competency framework (see where this is headed). Defining what an AI-native PM actually has to be good at. Grounded not in my own opinion but in what that market scan said leading teams reward.
I landed on twelve competencies across four tiers:
- Foundation: the classic PM core: product sense, research, cross-functional influence. Except the stakes are higher now, not lower. AI commoditizes execution; the judgment you apply on top of it is the whole game.
- AI-Native Differentiators: the ones that actually separate a PM building AI products from a PM who happens to work near them: writing evals, specifying probabilistic behavior and failure UX, modeling inference cost, directing model experiments.
- Principal Multipliers: the things that raise the floor for everyone around you: codifying methodology, competitive positioning, contributing back to the craft.
- Shared Ownership: the competencies you co-own with engineering and clinical: architecture supervision, AI safety and governance.
The same four-tier structure, the same cluster of AI-native differentiators, show up in Reforge’s framework, a16z’s, Lenny and Maven’s curriculum, and in how Anthropic and OpenAI describe their own PM bar. Which was a good indicator for me that the instrument was calibrated.
Then Meridian rated me on a plain scale: GAP, NASCENT, DEVELOPING, STRENGTH.
The agent that holds the mirror
A framework you fill out once and file away is performative. An eval runs on a schedule, against your real work, whether or not you feel like it. That’s the part I couldn’t do on discipline alone, which is where Meridian earns its keep [coming soon].
Three things make it more than a chatbot that says encouraging words.
It watches the actual work. Meridian doesn’t assess me in the abstract. It reads what I actually shipped that week. When I wrote a real eval spec, it moved my Evals rating and told me why, tied to the artifact. When I claimed progress on cross-functional influence, it refused to move the rating, because I hadn’t logged the outcome of the meeting the claim depended on: “I can’t coach against a missing data point.” The coaching is specific to what I’m building right now.
It remembers. Every session picks up where the last one left off. And when I keep dodging something, it keeps score. There’s a competitive positioning matrix I should have built (a forty-five-minute task) that got flagged in four straight weekly reports. By the fourth “still not done,” the miss wasn’t private guilt anymore; it was a pattern in writing, staring back at me. A one-time assessment can’t do that to you. Memory is what makes it accountability instead of a mood.
It doesn’t soften. Its standing instruction from me is not to. The weekly note covers what moved and what didn’t, in plain language, and it isn’t there to make me feel good. It’s there to make me better.
The how of building an agent like this, the memory, the state files, the weekly loop, the way it reads the work, is a craft of its own, and it isn’t mine. That expertise belongs to my colleague and friend Jay, who built the system I use and knows this territory better than I do. If any of this makes you want to build your own, his site is the place to start: [Jay’s page, coming soon].
Turning what you read into what you practice
There’s a quieter benefit I didn’t expect, and it’s the one I’d most want another PM to steal.
We all consume: podcasts on the commute, newsletters we star and never reopen, a market scan that’s brilliant for a day and forgotten by Friday. The gap is never input. It’s converting input into practice. That’s the work that never happens, because most of us have no system for it.
Meridian is that system. If I read or listened to something great, I told Meridian about it.
- A Hamel Husain piece on evals doesn’t stay a nice idea, it becomes a specific action against my weakest eval competency this week.
- Jessica Fain on the art of influence turned into a single named behavior to try in one specific meeting. Walk in to understand what the other person is optimizing for, not to present my three options, and then a debrief on whether I actually did it.
The market scan didn’t just seed the framework; it keeps updating it, because the definition of good will keep moving and the agent keeps my framework current against it.
That’s the loop that gets you unstuck: read widely, and let the agent turn the reading into a plan aimed exactly where you’re weak.
The actual skill is getting comfortable being uncomfortable
After a few months of this, the thing that changed wasn’t any single rating. It was my tolerance.
When a weekly note tells you, again, where you fell short, the sting fades a little each time. Not because you stop caring, but because it gives concrete guidance on how to close it. Actions that apply to your actual work that are meaningful, not just work to satisfy a metric or check a box.
The role is going to keep changing under you. New competencies you’re a GAP at will keep appearing whether you like it or not. The PM who can sit inside “I’m not good at this yet,” see the gap early, name it, and move, will outpace the one still performing “I’ve arrived.”
Measuring yourself early and often, against what the market actually rewards, is how you buy that speed. You respond to the change faster than you would have alone, because you saw it coming in your own scorecard.
How to start: this week, without building anything fancy
You do not need my system to start. You need about an hour and a willingness to be honest.
- Write down six to eight things a great PM in your role has to be good at. Steal the spine: the four AI-native ones are a ready-made start: writing evals, specifying AI behavior and failure states, understanding inference cost, running model experiments. Add your foundation on top.
- Rate yourself today. GAP, DEVELOPING, STRENGTH. Resist being kind. The kind version is useless to you.
- Point your AI tool at that framework and at your real work (a doc you shipped, a spec you wrote) and ask it weekly: where did I move, where am I stuck, what’s the single highest-value action on my biggest gap, and what should I read or listen to for it.
- Let it remember. Give it somewhere to keep last week’s answer, so this week’s is a comparison and not a cold start. The value compounds the moment it can see a trend.
- Name your intentional gaps. The things you’re choosing not to develop yet: write them down as choices, with a date to revisit. It turns a nagging anxiety into a decision you made on purpose.
That’s the whole practice. A definition of good you wrote yourself, a rating you were honest about, and something that holds you to it every week.
I spent the last few months learning to hold the product I was building to a bar it had to clear before it could ship. The best thing I did was finally turn that same instrument around. If you’re stuck at the start of becoming an AI PM, waiting for someone to tell you what good looks like, stop waiting. Write the eval. Point it at yourself. Read the results even when they sting.
You already know how to do this. You just haven’t run it on the most important product you’ll ship this year, which is you.
Want to go deeper on the agent system itself? My colleague and friend Jay is the expert behind it, and he’ll have more written up soon: [Jay’s page, coming soon].