Additional thoughts on session three of the AI Evals and Analytics Playbook1. The first part is here2. Every major shift in how we build and ship software has been followed by a wave of tooling that automates the tractable part and leaves the actual problem to the practitioner. Agile gave us story pointing ceremonies and […]
The Hard Part of AI Evals Isn’t the Tooling Read More »
Someone posted a thinking piece in a Slack channel I’m in last week, long and earnest and well-structured, arguing that quality engineering needs to evolve for the age of agentic AI, that we need to stop thinking about quality as testing and start treating it as a systemic property. I read it and felt a
The Verification Trap Read More »
Session two closed with a question I couldn’t answer.1 When a product scores 78% on a given metric, what tells you whether that’s good enough to ship? I flagged it as something session three would probably address, and it did, though not in the way I expected. The question can’t be meaningfully answered until you’ve
The anatomy of a metric Read More »
Session one left me with two things I hadn’t resolved.1 The first was a line the instructor said almost in passing: “the hard part is scalability, not automation.” I wrote it down because it piqued something, but I couldn’t quite work out what problem it was pointing at. The second was a question I kept
The AI evals field chose a flawed tool and stuck with it Read More »
If you’ve been following me on Linkedin, or indeed have been reading my blog for a while, you’ll know that I’m interested in AI applications. So it’s probably not coming as a total surprise to you that my main learning goals this year evolve around AI evals, “which are systematic tests and measurements to that
What I don’t understand about AI evals (yet) Read More »
You know the feature inside out. You have tested it, broken it, rebuilt the test suite around it twice. So you ask the AI to generate a few test cases. Save yourself twenty minutes. What comes back looks reasonable. The structure is right. The naming conventions are close enough. Then you start reading. The first
You were promised speed. You got a new job instead. Read More »
I’ve been seeing a lot of posts on LinkedIn recently that make the same claim: speed of writing code was never the bottleneck in software engineering. The real bottleneck was always thinking, choosing the right thing to build, understanding the problem. AI has just made that painfully obvious. I partly agree, but the framing is
Why the bottleneck was never just about thinking Read More »
