A few years into my QA career, my manager asked if I had any tips for handling support tickets efficiently. I said of course, and put together a wiki page (this was before Confluence removed wiki from companies, anyways) describing exactly how I had my desktop set up. Most people never followed it. Some said […]
Inconsistent AI results come from a lack of shared understanding Read More »
This is part of a series where I write about what I’m learning in an AI evals and analytics course. Earlier posts cover the basics of evals, sniff tests, and quantitative evaluation with LLM-as-a-judge. In traditional testing, the oracle is trustworthy. You know what the right answer is, or at least you know who decides.
The oracle has opinions Read More »
Additional thoughts on session three of the AI Evals and Analytics Playbook1. The first part is here2. Every major shift in how we build and ship software has been followed by a wave of tooling that automates the tractable part and leaves the actual problem to the practitioner. Agile gave us story pointing ceremonies and
The Hard Part of AI Evals Isn’t the Tooling Read More »
Someone posted a thinking piece in a Slack channel I’m in last week, long and earnest and well-structured, arguing that quality engineering needs to evolve for the age of agentic AI, that we need to stop thinking about quality as testing and start treating it as a systemic property. I read it and felt a
The Verification Trap Read More »
Session two closed with a question I couldn’t answer.1 When a product scores 78% on a given metric, what tells you whether that’s good enough to ship? I flagged it as something session three would probably address, and it did, though not in the way I expected. The question can’t be meaningfully answered until you’ve
The anatomy of a metric Read More »
Session one left me with two things I hadn’t resolved.1 The first was a line the instructor said almost in passing: “the hard part is scalability, not automation.” I wrote it down because it piqued something, but I couldn’t quite work out what problem it was pointing at. The second was a question I kept
The AI evals field chose a flawed tool and stuck with it Read More »
If you’ve been following me on Linkedin, or indeed have been reading my blog for a while, you’ll know that I’m interested in AI applications. So it’s probably not coming as a total surprise to you that my main learning goals this year evolve around AI evals, “which are systematic tests and measurements to that
What I don’t understand about AI evals (yet) Read More »
You know the feature inside out. You have tested it, broken it, rebuilt the test suite around it twice. So you ask the AI to generate a few test cases. Save yourself twenty minutes. What comes back looks reasonable. The structure is right. The naming conventions are close enough. Then you start reading. The first
You were promised speed. You got a new job instead. Read More »
I’ve been seeing a lot of posts on LinkedIn recently that make the same claim: speed of writing code was never the bottleneck in software engineering. The real bottleneck was always thinking, choosing the right thing to build, understanding the problem. AI has just made that painfully obvious. I partly agree, but the framing is
Why the bottleneck was never just about thinking Read More »
Every time the topic of using AI for code reviews comes up, someone will eventually say that it is like letting the AI mark its own homework. It sounds clever. It also sounds responsible. Underneath it sits a familiar discomfort about oversight, trust, and professional judgement. As an analogy, it points attention away from the
Review as analysis, not authority, when working with AI Read More »
