I’m partnering with SmartBear on a sponsored series of content about tools for the AI era. As always, all opinions are entirely my own.
The last time I wrote about this I had pointed BearQ at my live website, let it build a bunch of tests on the course catalogue page, and showed you what it took to dismiss a finding it got wrong. Ninety seconds, as it turned out, and I argued that the price of dismissing the wrong answers is a better way to judge a tool than the cleverness of the right ones.
Then I redesigned my homepage, and several of those tests stopped passing, as you would expect.
Nothing was broken. The site was fine, better than it had been. The tests were checking widgets that were no longer where they used to be, because I had removed them.
The thing that actually kills test suites
Many arguments about test automation are conducted as though the hard part is detection. Will it find the bug, how many false positives, what is the hit rate. I have sat in a lot of those conversations and I have never once seen a suite abandoned because it failed to find bugs. They get abandoned because they cost too much to keep true.
The mechanism is always the same and it is dull enough that not a lot is written about it. Somebody ships an intentional change and fifteen tests go red. Each red test is now a question rather than a result: is this a defect we introduced, or is this a test describing a version of the product that no longer exists. Somebody has to answer that one failure at a time, and the answering is not automatable, because it needs institutional knowledge no test suite has.
Do that on a Friday afternoon with a release waiting and you learn quickly that there is a cheaper option, which is to stop looking. The suite does not get deleted. It gets ignored, then it gets marked as flaky, and about two quarters later somebody proposes rewriting it from scratch.
The same scarce resource, one step over
In the last piece I ended up somewhere I had not planned to go. When a tool can look at your application without being told what to look at, finding candidate problems stops being the constrained resource, instead the decision making, and executing, is the new bottleneck.
Maintenance is the same problem. A failing test after a redesign is a claim: this is wrong. It happens to be a claim the tool is entitled to make, because as far as it knows the button really has vanished. The expensive part is adjusting to the change rather than detecting the test case.
Which means the useful question about a testing tool is the same one both times, and it is about cost rather than cleverness: what does it cost you when it is right about something you no longer care about.
What did I do about it on camera?
I dealt with the failed tests in the video, even though watching yourself discover that your own tests are now wrong is maybe not a great look for somebody making a video about testing tools. But I think it’s worth seeing, and honestly it went really well.
Two other things from the session are worth naming.
The first is that the agents run in the background. I could set work going and carry on doing something else rather than sitting and watching a spinner, or the cutesy scroll of action verbs that seems to be the (slightly annoying) LLM version of the spinner. A redesign generates a pile of re-runs all at once, and the cost of that pile depends heavily on whether it blocks you for an hour or happens while you get on with the next thing.
The second is that I misunderstood what the application environment is for. The fix was straightforward once I understood it, and I left that section in rather than reshooting it, so anyone about to make the same assumption can avoid it.
Where this leaves the argument
The through-line in this series is that AI moved the slow part of shipping software from writing code to confirming it works, and that the confirmation has to happen against the running application rather than the source, which is what SmartBear calls application integrity, which they define more broadly as continuous, measurable assurance that software works as intended, with the governance to operate at AI speed and scale. Two videos in, the part I keep arriving at is downstream of all of it.
Once a tool can examine a running application on its own, the volume of things it can tell you stops being the constraint. What it costs a person to respond to each of those things becomes the new constraint, both when the tool is right, and when the tool gives you questions rather than answers. Both are judgement work, and both need to be done by somebody who already has a job.
How cheap a product makes those two moments is a design decision rather than a model capability, and BearQ did really well for both.