I’m partnering with SmartBear on a sponsored series of content about tools for the AI era. As always, all opinions are entirely my own.
I have a long-standing suspicion of tools that promise to find problems for me, and it comes from watching what happens to them about three months into the journey. Somebody switches on the static analysis, or the accessibility scanner, or the dependency checker, and for a fortnight people read the output and argue about severity levels, and then the count climbs past what anyone can work through in an afternoon. At which point everybody stops paying attention.
None of those tools failed by finding nothing. They found plenty, but they don’t make it easy for humans to work through and resolve the defects they find. That work is an added cost on top of what people already have to do for work – fixing scripts, updating test automation – so if it gets too onerous people just stop doing it.
I went into recording a demo of BearQ with that history, and the most interesting take away for me was not the bug it caught, but the steps that followed.
Scripted testing vs. autonomous testing
I pointed BearQ at my own site, let it crawl, and it created a test on the course catalogue page. The steps it generated were unremarkable in the way you would want them to be: navigate, hover over the course items, check some attributes. It ran, and the steps passed.
After running the tests it flagged two things not listed in those steps. One was a course heading, which it reported as not clickable. The other was the close button on the login modal, which it reported as sitting outside the viewport where nobody could reach it.
A scripted test fails on what you told it to check. That is the arrangement you expect when you write assertions: you get precision, and you get exactly the coverage you were able to think of in advance. SmartBear’s argument for this class of tool is that it explores the running application the way a person would rather than executing a script you wrote, which means it is not confined to your list.
That is the capability, and it is also the tradeoff. Something that only reports on what you asked about won’t surprise you, and something that can explore beyond your existing test plans has to give you a way to evaluate what it finds.
Noise vs. signal
Before checking the heading myself I used the feature it offers for exactly this situation, which SmartBear calls Verify Issue. It re-ran the check on its own and came back with its reasoning: what it had looked at, and why it had concluded what it did. Then I clicked the heading, because that is the only way to actually know, and it behaved normally. I marked the issue resolved and closed it. Two clicks.
It surfaced something about my application that didn’t hold up when I checked it myself, and that is the single most common thing my readers tell me about AI tooling. When I asked, 65% named output quality as their leading concern, and they described it in language that has nothing to do with whether the tool is clever:
“The amount of noise they generate so you have to sift through to weed out the incorrect outputs”
“we dont know whether the information generated by the system in correct or not”
Neither of those is a complaint about detection. Both are complaints about judgement, about the work that starts after the machine has spoken and somebody has to decide whether to believe it.
The bug that wasn’t
It took me 90 seconds to re-run the test case, observe it live, do a quick manual and close it with two clicks.
Compare that to a wrong answer from the tools I described at the top. A scanner tells you a dependency has a critical vulnerability, and working out whether it applies to your usage means reading the advisory, tracing whether you call the affected path, and then writing down why you are not fixing it somewhere the next person will find. That is an afternoon for one finding you are going to dismiss. Do that fifteen times and you understand why the dashboard stops being opened.
One way to define the value of a tool that finds things is the cost of dismissing the ones it gets wrong. That is a criterion I would use now, on top of the hit rate I see in a vendor demo.
The actual real life bug
The close button on the login modal sat outside the viewport and could not be clicked. I ran Verify Issue on that one too, it confirmed the finding, and it was a genuine bug on my own site that I had not known about.
Nobody would ever have written that assertion. You do not sit down to write a test checking whether a modal’s close button is within the bounds of the screen, because if you had thought of it you would have fixed it. The findings worth having are, almost by definition, the ones nobody thought to look for, which puts them in the same bucket as the false positives and means they are found through the same process.
A new bottleneck
The argument running through this series is that AI moved the slow part of shipping software from writing the code to confirming that the code does what somebody wanted, and that the confirmation has to happen against the running application rather than against the source, which is what SmartBear refers to as application integrity. Meaning the confirmation layer has moved again, one level down. When a tool can look at your application without being told what to look at, finding candidate problems is no longer the constrained resource, and deciding about them is. Every finding is a claim that some person has to accept, reject, or investigate, and that person’s time and attention is the scarce resource in the system.
That cost does not disappear because the tool is autonomous. What matters is how much context, evidence, verification, and control the product gives you to make that judgement quickly.
The demo
The full walkthrough is on my YouTube channel, including both issues being investigated in the interface rather than described after the fact.