I’m partnering with SmartBear on a sponsored series of content about tools for the AI era. As always, all opinions are entirely my own.
A while ago I watched a mobile app go from a sales prop to a production system in the space of a single meeting. It had been built as a demo, the kind of thing a consultancy puts in front of a prospect to win a development contract, and it was never meant to run in anger. The client loved it. They loved it so much that they did not want the polished version we were proposing to build next, they wanted this one, the demo, and they wanted their own developers onboarded to take it over and keep it running. The scramble after that meeting was real, because the thing now heading into production had been built to be looked at once and then forgotten, and nobody had written a single test for it.
That story used to be unusual. It is becoming ordinary, because the cost of building something worth showing has fallen through the floor, and the number of demos, experiments and might-not-survive prototypes has gone up to match.
The software nobody budgeted to test
I wrote last time about the serious end of this problem, the business-critical application where the stakes are obvious and the testing was always going to happen one way or another, and I made the case for application integrity, the discipline of checking the running application against what it is supposed to do rather than checking the source against what the code is supposed to do. This is about the other end. It is about the software that was never important enough to test, right up until the afternoon it was. It is not about the mature product with a large existing suite either, where the interesting question is what an agent adds to the tests you already have, which is a subject for a later piece in this series. Here I mean the software that shipped without tests because product-market fit was not confirmed yet, and speed to feedback mattered more than the coverage and hardening you give something once you have decided to keep it.
For as long as I have been around testing, software like that demo got no real coverage, and the reason was plain economics. To test an application properly you had to understand it, understanding it well enough to write a useful suite took a person a meaningful stretch of time, and you could not justify that time on something whose future you had not yet confirmed. So the experiments and the demos and the internal tools and the proofs of concept all shipped with nothing, and most of the time that was the correct call, because most of them really did turn out not to be worth keeping. The trouble was always the ones that were.
And the demo was never the exception. Shipping without a suite is the ordinary condition, not a special failing of the careless: when researchers combed through ten thousand of the most popular, actively maintained repositories on GitHub in 2026, only about a third kept a dedicated test directory.1 The developer surveys say the same from the other side, with roughly two in five people who work on testing reporting that they write no automated tests at all.2 If that is the baseline even where the resources and the reasons to test are all present, the week-old demo never stood a chance.
AI has changed the arithmetic on both sides. It has made the building cheaper, so there are far more of these things than there used to be. GitHub’s own figures for 2025 have developers creating more than two hundred new repositories a minute, and the number of projects pulling in a large language model rising by almost half again across a single year.3 A lot of that is exactly the fast, speculative, here-this-week software I am describing, and it is arriving faster than anyone’s testing practice was built to absorb. The feeling my readers describe, of being one step behind all the time, is partly this: the volume of things that could plausibly need checking has climbed, and the hours available to check them have not.
What an hour buys now
Here is what has actually shifted. Standing up a test surface for one of these applications no longer takes the deep, days-long study it used to. A tool like BearQ, the agentic QA system SmartBear describes as a set of always-on teammates, works by exploring the running application the way a user would, learning the workflows by moving through them rather than by reading the code that produces them. That matters here for a reason that has nothing to do with sophistication: the person setting it up does not have to become an expert in the app first. You point it at the thing and you have a basic, working test layer in under an hour.
Three things follow from that. The first is the obvious one, that assurance you can buy for an hour of setup is affordable for software that could never justify a week of it. The objection I hear most about testing tools is that the return does not justify the cost, and that objection was usually right when the cost was measured in days of a skilled person’s attention. At an hour the sum is different, because you are weighing a small, known, upfront cost against the chance of having no safety net at all on something that turns out to matter.
The second is that the tests are produced by something that did not write the application, so they come at it from outside the assumptions baked into the build. One of the most common complaints I hear about AI-generated tests is that they read like a web crawler rather than a user, that they exercise the code with no sense of what the person in front of the screen was trying to do. An agent working through the running application, against what it is meant to do rather than how it is wired together, sits closer to the user’s vantage point than to the code’s, and that perspective is worth something on its own, separate from the time it saves.
The third gets undersold. The tests and the reports these tools produce are easy to share and easy to read, and that turns them into more than a technical artefact. The times I have seen a team that barely spoke start arguing productively, in workshops and in the conversations afterwards, it has usually been over something concrete, and a readable account of what the application does and where it breaks is about as concrete as it gets. For a profession that spends a lot of its energy worried about being treated as the bottleneck rather than a contributor, a tool that puts your work in front of the rest of the team in a form they will actually read is worth more than the coverage alone.
But why test something you are going to throw away?
The common objection to all of this is that I am inventing a reason to test things that do not deserve it. If a piece of software is a throwaway, let it be a throwaway, and do not manufacture confidence in something that should be allowed to disappear. There is a sharper version too, that a quick generated test layer hands you the comfort of a green suite with nothing behind it, and false confidence is worse than no confidence at all.
Those are good arguments, or rather, they were in the before times. The first half of it no longer works, because the premise underneath it has moved. The objection assumes that testing the throwaway costs something worth weighing. That was a safe assumption when a test layer meant days of work. It is a much weaker one when a good-enough starting layer, one that drives the running application and checks it does roughly what it is for, can be generated in the time it takes to read this piece. When the effort the objection priced in is mostly gone, the conclusion it reached goes with it. You are no longer choosing between an expensive suite and a shrug. You are choosing between a cheap baseline and nothing, on software whose future you cannot yet know.
And you genuinely cannot tell. The instinct to sort the keepers from the throwaways at the start runs straight into the fact that nobody is much good at it. The public numbers on deliberate proofs of concept point the other way if anything, with industry analysts putting the share that reach production at around a fifth and some surveys lower still,4 so I am not claiming most experiments survive. I am claiming you do not know which ones will, and that the ones that do have a habit of becoming load-bearing without anyone deciding they should. The demo the client insisted on keeping is the pattern, not the exception. When one of them survives, the shortcuts remain in place, and the cost of putting quality back in afterwards is higher than the cost of having had a baseline from the start. The exact multiplier people like to quote for that is folklore, but the direction of it is one of the steadier findings in software economics, that gaps grow more expensive to close the longer they sit,5 and that organisations already spend most of their development effort finding and fixing rather than building.6
The false-confidence half of the objection is fair, and the answer is to be clear about what the baseline is for. A generated test layer on a young application is a floor you build up from. It is there to catch the survivors doing something they obviously should not, before anyone has paid for the proper suite. Read as a guarantee it would be dangerous. Read as the floor it is, it beats the nothing these applications shipped with before.
There is a line, and drawing it is what keeps this from turning into test-everything zealotry. Some work really is disposable, the spike you will delete tomorrow, the one-off script, the experiment that never touches a user or feeds a decision, and for that an hour is an hour wasted. The test I use is one question: could this plausibly survive, reach a user, or feed a business decision? If the answer is no, let it stay disposable with a clear conscience. If the answer is even a maybe, the hour is cheap insurance against the meeting where the maybe becomes a yes.
The floor, and who it gets talking
None of this displaces the case I made last time. The discipline of running a testing agent well, the metrics and the calibration and the judgment about where the human belongs, still matters most where the stakes are highest. What I have come round to is that the same tools doing that serious work have a second use that is easy to miss, which is bringing application integrity within reach of the software that never had any of it. The speed is what makes them fit there, and the side effect of the speed is that a kind of software that used to ship with no record of what it does now arrives with a readable one, which is the first thing a divided team has had to talk about in a while.
The case for it is not that the experiments will survive, because most will not. The case is the shape of the bet. The cost of the test surface is small and you pay it now. The cost of going without it is large and you pay it later, on precisely the experiments you could not pick out as keepers when they were born. Given the choice, and given that the price of being wrong has dropped to about an hour, I would rather have the floor in place and not need it than need it in the meeting where the demo becomes the product.
- “What’s Inside a GitHub Repository? An Empirical Study on the Contents of 10K Projects,” arXiv:2605.16701, 2026. A study of 10,000 repositories with at least 100 stars and 100 commits that were active in 2026, of which around 35% held a
testsdirectory and 30% atestdirectory. It detects organised test directories, so projects with test files co-located beside their source are undercounted; read it as a measure of software without a dedicated suite, not of literally untested code.↩︎ - JetBrains, The State of Developer Ecosystem 2023, testing section: among developers involved in testing, 58% write automated tests, leaving roughly two in five who do not. A self-selected survey that likely skews towards people who already test.↩︎
- GitHub, Octoverse 2025. Figures are GitHub’s own platform telemetry; treat the direction as solid and note the platform’s interest in the AI story.↩︎
- Gartner has put AI proof-of-concept-to-production at roughly 20%, reported via The Register, 2025; a 2025 MIT/NANDA report put generative-AI pilots reaching production nearer 5%. Both measure deliberate POCs, not throwaway code, so they are used here as a floor on how badly we predict survival, not as a direct figure.↩︎
- Barry Boehm, Software Engineering Economics (1981). The popular “100x” version, usually attributed to an “IBM Systems Science Institute” chart, is untraceable and Boehm called his own figure notional, so only the direction is cited.↩︎
- NIST, The Economic Impacts of Inadequate Infrastructure for Software Testing (2002): organisations spend roughly 80% of development cost finding and fixing defects. Dated, but a primary economic study.↩︎