Automated testing in the AI era: speed without confidence is just risk at scale

Not long ago, a common scenario in software companies was the high time cost of writing automated tests. On one hand, they guaranteed that the system’s behavior matched what was expected, as long as the business rule had been implemented correctly. On the other, they took a while to write, which ended up determining whether an application would have good coverage or not. Of these tests, unit tests were always the most numerous and the most valuable.

Today, though, AI tools are changing that whole process and how software gets built. A developer can now get tests for a feature in a matter of seconds, including integration and e2e tests, easily pushing coverage into the 80% range. It got genuinely simple. Agents can navigate a codebase, create files, change APIs, refactor modules, implement pretty much anything in very little time, drastically cutting the cost of producing code.

But here’s the catch. There’s a massive difference between producing code quickly and producing software you can trust, because faster development doesn’t mean better quality — it can just as easily raise the odds of shipping incorrect behavior. Given all that, it’s worth understanding that testing is no longer just a traditional quality practice; it’s become one of the main feedback and verification mechanisms for AI-assisted development.

But before we get into the current state of software development, let’s rewind and look at the broader context behind automated testing.

Testing doesn’t prove the software is correct

In 1969, at the NATO Software Engineering conference — a conference convened to decide where the field of software was headed — Edsger W. Dijkstra made an observation that would become one of the best-known lines in the field:

“Testing shows the presence, not the absence of bugs.”

The line appears in the proceedings of the Software Engineering Techniques conference, in the context of a discussion about the limits of testing.

More than half a century later, that distinction still holds.

Let’s take an example. Picture a simple function:

function calculateDiscount(total: number) {
    if (total >= 1000) {
        return total * 0.9
    }

    return total
}

We can write dozens of tests for it. If they all pass, we know certain behaviors have been verified.

But that doesn’t mathematically prove no incorrect case exists.

There could be a business rule we forgot. An edge case we didn’t think of. A misread requirement. Or we might simply have written a test that confirms the exact same wrong assumption baked into the implementation.

That’s always been true.

With AI, though, an interesting consequence shows up: the AI can generate both implementation and test from the same wrong interpretation.

When the AI misreads the requirement, the mistake propagates into both the code and the test — the result is a “passed” that doesn’t actually mean anything.

A green test still just means the observed behavior matches what the test expects. It doesn’t necessarily mean we built the correct behavior. If the business rule is implemented wrong, and the test checks that same rule, the behavior is wrong even though the test passed.

That’s one reason tests don’t replace review (human or AI), domain understanding, acceptance criteria, and product validation — no matter who wrote the code.

Tests as a safety net

Besides Dijkstra, another important figure in computing is Michael Feathers. In his 2004 book Working Effectively with Legacy Code, he offers a deliberately provocative definition:

“To me, legacy code is simply code without tests.”

Feathers’ concern wasn’t legacy code itself, but the fact that without tests, we can’t quickly tell whether a change preserved the behaviors that matter.

That idea may be even more relevant now, with AI-assisted development.

A relatively large change used to require hours of manual work. Now we can just ask:

Refactor this module to separate the business rules from the infrastructure.

Or:

Migrate this implementation to a different ORM without changing the API’s behavior.

An agent can touch dozens of files. With that, the question shifts from “How long will this change take to write?” to, increasingly, “How do we know this change is still correct?”

That’s where a test suite works as a safety net.

AI can massively increase the amount of change we’re able to produce. Tests help scale our ability to verify it proportionally.

That might be one of the biggest assets of testing in this context: they make sure human expectations about the system get checked through executable tests.

Does the testing pyramid still hold up?

In Succeeding with Agile, Mike Cohn popularized the Test Automation Pyramid. The original idea splits the strategy into three levels:

• lots of Unit Tests;
• a smaller number of Service Tests;
• few UI Tests.

Martin Fowler later helped popularize the concept further. The core principle is simple: a healthy suite tends to have far more small, fast tests than broad tests that run through the whole application.

The pyramid doesn’t need to be read as a rigid ratio, and later literature itself acknowledges that different systems can call for different distributions. There are alternative strategies like the Testing Trophy and the Testing Honeycomb, and the line between “unit” and “integration” isn’t always agreed upon. Fowler notes that the pyramid is still useful mainly as a heuristic: use tests of different granularities, and avoid relying too heavily on broad, slow, brittle tests.

In other words, it’s still relevant with AI in the loop.

An agent that keeps changing code continuously needs fast feedback. If every change requires a twenty-minute E2E suite just to know whether something fundamental broke, the development cycle loses most of its advantage. On the other hand, relying only on unit tests is dangerous too. An agent can produce components that work perfectly in isolation and still wire them together incorrectly.

So the more useful move isn’t following the pyramid literally — it’s building layers of confidence:

Each layer checks something different: E2E covers critical flows, integration verifies components working together, unit covers rules and behavior.

In practice, that means mapping each layer to a specific kind of risk, instead of applying all three generically:

• Unit covers the business rules — each use case tested in isolation, with no dependency on a database, network, or any external collaborator. It’s the cheapest layer to maintain and the one that points to the error closest to its cause.
• Integration covers the points where the system touches the real world — a repository implementation, a database query, a call to an external service. The risk here is rarely the rule itself; it’s how it gets persisted, retrieved, or translated between systems.
• E2E covers the full flow, but the cost depends on the layer: API-level e2e (no browser, hitting the endpoint directly) has become cheap to write and fast to run — today it’s reasonable to cover most backend functionality this way, guilt-free. UI-level e2e (browser, clicks, screens) is still expensive to run, even with AI cutting the cost of writing it — there, selectivity still matters: reserve it for the flows the business can’t afford to let break unnoticed.

The faster the relevant feedback, the shorter the gap between producing a change and finding out it’s wrong.

Red, Green, Refactor — now with AI in the loop

Another common practice around automated testing is TDD (Test-Driven Development).

Kent Beck’s popularized TDD cycle is simple:

Red: write a test that fails.
Green: implement just enough to make it pass.
Refactor: improve the design while preserving behavior.

Today, AI itself can carry out TDD based on the nature of the feature and the stated functional requirements, respecting their rules and expected behavior.

For example:

Implement a commission calculation system. Zero-value sales generate no commission; sales up to $10,000 earn 5%; above that, apply the higher-bracket rule. Also implement automated tests for this behavior.

The AI can then produce the implementation, write the tests, run them, observe failures, fix the code, and run the suite again. And the principle holds: changes get produced inside a fast cycle of implementation, verification, and feedback.

In this scenario, the developer’s role shifts increasingly toward correctly defining the expected behavior, while it falls to the AI (and the developer too, when it makes sense) to judge whether the code and tests actually represent that behavior.

But in a context where the AI writes both the feature and the tests, a problem can show up: what happens when the AI itself misreads the requirement and writes both the code and the tests around that same mistake?

The problem with AI writing its own tests

Consider this flow:

At first glance it looks great. But there’s a circularity hiding in it: if the AI misread the requirement, it can implement that interpretation and then write tests that confirm the exact same interpretation — code and tests agree, but both are wrong.

In practice, few teams ship straight from a prompt. The real flow usually involves an issue, a feature branch, a PR, and CI running before the merge. But that alone doesn’t solve the problem: if the AI wrote the code and the tests from the same wrong interpretation, the PR will arrive with green CI, and if the review is just “tests passed, diff looks reasonable,” the same circularity happens anyway — just now hidden behind a process that looks rigorous.

That’s why a more robust strategy separates, when it matters, specification from implementation.

Either way, the fundamental question stays a human one — and depends on the specific product and the feature’s acceptance criteria: which behaviors does this product need to guarantee?

What should we actually test?

The ability to generate tests at scale creates an additional risk: confusing number of tests with suite quality.

An AI can produce hundreds of tests in minutes. That doesn’t mean they’re good tests.

A valuable suite needs to catch meaningful changes in behavior. Tests that only mirror internal implementation details can boost coverage while, at the same time, making refactors needlessly painful — any internal change breaks the test, even when the behavior is still correct.

Compare these two examples:

expect(repository.findUser).toHaveBeenCalledTimes(1)

This test checks an implementation detail: how the system reached its result. Swap the repository for another one, or add a cache in front of the call, and the test breaks — even though the behavior is still correct.

expect(result.status).toBe('approved')

This one checks observable behavior: what the system produced. It survives practically any internal refactor, as long as the result stays correct.

The question stops being just:

How much of the code is covered?

And starts including:

If the behavior that matters changes incorrectly, will some test catch it?

That difference matters even more now that generating test code is nearly free: it’s easy to ask AI to “increase coverage,” and just as easy for it to respond with dozens of tests that lock the implementation in place instead of protecting what actually matters.

Tests are also context for AI

This isn’t unique to AI — it’s always been true for developers trying to understand a system’s spec, long before AI entered the picture.

A well-written test reveals:

Given
    customer has an active subscription

When
    customer cancels the subscription

Then
    access remains active until the billing period ends

That carries domain knowledge.

When exploring an unfamiliar codebase, an agent can read tests to find usage examples, API contracts, invariants, edge cases, and behaviors that shouldn’t change. In that sense, tests end up playing at least three roles at once:

• verification, because they catch unexpected behavior;
• executable documentation, because they record examples of expected behavior;
• context for agents, because they make part of the system’s intent readable and checkable by automated tools.

The better they express behavior, the more value they carry in all three roles.

Tests as trust infrastructure

Anyone in the field knows that, for decades, automated tests have been championed because they let us change software with more safety, and even more ease.

Michael Feathers put that relationship between tests and the ability to change code at the center of his definition of legacy code. Mike Cohn showed that confidence needs to come from a balanced testing strategy. Kent Beck put tests inside the development cycle itself with TDD. And Dijkstra reminded us, long before any of that, that tests have limits: finding evidence of correctness isn’t the same as proving the absence of defects.

None of those ideas lost relevance with AI. What changed was the speed.

What changed wasn’t the value of tests — it was the length of the cycle between having an idea and finding out whether it’s right: from hours/days to seconds/minutes.

A significant change used to require hours or days of implementation. Today, models and agents can generate, modify, and refactor large parts of a codebase in minutes. When producing change gets cheaper, a bigger share of the problem shifts away from implementation and toward specifying, verifying, and deciding.

Given all that, automated tests are even more essential in this era, and a few questions remain fundamental:

Did this change do what it was supposed to do?

Was the behavior that needed to stay the same actually preserved?

Do the components still work together?

Can we ship this change to production with confidence?

Automated tests don’t perfectly answer all of these. As Dijkstra observed decades ago, they don’t prove the absence of bugs. But they provide something especially valuable in an increasingly fast development process: automated, repeatable, executable feedback on software behavior.

AI lowers the cost of producing change. Tests lower the cost of verifying part of that change.

It’s precisely the combination of those two capabilities — fast generation and fast verification — that lets us turn development speed into delivery speed, without turning that same speed, in equal proportion, into risk.

In the age of AI-assisted development, then, automated tests aren’t just a safety net anymore. They become part of the trust infrastructure that humans and agents rely on to keep changing software continuously.