Adoption is widespread, but the metric matters
To look at the other side of the workflow, DeviQA published an internal survey in July 2026 of 300 people working in QA, SDET, and test leadership roles. Fifty-two percent said they had noticed more bugs since their development teams began using AI, and 58% reported a heavier testing workload. That is a signal worth investigating, not a representative estimate of the whole industry: it is a vendor’s own survey based on people’s recollections.
To look at the other side of the flow, DeviQA published its own survey in July 2026 of 300 QA professionals, SDET and testing leaders. 52% said they perceived an increase in bugs since development began using AI and 58% perceived more testing load. It is a relevant signal to investigate, not a representative estimate of the sector: it is an internal supplier sample, based on self-reported responses and experiences.
DORA’s 2025 report describes AI as an amplifier of the system in which it is used. Fast feedback, strong internal platforms, and clear priorities can help teams make better use of it; weak processes and hard-to-review changes can magnify existing problems. The findings show associations, not a universal cause-and-effect rule. Teams should measure the whole delivery process instead of attributing every change to one tool.
DORA 2025 proposes to read AI as an amplifier of the existing work system: quick feedback, internal platforms and clear priorities help; Weak processes and changes that are difficult to review are also amplified. Research reports associations, not a universal causal rule. The practical message is to measure the entire flow, not attribute variation to a single tool.
What to measure before changing the process
Records over several iterations how many changes are made and what size; how many defects appear before and after publishing; its severity; how many times a case is reopened; how long it takes QA to receive and respond; and how much work is repeated. Separate changes made with the help of AI only if the team can identify them consistently and without monitoring each person.
Compare the periods carefully: use the same definitions, note product or equipment changes, and review both what the tests detect and the alerts that turn out to be false. If more changes are released, the total number of defects may increase even though each change has fewer. Comparing defects with the number and size of changes helps understand what happened.
Suppose a team used to ship ten changes a week and find four defects. After adopting an assistant, it ships twenty changes and finds six. The total is higher, but the rate has fallen from 0.4 to 0.3 defects per change. Without a denominator, “more bugs” can simply reflect more changes. Detection can also rise because the team added tests, improved logging, or reviewed changes more closely. In that case, finding more failures may mean the team is seeing problems it used to miss, not that the product has become less reliable.
It is advisable to separate four measures: defects found before publishing; defects that reached production; severity and impact; and cost of investigating and correcting them. Finding more minor issues during testing does not equate to having more user incidents. Conversely, a stable rate can hide greater risk if the number of changes increases or if failures are concentrated in payments, permissions or data. Decide what you are going to tell before comparing.
Why review AI-generated code if tests already run?
Passing tests does not prove that a change is maintainable, follows the project’s conventions, or has adequate coverage. In a 2026 study, METR asked four maintainers from three projects to review 296 agent-written changes that had passed an automated evaluation. The maintainers accepted a smaller share than the evaluator did, a gap of 24 percentage points. The study was limited, and agents could not revise their proposals after receiving feedback. Still, it illustrates an important distinction: passing tests and meeting a team’s standards for merging are not the same thing.
Therefore, in addition to checking that the function does what is expected, check if the change affects other functions, how it handles errors, if it respects security and if the new tests really detect the problem. A generated test can repeat the same error that the code has. It asks each test to relate to a requirement and checks to ensure that it fails when the function it was intended to protect is removed.
For four to six weeks, record a baseline with stable definitions and select comparable changes by size, risk and component. If the team uses AI, label the change voluntarily and at the job level, not the person level. Compare groups or periods when feasible, note simultaneous changes—new coverage, refactor, team turnover—and normalize by changes, modified lines, or exposure. Don't use a single percentage as a goal: it may encourage hiding defects or reducing coverage.
A useful dashboard shows how many defects appear, how serious they are and at what stage they are detected; how many reach production or reopen; how long it takes for the team to receive a response; and how many automated alerts a person confirms. Keep examples to understand where the problems come from. If the team works faster, but more serious defects arrive in production or rework increases, the process has not improved. If you catch more before you publish and less make it to production, you may be finding problems better now.
A recent survey points to a heavier QA workload
The practical response is not to add tests blindly. First find where work is backing up: waiting for an environment, unclear requirements, late reviews, difficult test data, flaky failures, or alerts that nobody can triage. Each calls for a different fix. Adding another AI tool may create more noise if the team cannot see what changed or what outcome was expected. With a clear diff, risk, and expected result, the team can prioritize review. Track waiting time separately from hands-on testing time to tell a capacity problem from a workflow problem.
The practical consequence is not to assign more blind tests. It is worth asking where the queue forms: waiting for environment, incomplete requirements, late review, data that is difficult to prepare, intermittent failures or too many unacknowledged alerts. Each cause requires a different response. If the team doesn't know when a delivery will arrive or what changed, a second AI tool can add noise; If you know the diff, risk, and expected outcome, you can prioritize better. It measures the waiting time separated from the effective testing time: this is how you distinguish a capacity problem from a flow problem.
An absolute count answers how many defects were found; it does not alone indicate whether each change is worse. Suppose a team reviewed ten changes and found four defects, and then reviewed twenty and found six. The total grew, but the exchange rate fell. Even that rate can be misleading if the new changes are larger, touch different components, or receive more coverage. Use various denominators depending on the decision: revised changes for triage load, size and risk to compare exposure, executions for stability, and users or transactions to estimate impact. Don't turn lines of code into a universal measure of quality: they serve as context, not a substitute for functional risk.
Also separates severity, time of detection and cost. A blocking defect detected before releasing can be a sign that the tests are working; a minor one in production may be less serious than a permissions failure even though both count as “a bug”. Record whether the problem was in requirement, implementation, integration, data or environment. This classification serves to correct the process and not blame the author or the tool. Keep representative examples and review what changed between periods; a metric without an explanation of the case can push the team to optimize the wrong number.
How to evaluate a tool before adopting it
Before adding AI to a workflow, write down a question that could have a “no” answer—for example, “Does generating tests from stories reduce omissions without adding review time?” Choose a bounded task, record a baseline, and agree on what counts as a missed case, duplicate, valid alert, or rework. Compare several iterations and changes with similar risk. If possible, alternate between the existing and assisted workflows. If other factors change—coverage, staffing, release cadence, or tools—record them. A simple before-and-after comparison cannot establish that AI caused the difference.
The controlled test also needs a stopping rule. If the tool does not have sufficient context, uses real data without authorization, modifies a non-isolated environment, or produces results that no one can reproduce, it should be stopped and pending execution. Human review focuses on relevant risks, not accepting everything the model generates. At closing, it presents results along with sample limits, false positives, escaped defects, and review time cost. If the data does not support a conclusion, that is a valid conclusion: the next step is to improve the measurement, not declare success.
An automated alert is an observation to investigate, not a confirmed bug. It may point to a new defect, a known regression, a harmless visual difference, an environment problem, or a test that no longer reflects expected behavior. If a dashboard counts every alert as a bug, it can make a tool look more effective while duplicates and noise pile up. Use separate states for received, reviewed, reproducible, confirmed, duplicate, and dismissed findings, and record why an alert was dismissed. That gives the team a clearer view of triage accuracy and effort.
Look at timing as well as volume. A failure detected in minutes does not have the same cost as one reported at the end of an overnight suite. An accurate alert that nobody sees until after release does not help prevent the issue. Record when the system reported it, when someone reviewed it, and when it was closed. If alerts rise while the confirmed share falls and review takes longer, the next experiment should improve selection and context—not generate more cases. Earlier detection of relevant defects combined with less downstream rework is stronger evidence that the tool is helping.