For much of my career, automation had a straightforward purpose: make work more consistent. People forgot things, interpreted instructions differently, and made mistakes. Systems helped reduce that variability.
I remember information moving between departments on paper, being reviewed, re-entered, and reviewed again before anyone could act on it. Automating those steps improved speed, but the value I came to appreciate most was predictability. A defined process could be repeated, checked, and reconciled. That never made it infallible, but it gave organizations a more consistent basis for operating.
Now generative AI is being used to interpret information, develop recommendations, and support judgment. We value its ability to work with ambiguity—the very condition earlier automation often tried to reduce.
That is the automation paradox: we are introducing a different kind of uncertainty into processes built around repeatability. The question it raises is plain:
Can we trust the outputs enough to act on them?
Predictability made scale possible
Mainframes, transaction processing, batch jobs, reconciliation, and audit trails gave organizations ways to perform work consistently and investigate exceptions. Just as important was knowing what a process was supposed to do and being able to check whether it had done it.
That consistency supported scale. More work could pass through a defined process without requiring someone to interpret every step.
Predictability had its own failure mode: a system could repeat the same mistake faithfully. Controls, testing, and reconciliation helped establish whether the repeated result was the right one.
Distributed computing, the internet, and cloud services extended those processes across organizational boundaries. Each shift introduced security, coordination, vendor, and governance challenges, but none, by itself, turned a process's rules into judgment.
Generative AI introduces a different condition.
From executing rules to generating answers
A defined process is deterministic: it follows specified rules, and under the same relevant conditions it produces the same result. A generative language model produces responses through statistical prediction. It can generate useful, accurate answers, but it can also produce plausible content that is wrong or inconsistent. NIST describes this risk, which it calls confabulation, as a natural result of how generative models are designed.
This matters when the output becomes a recommendation someone relies upon. At that point, the basis for trusting it changes.
The checks that served defined processes—did it run, did it complete, does it reconcile—do not show whether a recommendation is sound. A complete answer can still rest on an incomplete understanding.
Imperfect information can become persuasive
Consider what happens when the information supplied to a system includes outdated procedures, incomplete documentation, unclear ownership, and exceptions known only to the people handling them.
Those weaknesses already affect organizational decisions. Generative AI can present an answer whose fluency makes the weaknesses harder to notice.
An explanation can sound coherent while omitting a critical exception. A recommendation can appear authoritative without sufficient support. NIST specifically warns that confident false content—and explanations or citations offered in support of it—can encourage inappropriate reliance.
The old phrase was "garbage in, garbage out."
Generative AI adds a new one: garbage in, persuasive garbage out.
Poor input is not the only source of error. Even with sound information, generated output still needs appropriate validation. Improving the information and validating the answer address different parts of the problem.
Oversight is work, too
Reducing manual effort deserves attention. So does the work required to determine whether an output can be relied upon.
Someone may need to investigate an unexpected recommendation, trace its supporting information, assess its limitations, or decide when it should not be used. These responsibilities require people who understand the subject and the consequences of getting it wrong.
This is not a forecast of whether AI will create or eliminate jobs overall. It is work an organization must account for when deciding what to automate.
The oversight should fit the use. Reviewing a draft summary and authorizing an operational action carry different consequences. The question is what validation and intervention the particular use requires.
Putting a human in the loop is not enough on its own. That person needs the knowledge, time, and authority to challenge the output.
The operating model has to account for both the capability gained and the oversight that comes with it.
The question remains
The automation paradox asks us to reconsider what makes a system's output trustworthy. Defined processes earned trust through repetition and reconciliation. The outputs of generative AI must earn it through validation.
We can benefit from those capabilities while recognizing the judgment still required: understanding the information behind an answer, testing whether it supports the recommendation, and deciding when reliance would be misplaced.
The technology has changed. The question remains:
Can we trust the outputs enough to act on them?
