Executive Summary
Once model output can initiate a state-changing operation or supply its parameters, the model is in the transaction path, and the architecture has to change with it.
For CTOs, VPs of Engineering, and SaaS product leaders, this is the point where agent risk becomes business risk. A model that suggests the right refund becomes a system that can issue it twice, apply it to the wrong tenant, or report success when half the work failed. Those failures land on support, security, finance, and customers, and they cost the trust that made the agent worth shipping.
A chat interface and existing APIs look sufficient while a person completes every transaction. Remove the person and five engineering responsibilities surface: authority and policy, durable execution, transaction-safe tool contracts, containment and recovery, and evidence. This article traces one state-changing action, an invoice due-date extension, through all five.
Key Takeaways
- The boundary is the transaction path, not the label on the interface. Copilots can trigger actions and agents can be read-only; the line is whether model output can initiate a state change or an external side effect.
- Two operating models, the same product execution boundary. The agent operator owns orchestration and request intent; the product still enforces identity, authorization, business policy, transaction semantics, and evidence at its boundary.
- Five responsibilities sit around a model that only proposes. Authority and policy, durable execution, transaction-safe tool contracts, containment and recovery, evidence and operations. Each one prevents a specific business failure, from unauthorized changes to duplicate transactions.
- Evals are release gates, and cost is measured per verified outcome. A cheaper model call is not cheaper if it raises retries and failed actions. For an external agent, the measure narrows to verified operations at the product boundary.
Where AI Suggestions End and AI Actions Begin
Take one request: extend invoice INV-1047’s due date by 14 days. While a person reads the suggestion, judges it, and performs the update, the model is advisory. The moment model output can initiate that operation or supply its parameters, the model has entered the transaction path, and the transaction path is where the engineering changes.
A human approval may gate the commit. It does not remove the model from the path, because the product still needs authorization, durable execution, retry safety, recovery, and evidence on the far side of the click.
Copilot or Agent: What Actually Changes the Engineering
The copilot-versus-agent debate is mostly a labeling argument. Copilots can trigger actions. Agents can spend a whole session reading. The line that changes the engineering is whether a model-selected operation can change product state or produce an external side effect. Both halves of that sentence count: sending a contractual communication may change nothing inside the product while committing the business to something outside it.
Concrete versions of the crossing are easy to list. Extending a due date. Issuing a refund. Changing access rights. Updating a system of record. Each one is a privileged product workflow, and the moment a model can select and trigger it, the operation deserves the same discipline as any other state-changing transaction in the product.
Six Questions a Chat Interface Cannot Answer
A chat interface plus API calls can look sufficient for a long time, because a person does the unglamorous half of the job: interpreting each response, judging whether the suggestion is safe, and performing the actual transaction. Remove that person and the hidden responsibilities surface as questions the stack cannot answer:
- Identity: Who is acting, and on whose behalf?
- Permission: Is the operation permitted for this tenant, this resource, this amount, this state?
- Failure: What happens when the process dies halfway through?
- Retries: Can a retry duplicate the side effect?
- Final state: How does the product establish the final state?
- Evidence: What evidence survives for support, security, and finance?
Product-Owned and External Agents Share One Execution Boundary
Before the architecture, a distinction that saves teams from building the wrong thing. Agents reach a product in two ways:
- Product-owned agent: The agent is embedded in the SaaS product itself. The product team owns the runtime, orchestration, permissions, user experience, and the operational controls around it.
- External agent: The agent belongs to a customer or a third party and acts through agent-facing tools the product exposes. The external runtime is not yours. The tool contract, the authorization boundary, business policy, transaction behavior, and the evidence of what happened still are.
Why an Agent Interface Does Not Transfer Responsibility
Teams routinely assume that exposing an agent interface transfers responsibility to whoever runs the agent. It does not. The external caller chooses when to request an operation. The product decides whether the operation is permitted and carries it out safely.
The transaction controls in this article do not get lighter when the agent is external. What narrows is the observability boundary: the product can attest to the caller, the delegated principal, the request, the policy decision, the execution, and the resulting state. It cannot attest to events inside a runtime it does not operate. Either way, the product answers the same questions, and they cluster into five responsibilities.
The Five Responsibilities Your Product Must Engineer Around the Model
The questions from the opening map onto five engineering responsibilities. None of them lives in the model, and each one protects the business from a specific failure.
The model may propose and sequence actions. Deterministic services authorize, execute, record, and verify them.
1. Authority and Policy: Who Can Act, and for Whom
Four decisions hide inside the word “allowed,” and agents add a fifth. They need separating:
- Authentication: Establishes who is calling.
- Authorization: Establishes which resources and operations that identity may reach.
- Business policy: Establishes whether the action is allowed given current rules and product state.
- Approval: Establishes whether a person has authorized this specific instance where the policy requires one.
- Delegation: Establishes whom the verified caller acts for. The product needs both identities, the agent and the principal it represents, before the other four decisions mean anything.
Scoped OAuth is useful and rarely sufficient, because a token scope seldom expresses the full business decision: the tenant boundary, the resource, the amount limit, the lifecycle state, the time window. Where approval is required, bind it to the exact resource, arguments, expected version, and an expiry. An approval that floats free of its arguments lets someone substitute different arguments after the click.
The model can interpret intent and propose parameters. It must never determine its own authority. Deterministic services enforce permissions and policy, and they do not care how persuasive the proposal sounded.
Business impact: Only permitted changes reach the system of record, and the agent never becomes a back door around rules the product already applies to people.
2. Durable Execution: Keeping Long-Running Work Alive Through Failures
Model context is not durable workflow state, and confusing the two is a common cause of partial-failure bugs. A production runtime preserves three kinds of state:
- Progress: Task status, completed and pending steps, and external operation identifiers.
- Control: Approval state and deadlines, retry history, and cancellation status.
- Context for recovery: The policy and tool-contract versions in force, and whatever compensation state recovery will need.
Long-running work has to survive model calls, process restarts, and network interruptions.
The MCP 2026-07-28 specification ships Tasks as an optional extension for long-running operations, with durable task handles, polling, and cooperative cancellation. It is useful at the boundary, and it is not a substitute for a workflow engine or for the business transaction design behind it.
Business impact: A crash or restart never leaves the business guessing whether an operation ran, and no one has to reconstruct half-finished work by hand.
3. Transaction-Safe Tool Contracts: Making Every Retry Safe
An agent-facing action needs a contract stricter than a typical endpoint:
- Inputs and authority: Typed inputs and outputs, and the required identity and authority.
- Preconditions: The conditions that must hold, and the expected resource version.
- Outcomes: Idempotency semantics, terminal and non-terminal results, retryable and non-retryable errors, a stable operation identifier, and a stated verification method.
The schema checks shape. It does not check truth. Model-produced arguments are untrusted input, so server-side controls still enforce tenant ownership, ranges, destinations, and current-state preconditions on every call.
Idempotency deserves its own caution, because idempotency is not exactly-once execution. The handler may run more than once, and the caller may still receive no response. What it guarantees is that retries with the same key and the same semantic request do not create additional intended business effects, which is what makes retries safe to attempt, as AWS’s Builders’ Library lays out.
After an ambiguous timeout, the workflow retrieves the operation outcome or reads authoritative state before deciding to retry. The contract defines the key’s scope, retention period, request fingerprint, and replay response, and it rejects a reused key with materially different arguments.
None of these requirements originates with agents. They are the same disciplines high-volume payout and licensing workflows have always demanded, including systems we engineer for a global brand-licensing organization. An agent changes who initiates the action, not the standard it must meet.
Business impact: Retries recover from failures without issuing a second refund, a second update, or a second customer message.
4. Containment and Recovery: Limiting Damage and Recovering From Mistakes
Vague talk of guardrails dissolves into a short list of concrete controls: maximum amounts and action limits, allowed resources and tenants, rate and concurrency ceilings, restricted destinations, approval requirements, time windows, circuit breakers, and explicit marking of irreversible steps.
Two agent-specific additions belong on that list. Retrieved content and tool output are untrusted data, so nothing an agent reads gets to widen what it may do. And limits are cumulative as well as per action, because a ceiling on one call means little if the same total can be moved across ten. The Agent Access Matrix decisions (article soon to be published) feed these controls directly; the matrix is finished only when it exists as runtime policy.
Recovery needs three words kept apart:
- Cancellation: A request to stop further work.
- Rollback: Reversal inside a transaction boundary.
- Compensation: A new business action intended to offset an earlier committed one, in the sense of Microsoft’s compensating transaction pattern.
Stopping a workflow does not reverse its side effects, and compensation can itself fail, so it needs the same durable, resumable, auditable execution as the original action.
Business impact: Limits cap how much a single bad call, or a series of small ones, can move, and recovery follows a defined business rule rather than an improvised fix.
5. Evidence and Operations: Proving What Happened After Every Action
After the action, the system should be able to reconstruct the whole story: who or what initiated it, on whose behalf, which authority and policy applied, which tool ran with which arguments, what state existed before, what changed, and whether the intended outcome was verified.
How far that record reaches depends on who runs the agent. For a product-owned agent, the record can extend into the runtime: model and runtime versions, the orchestration path, the user-visible response, token usage, and total task cost. For an external agent, the authoritative record begins and ends at the product boundary. The product cannot truthfully record what someone else’s agent told its user, and should not pretend to.
Observability and audit draw on the same infrastructure but aren’t the same thing. A diagnostic trace is not automatically an authoritative audit record, and the audit record is the one that has to stand up later. Hidden model reasoning stays out of both.
Business impact: Support, security, and finance can reconstruct and stand behind any action the agent took. The next section shows all five responsibilities working on a single request.
How One Invoice Extension Moves Through All Five Responsibilities
Abstractions earn less than a single traced action. Back to INV-1047: extend the due date by 14 days, and notify the customer.
- Authority and policy. The gateway establishes the verified caller and the principal it acts for, then checks the tenant, access to INV-1047, the invoice’s current state, the extension limit, the time window, and whether this instance needs approval.
- Contract. The model proposes a narrow
extend_invoice_due_dateoperation: invoice ID, requested date, reason, expected resource version, and an idempotency key. The product validates every value server-side, because the arguments came from a model. - Durable record. Before anything mutates, the workflow writes a stable operation record carrying the caller, the request fingerprint, the policy and tool-contract versions in force, the approval state, and a status of pending.
- Contained execution. The invoice update runs as a conditional write against the expected version. The customer notification persists as a separate durable step, through a transactional outbox, so the two effects can succeed or fail independently and visibly.
- Evidence and verification. The workflow rereads the invoice from the system of record, preserves the before and after states with the authority applied, and returns a result that distinguishes
invoice_updated,notification_pending, andnotification_sent.
The record the on-call engineer opens when the customer asks what happened.
Operation op_9f27c1, extend_invoice_due_date on INV-1047, requested by an external agent for a verified finance principal.
operation record
Request fingerprint, policy v14, tool contract v3, approval granted and bound to these arguments, idempotency key held for replay.
invoice update committed
Conditional write against expected version 7 succeeded. New due date read back from the system of record and preserved beside the prior state.
notification unknown
Provider timed out after send. Delivery is unverified, so the step is neither failed nor complete until the provider message ID is queried.
retry rule
Only the notification retries, with the same idempotency key. The invoice update is never replayed; reversal happens only by explicit business rule.
What Happens When the Notification Times Out
Now the partial failure, because production lives there. The invoice update commits and the notification provider times out. The workflow treats delivery as unknown rather than failed, queries it by the operation or provider message ID, and retries only the notification with the same idempotency key. It never replays the invoice update.
If delivery stays unverified past policy, the workflow escalates to a person or takes a policy-approved fallback. It reverses the due date only where an explicit business rule calls for compensation. A system that cannot tell the two outcomes apart will either retry a transaction that already committed or claim success while the customer never hears.
Designing that fork is not an edge case. In the architecture reviews I have run this year, the question that most often exposes the gap is what happens when the tool call times out.
What this means for your build: the model is a small fraction of the system. Budget for the gateway, the workflow state, the verification reads, and the evidence store, because those are what turn a demonstration into a product.
That surrounding layer is the work Net Solutions’ Data & AI Engineering team does with product teams, from authorization and durable workflow state to tool contracts and the audit record. It also exposes a tempting shortcut: route every action through a person.
Why Approval Fatigue Is Not a Safety Architecture
Approval as the sole control transfers risk without creating supervision. Prompts pile up, the approver lacks the context to evaluate each one, and high-risk and low-risk actions look identical on screen. A person clicking yes on reflex carries the accountability while exercising none of the judgment.
The stronger design mixes three modes:
- Policy-bounded execution: Handles low-risk actions inside limits people set in advance.
- Meaningful approval: Covers the specific higher-risk actions where a person has real context and a real choice.
- Denial or escalation: Covers everything the system cannot safely decide.
Approval then means something, because it is rare. That balance holds only if the agent has proved its behavior before release.
How to Release and Measure a State-Changing Agent
Treat Evals as Release Gates, Not Demonstrations
Evals belong before release, functioning as gates rather than as demonstrations afterward. The suite that earns a state-changing agent its release covers representative tasks, authorization and policy-violation attempts, partial failures and timeouts, retry and duplicate-action behavior, recovery and compensation paths, and verification of end state.
Production tracing, monitoring, and audit capture run before the first real action, not after it.
Measure Cost per Verified Outcome, Not per Call
For a product-owned agent, the metric is cost per verified successful task, sitting alongside reliability, intervention, and safety measures rather than above them. For an external agent, the product measures what it can see: verified operation success and product-side cost per operation.
Either way, a cheaper model or a shorter prompt is not cheaper if it raises retries, interventions, or failed actions. Teams that track cost per call are optimizing the price of proposals. The product pays for outcomes.
Assistance Is a Feature; Execution Is an Architecture
You can implement assistance as a feature. State-changing execution must be engineered as part of the product’s architecture, with authority, durable state, safe contracts, containment, and evidence surrounding a model that only ever proposes. Build those five and the agent inherits the trustworthiness of the product around it. Skip them and the agent inherits everything else.
FAQs About AI Agent Architecture
Less than the labels suggest. Copilots can trigger actions and agents can be read-only. The dividing line that matters for engineering is whether model output can initiate a state-changing operation or an external side effect, or supply its parameters. A human approval may gate the commit; it does not take the model out of the transaction path.
No. MCP standardizes how clients discover and invoke tools, defines an optional transport-level authorization framework, and offers an optional Tasks extension for long-running operations with cooperative cancellation. It does not provide your product’s business policy, durable workflow runtime, transaction semantics, or recovery and compensation design. Those remain the product’s responsibility, whichever protocol sits at the boundary.
No. A request handler may run more than once, and the caller may still receive no response. Idempotency means retries with the same key and the same semantic request do not create additional intended business effects. After an ambiguous timeout, the workflow retrieves the operation outcome or reads authoritative state before deciding whether to retry, and a reused key with materially different arguments fails validation.
Approval is one control, not a safety design. When prompts are repetitive, low on context, or identical for high-risk and low-risk actions, approval transfers risk to a person without adding real supervision. A stronger design combines policy-bounded execution for low-risk actions with meaningful approval reserved for the specific actions that deserve it.
Identity, authorization, business policy, transaction semantics, and evidence, all at the product boundary. The customer may own the agent and its runtime, but the product decides whether each operation is permitted and executes it safely. Its authoritative record covers the caller, the request, the policy decision, and the resulting state, and it stops at the boundary of the runtime it does not operate.