AI GOVERNANCE / SECURE AGENTIC SYSTEMS

Agentic AI governance: From policy to productionE

Four disciplines for building secure, reliable enterprise AI agents.

By Adnan Masood, PhD

Consider an AI agent that issues the correct refund but bypasses the required approval. Accuracy tells us whether the amount was right. Governance asks whether the agent had authority to act, whether it protected customer data, and who is accountable when something goes wrong.

The enterprise challenge is to carry that accountability through development, release and daily operation, wherever an agent runs.

DIVIDER

What AI governance actually means

AI governance is how an organization assigns responsibility for AI, sets acceptable boundaries, and verifies that those boundaries are respected. It includes security, privacy, fairness, reliability, and human oversight.

Different teams see different parts. Business leaders need ownership and risk visibility. Compliance teams need requirements and assessments. Developers need implementable controls. Auditors need evidence. Security teams need ways to prevent and contain harm.

Agentic AI governance extends this responsibility to systems that choose tools, take actions and adapt across multiple steps. It requires checking both the quality of the result and the authority behind each consequential action.

I organize the work into four connected disciplines: business governance, engineering controls, evaluation and adversarial testing. Each produces evidence the others need.

DIVIDER

1. Establish ownership and boundaries

Start with an inventory covering internally built agents and those embedded in purchased software. Record each agent’s purpose, business owner, deployment location, data access, available tools, and permitted autonomy. Require registration at deployment and track delegated agents.

Then assess consequences. An assistant drafting an internal summary should not face the same approval process as an agent changing customer records. Decide which actions may run automatically, which require review and which are prohibited. For decisions affecting people, establish a route to explanation and human review.

The practical outputs are an agent register, risk assessment, control requirements and an approval record. Map applicable obligations to named controls and owners. NIST’s AI Risk Management Framework and ISO/IEC 42001 provide useful foundations; legal and compliance teams still need to determine the obligations applicable to each deployment.

DIVIDER

2. Turn policy into enforceable controls

“Do not disclose customer data” is a requirement, not an implementation.

Give each agent an identity and only the permissions it needs for its task. Limit access to data, tools, and destinations. Keep credentials outside model context. Use isolation and data-protection controls alongside content filters.

Enforce consequential decisions outside the model. Before a tool executes a refund, a separate authorization check should validate the customer, amount and required approval. Human approval should bind to the proposed action. When agents delegate work, validate the receiving agent’s authority rather than automatically passing along broader permissions.

Robustness also requires ordinary engineering discipline: bound retries, prevent duplicate transactions, and define a safe fallback when dependencies fail. Impose execution and spending limits so an agent cannot keep retrying indefinitely. Review the tools and dependencies it can access, including their provenance and approved versions.

The outputs are deployed controls, reviewed configurations, approval workflows and tested recovery procedures. Engineering teams should be able to show where each requirement is enforced.

DIVIDER

3. Evaluate the application, not just the model

A model benchmark cannot establish whether an enterprise workflow works. Test the assembled system: instructions, retrieval, tools, permissions, and interactions with other agents. Verify the resulting business state, beyond the agent’s claim that it completed the task.

Build evaluations from realistic tasks and known failures. Include missing information, unavailable services, ambiguous requests and repeated execution. Measure task completion alongside unauthorized actions, data exposure, appropriate escalation, latency and cost per successful task. Evaluate fairness where outcomes affect people.

Repeat important tests because agent behavior can vary. Use deterministic checks where possible and calibrate model-based judges against expert human review. Define release thresholds based on business risk; a high average score should not hide a critical control failure.

Outputs include versioned test suites, scorecards, failure records, and evidence-backed release decisions. Rerun affected evaluations when models, prompts, tools, or policies change.

DIVIDER

4. Test how the system can be abused

Evaluation asks whether an agent performs its intended job. Red teaming asks how someone could redirect it or misuse its capabilities.

Test prompt injection through documents and tool responses, unauthorized access, poisoned memory, malicious tool behavior, and failures spreading between agents. OWASP’s guidance for agentic applications provides a useful threat model, including identity abuse, tool misuse, and cascading failures.

Findings should identify the affected control, severity, reproducible conditions, and remediation owner. Verify each fix and convert relevant attacks into regression tests. Security testing should leave the application measurably better protected.

DIVIDER

Keep the evidence connected after launch

Link each requirement to its owner, implementation, test and production evidence. For the refund agent, that means connecting the approval rule to the authorization check, the test that attempts to bypass it, and the transaction record showing what happened.

Record agent identities, action requests, authorization decisions and outcomes in protected logs. Operators need authority to pause agents, revoke access and recover safely.

For leadership, I would track the proportion of required controls with current evidence of effectiveness, unresolved high-severity findings, and time to contain incidents. Separate documented controls from those demonstrated in operation.

Use existing governance, identity, cloud, observability and security tools where they meet the requirement. Connect them through consistent identifiers and evidence exchange. A central view can support oversight while enforcement happens where actions execute. This is an architectural principle, not a requirement to buy one all-encompassing platform.

DIVIDER

What frontier labs and implementation partners contribute

Frontier-lab guidance reinforces the importance of system design. Anthropic recommends starting with the simplest adequate architecture and hardening agents through isolation and least privilege. OpenAI’s guidance similarly emphasizes tool approvals, evaluations and layered safeguards. Better models help; enterprises remain responsible for their deployments.

UST’s partnership with Anthropic brings frontier-model expertise together with enterprise engineering and industry delivery. The announced collaboration includes a commitment to train 20,000 associates and emphasizes human oversight and audit controls in operational systems.

The approach I advocate at UST connects business governance, implementation, evaluation, and security. For clients, the value should be practical: controls adapted to their risks and existing environment, with evidence that those controls work.

Start with a bounded workflow, establish its evidence, and expand autonomy as results justify it. Train engineers and reviewers alongside deployment. Client maturity should determine the adoption path, while the safeguards reflect the consequences of failure.

An enterprise agent is governed when its authority is bounded, its actions are checked, and someone remains accountable for the result.

Move agentic AI from pilot to production.