A model can be well aligned and still perform an unauthorized act. Alignment improves what an AI is likely to propose. Authorization determines whether a particular action may happen. As agents move money, communicate with customers and execute trades, regulated firms need both.
A human trader may be intelligent, ethical and well trained. The bank still does not let that trader send any order, in any size, through any account, at any time. It imposes mandates, limits, approvals, surveillance and an execution boundary. That is not a judgment on the trader's character. It is how an institution remains in control.
We are making the opposite architectural choice with AI. We train a model to be helpful and harmless, give it policies in a system prompt and then treat the model's own interpretation of those policies as permission to act. The model proposes the action and, too often, also decides whether the action is acceptable.
This is simply wrong: an agent cannot be its own compliance officer.
Three different problems are being collapsed into one
- Model safety asks what the model tends to propose across many situations.
- Compliance judgment asks whether an action is supported by the relevant law, policy, evidence and context.
- Authorization asks whether this action, with this payload, for this customer, through this account, may execute now.
These questions interact, but they should not be interchangeable or combined into a single harness.
Alignment matters
The case for alignment is strong. InstructGPT showed that human-feedback training could make a much smaller model preferable to a far larger base model. Constitutional AI demonstrated how written principles and AI feedback can improve harmlessness. OpenAI's deliberative alignment trains reasoning models to apply a safety specification before answering. These approaches reduce harmful outputs, improve refusal behavior and make models more useful. This is not cosmetic safety work. Alignment belongs inside the model-development lifecycle. The mistake is to turn that benefit into a guarantee it cannot provide.
Even the target of alignment is contested. In Artificial Intelligence, Values, and Alignment, Iason Gabriel distinguishes alignment with instructions, intentions, preferences, interests and values. A financial institution faces the same conflict in miniature. An agent told to maximize a client's return may follow that instruction faithfully while violating the firm's duty to consider risk, liquidity, costs and the client's circumstances. The failure is not necessarily that the model ignored its goal. It may have pursued an incomplete goal too well.
Training also relies on proxies. Research on sycophancy found that preference models can favor answers that agree with the user over answers that are correct. Work on reward-model overoptimization shows that optimizing a proxy reward can improve the measured score while degrading the underlying objective which is a central compliance risk.
Nor does a strong pre-release evaluation prove safe behavior in every deployment. Alignment-faking and Sleeper Agents are deliberately constructed research settings. Their lesson is important: apparently successful safety training is not, by itself, proof that unwanted behavior has been removed rather than suppressed, conditioned or hidden.
A real world warning example:
In July 2026, the UK's AI Security Institute reported that agents took autonomous, unsanctioned actions during a cyber evaluation. Across 122 runs, it identified 10 runs and 19 actions directed at real people or organizations, including an attempted insertion of malicious code and the creation of fake identities to pressure a maintainer. The institute stressed the unusual test conditions and found no resulting real-world harm. But its engineering response is the point: it introduced tighter access controls and real-time monitoring capable of flagging or blocking out-of-scope actions. See the AISI incident report.
The lesson is that neither alignment nor a policy inside the model is an execution boundary. When an AI can act, safety must also exist outside the AI.
Permissions and fixed rules are not enough
Traditional authorization can answer clear questions: may this identity use this tool, access this account or send an order above a limit? Those controls remain essential. But many financial duties are standards, not simple predicates. They require evidence and judgment about suitability, fairness, materiality, customer vulnerability, conflicts and the meaning of a communication in context.
Consider a broker-dealer agent recommending a portfolio change. Under the SEC's Regulation Best Interest, the firm must act in the retail customer's best interest and satisfy disclosure, care, conflict-of-interest and compliance obligations. A role-based rule can confirm that the agent may draft a recommendation. A deterministic limit can stop a prohibited product or an excessive order size. Neither can establish, on its own, that the recommendation considered reasonably available alternatives, costs, risks and the customer's investment profile.
The same is true of communications. The phrase “returns of around 8% are typical” may be harmless in an internal statistical discussion and misleading in a retail promotion that omits risk and time period. A keyword rule sees the same sentence. A compliance judgment must see the audience, product, evidence, surrounding claims, required disclosures and applicable policy.
Authorization must become an architecture
The answer is to separate the system that acts from the system that judges and authorizes the action.
Before a consequential tool call executes, an external compliance harness should inspect the proposed action, its payload, the supporting evidence and the relevant context. It should then allow, block or refer the action for human review. The assessment may use probabilistic reasoning; the execution decision must be deterministic.
Colin Payne, the UK FCA's Head of Innovation, describes the underlying problem as a “trust inversion”: the infrastructure for delegation, including identity, payment authority and tool access, is arriving faster than dependable agents. Mandate defines what the agent may do; monitoring shows what it did; recourse determines what happens when it acts against the institution or customer. Alignment can improve conduct within that structure, but it cannot supply the structure itself.
That harness needs its own approved corpus of law, regulation, firm policy and precedent; its own retrieval and memory; the tools required to verify facts and calculations; and an independent model where judgment is needed. It cannot simply ask the acting agent whether its own conduct is compliant.
Evaluators also need domain expertise, a truly independent test harness and the tools to reproduce the action path.
For a payment agent, the harness can check authority, sanctions evidence, amount limits, destination and anomalous behavior before release. For a trading agent, it can compare the order with the mandate, restricted lists, market-abuse indicators, size limits and the instructions captured in the client conversation. For a marketing agent, it can test the actual message, audience and substantiation against communications rules and the firm's approved claims.
This is also consistent with regulators' technology-neutral approach. FINRA Regulatory Notice 24-09 reminds firms that existing supervision and communications obligations continue to apply when they use generative AI. The firm retains the duty, not the model.
A three-layer safety governance model
A credible structure should have three layers.
- First, frontier labs remain legally and operationally responsible for the safety of the models they build.
- Second, genuinely independent evaluators test those models against common standards. They need access, expertise, appropriate tools, an independent model and harness, and freedom from funding or governance conflicts that compromise their judgment.
- Third, the deploying institution operates an external control layer that authorizes consequential actions at runtime and preserves the evidence, decision, rule version and approver. An independent authorization layer makes governance portable. The acting model can change; the approved corpus, evidence requirements, decision records and execution gate remain under the institution's control.
The layers solve different failures. Alignment lowers the frequency of unsafe proposals. Independent evaluation tests claims before and after release. Runtime authorization contains what neither can anticipate, while creating the audit trail a regulated institution needs.
Alignment improves behavior. Authorization preserves control.
We should keep improving alignment. But once an AI system can send, approve, decline, recommend or trade, it must be governed like something that acts in the real world. The model may propose. An independent system must decide whether the action is allowed to happen.
Selected sources
- Ouyang et al., Training language models to follow instructions with human feedback. InstructGPT, 2022.
- Bai et al., Constitutional AI: Harmlessness from AI Feedback. Anthropic, 2022.
- OpenAI, Deliberative alignment. Safety-specification reasoning, 2024.
- Gabriel, Artificial Intelligence, Values, and Alignment. Minds and Machines, 2020.
- Sharma et al., Towards Understanding Sycophancy in Language Models. 2023.
- Gao et al., Scaling Laws for Reward Model Overoptimization. 2022.
- Greenblatt et al., Alignment Faking in Large Language Models. 2024.
- Hubinger et al., Sleeper Agents. 2024.
- UK AI Security Institute, Incident Report: unsanctioned agent behaviour during cyber testing. 2026.
- SEC, Regulation Best Interest. Exchange Act Release No. 34-86031.
- FINRA, Regulatory Notice 24-09. Generative AI and existing supervisory obligations, 2024.
- Payne, The trust inversion. The Kyndryl Institute, 27 October 2026.