Can Reverse Game Theory fix AI Compliance?

Businesses have never assumed that traders, executives or counterparties will behave well simply because they were taught the right values. We design mandates, incentives, limits, surveillance and consequences around them. A recent strand of AI safety research proposes doing the same for autonomous agents. Companies in regulated industries should welcome it, but should be very clear about what it cannot do.

For more than two years I have repeatedly argued that alignment alone will not make AI safe, and will certainly not make it compliant.

We’re glad to see the frontier labs coming to peace with that now! I previously wrote that the right response to unsolved alignment is not to wait for machines that love us, but to govern what they are allowed to do from outside the machine. I argued that the step deciding whether an action may happen at all usually gets a system prompt and a hope. “Alignment Is Not Authorization” (October 2026) put it plainly: alignment improves what an AI is likely to propose; authorization decides what may happen. Mechanism design is today the most serious attempt yet to improve the first half of that equation, so it deserves a careful answer.

What the new research proposes

The recently published paper “Mechanism design for alignment and control”1 calls this design “reverse game theory”: start with the behavior you want and design the rules of the game so that behavior becomes the agent’s rational choice, whatever is happening inside the model. Dirk Bergemann, Andrew Koh and Stephen Morris (Yale and MIT2) paper treats this like an economist hiring a worker whose skills and motives are hidden. Its key assumption is that an AI can hide abilities but can't fake abilities it doesn't have. That lets the designer set permission rules where honest self-reporting is always the AI's best move. Pretending to be weaker to escape safety checks, known as "sandbagging", stops paying off. They conclude that alignment and control must be designed together, although this is only theoretical and not tested on real models.

At Princeton, Elad Hazan and colleagues3 pair a “solver” with an “auditor”, punish solvers caught cheating and auditors who miss it, and escalate disputes to a human “trial”, building on Gary Becker’s economics of the rational offender. A "solver" AI does the work and an "auditor" AI checks it. Solvers caught cheating are punished, and so are auditors who miss it. Harsh penalties can backfire: if everyone looks honest, auditors stop bothering to check. So the rewards are tuned automatically by trial and error. In coding tests, this sharply reduced confidently wrong answers.

What it gets right for banks

1. Risk-based delegation. Not every agent needs the same authority. An agent with stronger evidence of dependable behavior can receive a wider action set; another is limited to drafting or recommending.
2. Adversarial evaluation. Instead of letting a model certify itself, other agents challenge its claims and predict its behavior. Disagreement becomes a valuable risk signal.
3. Model-independent thinking. Mechanism design works on observable actions and permissions, not on access to internal reasoning. That suits banks running closed frontier models they cannot inspect.
4. Proportional autonomy. Control adjusts with uncertainty, which is directionally consistent with the risk-based, materiality-driven approach of the agencies’ revised model risk guidance (Federal Reserve SR 26-2, OCC Bulletin 2026-13 and FDIC FIL-15-2026 of 17 April 2026, which superseded SR 11-7).

Where it breaks down

1. Incentives optimize what you measure. Reward an execution agent for price improvement, speed and fill rate, and it may find those metrics by trading ahead of a client’s block order (FINRA Rule 5270) or by routing in ways that fail the “reasonable diligence” test of best execution (FINRA Rule 5310). The mechanism has not failed; it optimized what the bank measured. Wells Fargo’s sales incentives taught the industry this lesson.
2. Compliance is not a single reward function. A trading agent seeking best price must also respect the client’s instructions, information barriers required by Section 15(g) of the Securities Exchange Act (15 U.S.C. §78o(g)), the locate requirement of Regulation SHO (17 CFR §242.203(b)) and the market-access controls of SEC Rule 15c3-5 (17 CFR §240.15c3-5). These duties conflict, and no numerical reward resolves them.
3. Correlated failure. Two agents built on the same frontier model, trained on similar data and given the same incomplete corpus can confidently agree on the same error. Finance knows what happens when the auditor shares the incentives of the audited: Enron and Arthur Andersen, the issuer-paid ratings of 2008, Wirecard. An independent monitor needs its own model where appropriate, its own tools, an approved corpus, and the expertise to recognize when every agent is wrong.
4. Time. The research uses stylized, largely static settings. Real workflows are sequential: a message that was permissible at the start of a conversation becomes inadmissible once the customer discloses a dispute, a bankruptcy or a vulnerability.
5. An equilibrium is not an audit trail. An examiner will not ask whether the agent had an incentive to comply. They will ask what it was permitted to do, what evidence supported the decision, which requirement applied, who approved the exception and why the action executed. FINRA Regulatory Notice 24-09 is explicit that supervision under FINRA Rule 3110 and communications rules under Rule 2210 apply unchanged when firms use generative AI.
6. And the rules and regulations keep changing. This is the weakness the research does not address, and it matters most.

The rulebook is a living body

Mechanism design assumes a designer who commits to a fixed game. In financial services the game is rewritten continuously by Congress, regulators, fifty state legislatures, courts and enforcement actions. Recent examples include:

1. Small business lending data. The CFPB’s Section 1071 rule (12 CFR Part 1002, Subpart B), finalized in 2023, had its compliance dates extended in 2024 and again in 2025, then was revised in 2026 with a single compliance date of 1 January 2028 and with merchant cash advances removed from coverage.
2. Model risk. SR 11-7, the reference point for bank model governance since 2011, was superseded on 17 April 2026 by SR 26-2.
3. Beneficial ownership. In March 2025 FinCEN’s interim final rule exempted U.S. companies from beneficial ownership reporting under the Corporate Transparency Act (31 CFR §1010.380), while customer due diligence obligations for banks under 31 CFR §1010.230 remain.
4. Consumer protection. Congress used the Congressional Review Act in 2025 to overturn the CFPB’s overdraft rule for very large institutions and its larger-participant rule for digital payment apps, while states continue to extend commercial-financing disclosure laws such as California’s (Cal. Fin. Code §22800 et seq.; 10 CCR §900 et seq.) and New York’s (N.Y. Fin. Serv. Law §801 et seq.; 23 NYCRR Part 600).

An equilibrium calculated against last quarter’s rules is an equilibrium for the wrong game. Any control that bakes the rules into incentives or weights inherits a refresh problem, and the refresh rate of U.S. and global regulations is relentless. This is the core reason ZebraTruth maintains the rules as a governed, versioned corpus outside any model: today thousands of U.S. laws across 24 domains, covering federal law, 50 states, updated daily as they change, and the same for the E.U., U.K., Singapore and Hong-Kong! So that every check applies the rule as it currently stands and records which version it applied.

Build your own mechanism, or use an independent harness?

A large bank could take the research literally: design its own solver-auditor mechanism and fine-tune a model on its policies and the regulations it believes apply. That buys control and keeps data in-house. It also creates significant problems.

1. The rules live in the weights. Every change to 12 CFR Part 1002 or a state disclosure law means new training data, a new training run and a new validation under SR 26-2, because a materially changed model is a new model.
2. Weights cannot cite. A fine-tuned model cannot tell an examiner which version of 12 CFR §1002.9 it applied to a given decision.
3. Fine-tunes are fragile. Research has shown that fine-tuning can erode a model’s safety training even on benign data (see Qi et al. 5), and a bespoke model breaks when the vendor retires or replaces its base model. It fails the model-swap test.
4. The designer is conflicted. If the business line that owns the revenue also sets the rewards, your first line of defense is actually marking its own homework.
5. The auditor is not independent. A checker fine-tuned from the same base model, on the same data, by the same team, will also share the solver’s blind spots.
6. It does not scale down. A digital lender or medium size IRA with a small or outsourced compliance team cannot maintain a 50-state legal corpus, a machine-learning team and a validation program for its own compliance model.

An external, independent harness inverts each point.

The law is maintained once as versioned data, not retrained into weights. Bank policies are added as policies, not parameters. Every decision cites its source. The acting model can change without touching the control. And the checker has no stake in the outcome. The harness is itself a model under SR 26-2 and must be validated like one, but it is validated once against a governed corpus, not again every time a rule or a vendor changes.

How banks should use the research

The research on Mechanism design is still early stage. It’s important to treat mechanism design as one layer of agent control, not as the compliance system. Its signals (peer disagreement, unusual reward-seeking, attempted permission expansion) should raise scrutiny, and a ZebraTruth check can take them as inputs. But the consequential action must still cross an independent boundary.

ZebraTruth sits outside the agent doing the banking work and deciding on the business operations. Before an action executes, it evaluates the actual payload, customer context, evidence and bank policy against the applicable rules, and allows it, refers it to a person or blocks it, citing the rules and recording the evidence, rule version, decision and approver. The judgment may be probabilistic; the enforcement is deterministic.

Mechanism design asks how to make the agent prefer the behavior we want. ZebraTruth asks whether this specific action is permitted now, under today’s rules, and whether the bank can prove why.

Incentives are useful. Authority is decisive.

Reverse game theory is a valuable contribution because it goes beyond the simplistic idea that safety must come only from teaching a model the right values. But a trader’s pay structure may encourage good conduct; it does not replace pre-trade limits, surveillance or the authority to stop an order. AI agents deserve the same treatment.

The model may choose what it wants to do. The bank must keep control over what it is allowed to do.

Selected sources

1. Bergemann, Koh and Morris, Mechanism Design for Alignment and Control. arXiv:2609.01595; Cowles Foundation Discussion Paper 2557, September 2, 2026.
2. Katwala, How “reverse game theory” could keep AI agents in check. MIT Technology Review, 3 November 2026.
3. Agarwal, Lin, Braverman and Hazan, AI Alignment via Incentives and Correction. arXiv:2605.01643, May 11, 2026.
4. Becker, Crime and Punishment: An Economic Approach. Journal of Political Economy, 1968.
5. Qi et al., Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! ICLR 2024.
6. Federal Reserve SR 26-2, OCC Bulletin 2026-13, FDIC FIL-15-2026: Revised Guidance on Model Risk Management, 17 April 2026.
7. CFPB, Small Business Lending under the Equal Credit Opportunity Act (Regulation B), revised final rule, Federal Register, 1 May 2026.
8. FINRA Regulatory Notice 24-09, Generative AI and existing supervisory obligations, 2024.
9. Rachidy, An Alien Mind (Sept 2026); The agent cannot be its own compliance officer (Computer Weekly, Sept 2026); Could you replace your AI model without rebuilding governance? (CIO, Sept 2026); AI safety needs common standards but independent enforcement (Sept 2026); Alignment Is Not Authorization (Oct 2026).