How to test an AI checker on alerts you've already closed
Any bank weighing a new control on AI decisions should see it work on its own data before it goes anywhere near production. The safest way is on decisions that have already been made.
Shadow mode on history
Pick a set of alerts your bank has already decided, by analysts, by an AI agent, or both. The outcomes are known and nothing about them can change. TAF runs alongside, inside your bank's environment, and puts each past decision in front of multiple independent AI models, which retrieve the case records themselves. Nothing it does reaches production. No alert is reopened, no customer is affected, and no queue changes.
What comes out
For each past decision, one of two results: every model agreed with the decision that was made, or they didn't. Where they didn't, the record shows where the models diverged. Your own reviewers then look at those cases and judge whether the divergence pointed to something real, such as a missed fact or a weak rationale, or whether the original decision was sound.
What you learn
- How often independent models disagree with decisions your bank has already made
- Whether those disagreements point to real problems
- How much review work the divergent cases would create
- Whether the records meet what your auditors and examiners would expect to see
These are measurements on your data. We don't promise a number in advance. A pilot is how you get one.
What stays with the bank
TAF runs in your estate. Your bank chooses the AI models on the panel, and holds every record and both signing keys. Divergence IP supplies the software and operates nothing. Case data travels from TAF to the models your bank chose and back. It never reaches us.
Setting it up
The main work is connecting TAF to the records a case needs, such as customer due diligence, transaction history and prior alerts, as they stood when each alert was decided, as far as your bank can reconstruct them. Your team chooses the sample: how many alerts, from which period, and which alert types.
Why start with history
A control you haven't measured is a promise. Running it on closed alerts gives risk, compliance and audit something concrete to review before anyone decides whether it belongs in front of live decisions.
Interested in a design-partner pilot? robert@divergenceip.com