Five money decisions, the same tests for every AI model.
Score on the three shared tests
- TUC engine (rules, no model)75.0% 71.6–78.2
Contents
We built an eval that asks AI models the narrow money decisions a bank’s software makes as money moves: whether a merchant is a gambling business, which charge a receipt belongs to, what a receipt email says was paid, which transaction refunds a purchase, and what kind of spending a merchant code is.
We score every model on a public set and on a private copy, and every score carries its 95% interval. An answer we cannot read counts as wrong.
What we found#
On our own four labeled tests, TUC’s rule-based engine, which uses no AI model, averaged 74.9% (95% interval 71.7% to 78.0%), measured October 28, 2026.
The five tests#
What each test asks
| Test | Question | Public / held out | Labels from | Scored as |
|---|---|---|---|---|
| Gambling-merchant check | Is this merchant a gambling business? | 210 / 210 | Ohio and New York regulators’ lists; ISO 18245 merchant codes | Accuracy |
| Receipt to card charge | Which card charge paid for this receipt, or none? | 300 / 300 | Synthetic; answer known by construction | Accuracy |
| Total from a receipt email | What total does this receipt email show, or none? | 130 / 130 | Synthetic; answer known by construction | Accuracy |
| Refund to purchase | Which transaction refunds this purchase, or none? | 175 / 175 | Synthetic; answer known by construction | Accuracy |
| Spend type (agreement) | Is this kind of spending essential, discretionary or neutral? | 284 / none | TUC’s own keyword rule, frozen before any run | Agreement, never accuracy |
One record from each test
Gambling check
merchant EPIC GAMES mcc 5816 label not gambling
Spend type
mcc 5411 description Grocery Stores, Supermarkets label essential
Receipt to charge
receipt Chipotle Mexican Grill, $49.06 charges 11 on the card label CHIPOTLE 93495, $58.80
Receipt total
email Thanks for your order! Subtotal: $55.13 Shipping: $0.00 Tax: $4.79 Order Total: $59.92 label 59.92
Refund
purchase LYFT *RIDE ATX, $50.79 charges 8 on the card label LYFT *RIDE ATX, −$12.04
Results by model#
Our engine, the reference row
| Model | Four tests | Three tests | Spend type | Held-out | Per 1,000 | Median wait |
|---|---|---|---|---|---|---|
| TUC engine (rules, no model) | 74.9%71.7–78.0 | 75.0%71.6–78.2 | 72.9%67.6–78.2 | 74.9%71.7–78.1 | No model calls | No network call |
Results by test#
Our engine, test by test
- Gambling-merchant check73.8%67.6–80.0Weakest: sweepstakes casinos written as payment descriptors, 4 of 26
- Receipt to card charge79.7%75.0–84.0Weakest: charges with a tip over 3%, 0 of 30
- Total from a receipt email74.6%66.9–81.5Weakest: “You paid” lines, 0 of 10
- Refund to purchase71.4%65.1–77.7Weakest: partial refunds of 15–40%, 0 of 25
- Spend type (agreement)72.9%67.6–78.2Weakest: codes our rule calls discretionary, 30 of 90
Held-out results#
We also score every model on a private copy of the tests, so a model that has seen the public tests cannot score well by memory.
On the private held-out copy our engine averaged 74.9% (71.7% to 78.1%), against 74.9% on the public tests.
The copy was built October 28, 2026 from a secret seed. Its fingerprint is 589003b8…, the full value under Method. It guards against memorizing records, not against a model that has learned the templates, and it does not prove a model was not trained on the public set.
What this does not measure#
- TUC built these tests and sells the engine scored here and the Understanding API.
- All test data is public-sourced or synthetic. No real customer transactions. Scores say nothing about a bank’s own data.
- Five narrow decisions, asked one at a time with a fixed prompt. Not a measure of general model ability.
- Spend type is agreement with TUC’s own rule, not accuracy.
- Not the best each model can do. Every model gets the same plain prompt and the lowest reasoning setting the AI Gateway offers for it, with no examples, tools or retrieval. Tuning the prompt per model would change the ranking.
- Not latency at a bank. Model latency is wall time from our test machine over the public internet. The engine’s timing is in process and is never compared with a round trip.
- Not the hosted Understanding API. When it is scored, its row runs the API’s decision layer in process, without our own network hop or authentication.
Who built it#
We did, so read every score with that in mind. The data, prompts, scoring code and every model’s raw answers on the public tests will be published so anyone can rerun or challenge them.
We chose the tests, the case types and the spend-type rule. The test sets include cases our engine fails on purpose, and its weakest case types are in Results by test.
The prompts were written before any model was run and are the same for every model. Our own rows carry the words “TUC’s own” wherever they appear, and color on this page marks them, not a leader.
Method#
Headline score
Mean of four accuracies, each task weighted equally: gambling-merchant check, receipt match, receipt total, refund pairing. Labels come from public sources or from how the synthetic record was built.
Decision-task score
Mean of three accuracies (gambling-merchant check, receipt match, refund pairing): the tasks every row, including decision models, can be asked. Use it to compare decision models with the rest.
Spend type
Spend type (essential, discretionary, neutral) is agreement with a rule TUC wrote, not accuracy against an outside standard, so it is shown on its own and never averaged in.
95% intervals
95% bootstrap intervals over test records (2,000 resamples). Each task has 130 to 300 records, so intervals are several points wide.
Ranks and ties
Rows share a rank when their intervals overlap; a rank is 1 plus the number of rows whose whole interval is above this row’s. Overlap is a conservative test: two rows reported as tied may still differ.
Held-out copy
Held-out scores come from a private copy of the tests built from a secret seed and never published, so a model that has seen the public tests cannot have memorized them. Spend type has no held-out copy. Its fingerprint: 589003b8704420a98aead6dc9727abe7a7e632d59fb2191ecda98551e56bd8cb
Same prompt, same settings
Every model gets the same prompt, the same records and the same scoring. An answer we cannot read counts as wrong. Each model runs at temperature 0 where the AI Gateway lists it and at the lowest reasoning setting the Gateway offers, with no examples, tools or retrieval.
Repeatability
The first 20 public records of every test are asked two more times; we count how many got the same answer all three times.
Cost and speed
US dollars per 1,000 decisions: the Gateway’s reported cost per call where it gives one, otherwise tokens times the Gateway’s list price on the date shown. Median and 95th-percentile wall time of one request from the test machine, network included. The TUC engine’s timings are in-process microseconds and are not comparable.
Why these decisions#
These are what a bank’s software decides as money moves. A bank sees money move, and at that moment it can warn before a fee, block a gambling transaction, or prompt and automate saving. Our paper A Small Lever projects what that is worth if the ten largest banks and the minority depository institutions ran fee alerts and a gambling block with a default saving program, at central values. About 88% of the effect comes from the saving program, a bank product; the alerts and the block add about 0.015 points, within sampling error of zero.
Black-to-White ratio of median household wealth in 2050
This eval is built from that paper’s benchmark: the same five test sets, now put to every model. Read A Small Lever
Rerun it#
# free: the whole pipeline against a local fake gateway node scripts/selftest.mjs # calls, tokens and dollars for a full run node estimate.mjs # every row, with a key from the environment AI_GATEWAY_API_KEY=$AI_GATEWAY_API_KEY node run.mjs
Where the numbers come from
- Engine scores, tests, definitions and caveats:
docs/eval/site-data.mjs, generated Oct 29, 2026. - List prices:
docs/eval/models.json. Expected run cost:docs/eval/estimate.json. - Record counts and example records: the public test sets in
docs/study-2/bench/datasets. Held-out fingerprint:docs/eval/heldout/manifest.json. - A Small Lever:
docs/study-2/model/results.jsonand the paper’s text.