Does this old lending rule still make sense, given who's actually applying now?
Credit decisions pile up history fast. A bank has old underwriting rules, model versions, policy changes, years of rejected and approved applications sitting around. Nobody ever really goes back and asks whether an old rule is still doing what it was built to do.
Here's the specific version of that problem I wanted to poke at: a rule that rejects anyone with less than 12 months of credit history. When it gets introduced, that's probably a genuinely good rule, short credit history correlates with real risk, mostly young, early-career people who are financially unstable in a fairly predictable way.
Nobody ever redefines the rule. What changes is who shows up with a short credit history. A few years later, "new to credit" might just as easily mean someone who moved to the UK recently, or a gig worker with steady income who never needed a credit card before. Same threshold, same rule, a completely different population of people getting caught by it.
I wanted to know: can you actually catch that, and can you build something that tells you not just "why was this person declined" but "does the rule that declined them still make sense"?
Four layers, and I built them in the order that made sense to me, not the order that sounds most impressive:
A decision model, trained honestly on only what a real bank would actually observe, approved loans and their real outcomes. Worth sitting with what that constraint actually does: the model never sees anyone the rule already filtered out, so it can't really learn anything about the credit-history feature for the population where it would matter most. That's not a bug I introduced, it's a real, known problem in credit risk called reject inference, and it shows up plainly here, credit_history_months comes out as basically the least important feature in the trained model. Not because it doesn't matter. Because the rule already removed everyone where it would have mattered before the model ever got a chance to learn from them.
An explanation engine, permutation importance and the model's own coefficients (no SHAP, no network access while building this, src/shap_upgrade.py has the real version), plus an actual counterfactual search, not a canned "reduce your debt" line, a real number, found by perturbing an applicant's affordability ratio and re-scoring until the decision flips.
Strategy memory, the actual point of the project. Tracks, per quarter, what share of the people the rule rejects would genuinely have defaulted if approved, versus what share would genuinely have repaid fine. Only possible here because this is synthetic data and I get to know the counterfactual, a real bank would need a deliberate holdout experiment, approving a small number of would-be-rejects on purpose, to see this at all.
A decision review, a report generator that answers "should we change this rule" using nothing but Strategy Memory's real numbers. No language model improvising something plausible, every sentence traces back to a computed statistic. src/llm_narrative_upgrade.py has real code for a local LLM to rewrite the phrasing for a memo, with the numbers themselves locked, it only ever gets to choose words, never facts.
There's a small Streamlit app (app.py) tying the first and third layers together, pick a rejected applicant and see what they'd actually be told, or look at the audit and see whether the rule is still earning its keep.
Pick a rejected applicant, see what they'd actually be told, and whether the rule and the fuller model agree on their case.
The audit view: the same rule-health chart from above, live and interactive, plus the decision review report generated straight from Strategy Memory's numbers.
The rule prevented 48% of defaults among the people it rejected in its first year. In the most recent year, that's down to 34%. Wrongful rejections went the other way, 52% up to 66%. Meanwhile the default rate among approved loans alone, which is genuinely all a real bank would ever see without a deliberate experiment, barely moved, 29% down to 28%. That flat grey line is the whole point. Standard monitoring would have looked at that number, shrugged, and moved on.
Here's why: among the people the rule rejects, the share who are genuinely high-risk (rather than just new to credit) drops from about 69% to about 22% over four years. The rule never changed. The population underneath it did.
Two real limitations, both flagged rather than hidden:
Reject inference is a real, unsolved problem here, not a footnote. The decision model was trained only on approved loans, so its view of "who's risky" is shaped by the very rule this project is auditing. A proper fix would need a genuine holdout experiment. I didn't build that, it's a project of its own.
The prevented-default and wrongful-rejection numbers above are only knowable because this is synthetic data with a documented ground truth. A real credit team auditing a real rule this way would need actual experimental data, not just this kind of retrospective analysis, to make the same claim with the same confidence.
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txtRun in order:
python src/simulate_lending_history.py
python src/decision_model.py
python src/explain_decisions.py
python src/strategy_memory.py
python src/decision_review.py
python src/make_report.pyThen, optionally, the app:
streamlit run app.pythin_file/
├── src/
│ ├── simulate_lending_history.py the aging rule, the shifting population
│ ├── decision_model.py Layer 1, trained honestly, reject-inference included
│ ├── explain_decisions.py Layer 2, permutation importance + real counterfactuals
│ ├── shap_upgrade.py the real SHAP version
│ ├── strategy_memory.py Layer 3, the actual audit
│ ├── decision_review.py Layer 4, the report generator
│ ├── llm_narrative_upgrade.py the real local-LLM version of the narrative
│ └── make_report.py the figures
├── app.py the Streamlit app
├── data/
├── outputs/
├── figures/
└── README.md
A genuine reject-inference model, using techniques built for exactly this problem, would be the honest next step, rather than training blind to the population the rule already removed. I'd also want to try this same audit on a rule that isn't a hard cutoff, something continuous, and see if the same kind of drift shows up more subtly, harder to catch, probably more realistic.
I expected this project to end with an underwriting model.
Instead, it ended with a question about the data the model would never get to see.
The moment that changed everything was looking at the feature importance and noticing that credit history length barely mattered. That made no sense. A rule that rejects everyone with less than 12 months of credit history shouldn't suddenly decide credit history is irrelevant.
The explanation turned out to be much simpler than the model.
It had only been trained on approved loans, because that's the only outcome a lender ever observes. Anyone rejected under the 12-month rule never enters the training data at all. By the time the model starts learning, those applicants have already disappeared.
So the feature wasn't unimportant. It had been hidden from the model by the very rule I was trying to audit.
I hadn't set out to build an example of reject inference. I only realised I'd run into it after trying to explain a result that looked wrong. That ended up being the most useful part of the project, because it changed what I thought the system was actually capable of telling me.
There was another moment that reinforced the same idea.
While testing the explanation layer, I opened one of the first rejected applications from the recent data just to check that everything looked sensible. It turned out to be someone the rule had declined immediately, even though the fuller model estimated a default probability of only 23%—well below what would normally justify a rejection.
I didn't have to search for an edge case. It was already there in an ordinary slice of the data, which made it far more convincing than a carefully chosen example.
The engineering lesson was different again. Getting the application running took longer than building the model because of a dependency conflict introduced by a Streamlit update. The code itself wasn't the problem. Two unrelated libraries disagreed with each other instead. It was a good reminder that a surprising amount of engineering time disappears into problems that have nothing to do with the interesting part of the work.
The project started as an audit of a lending rule.
It finished by showing me the limits of what that audit could honestly claim. If the people a rule has always excluded never appear in the data, then any conclusion about that population is, at best, incomplete. I think understanding that limitation is more valuable than pretending the model knows more than it does.
Python, pandas, numpy, scikit-learn (LogisticRegression, GradientBoostingClassifier, permutation_importance), matplotlib, Streamlit, joblib