Communicating Credit Risk with Large Language Models: Evaluation of Explanations from Standard and Alternative Data-Based Models

Details

Credit decisioning requires explainability for regulatory compliance, but SHAP values and GNN explanations are too technical for most stakeholders. This paper evaluates whether LLMs can serve as an explanation layer, translating model outputs into stakeholder-appropriate risk narratives using Freddie Mac single-family loan data.\n\nThree pipelines are compared: standard tabular (XGBoost + SHAP), network-based (GNN + GNNExplainer), and bimodal (combining both data types). Three LLM configurations are tested: small fine-tuned Gemma 3 4B, large fine-tuned DeepSeek R1 70B, and zero-shot Gemini 2.5. A human study with credit risk professionals and non-professionals evaluates explanations on eight decision-relevant dimensions.\n\nThree key findings: First, the pipeline accounts for more variance in explanation quality than the LLM choice — switching from Gemma to DeepSeek doesn't compensate for weak evidence representations. Second, LLMs reliably identify influential factors but fail to consistently state the direction of influence — positive or negative impact on creditworthiness. This direction failure can mislead adverse-action communication. Third, professionals apply stricter evidentiary standards than non-professionals, meaning domain-aligned LLMs matter more for expert audiences.\n\nThe direction-of-influence failure has concrete regulatory implications under fair lending rules, where adverse-action notices must accurately explain the specific reasons for credit denial.