Amortized Nonlinear Variable Selection with Prior-Data Fitted Networks

| Source: arXiv AI

Tags: MambaPFN, prior-data fitted networks, variable selection, Markov blanket, Transformer

MambaPFN uses a pretrained Transformer to pick relevant variables in new datasets without retraining, and on a Facebook comments dataset beat the best baselines by about 20% while selecting the fewest variables.

Details

Model-agnostic variable selection helps interpret complex systems in biology, business and sociology, especially when relationships are nonlinear. MambaPFN (Model-Agnostic Markov BlAnket selection with Prior-data Fitted Networks) treats a Transformer as a learned mapping from a dataset to its relevant variables. It adds a scalable scheme for generating pretraining data, an objective targeting each variable's posterior inclusion probability, and a Transformer architecture modified for variable selection. Once pretrained it applies to new datasets with no parameter updates. The authors report strong selection in simulations and a scaling law: more pretraining datasets and larger models keep improving results. On a Facebook comments dataset it selected the fewest variables with the best efficiency and predictive performance, about 20% better than the best baselines.