Science has long begun with parsimonious models. A few parameters, each manually conceptualised, operationalised, and written down by a scientist for a reason, so that understanding came bundled with building. The preference had a practical backbone. Give a model too many parameters and it was expected to memorise its data rather than learn anything. Lately the habit has started to wobble. Enormously complex models now routinely match or beat simple ones at prediction, and models whose parameters far outnumber their data can settle into smooth, robust fits. The pattern is known as double descent, and a recent PNAS perspective asks whether Ockham's razor is losing its edge.

In much of practice the order of work has already turned around. Researchers begin with an opaque deep neural network that predicts well, and understanding becomes something to recover afterwards. So across neuroscience, physics, biochemistry, medicine, and economics, the same workflow now repeats. Train a deep network on a phenomenon until it predicts well. Run a post-hoc explanation method over it, such as SHAP, LIME, saliency maps, or concept attributions. Then read the explanation as insight into the phenomenon itself. The model relies on this feature, so nature must too.

Let's look closely at what each of those checks actually confirms. Reliability compares the model's outputs with the phenomenon's outcomes. When the record is good, you gain reason to trust the predictions, even though you cannot see how they are produced. Philosophers of science have defended just this kind of justification for opaque computation and for machine learning models in particular. What entitles the belief lives outside the model, in its history of agreement with the world.

Faithfulness compares the explanation with the model. A faithful explanation correctly reports which features the model relies on and what it computes. That is a statement about a description of the machinery, and it is settled entirely inside the model. The world never enters.

The scientist wants something different from either. They want reason to believe that a structure recovered from the model, perhaps a feature, a concept, or a counterfactual relation, corresponds to how the phenomenon is actually organised. One check looks at outputs. The other looks at a description of the machinery. Neither ever takes the phenomenon's structure as an input, and two checks that never look at a thing cannot settle anything about it.

The tempting reply is that the gap narrows as the model and the explanation improve, until a perfectly faithful account of a perfectly reliable model becomes, in effect, an account of the phenomenon. It does not, and the reason is structural rather than practical.

Tracking a phenomenon's outcomes does not require sharing its structure. Machine learning models often do not represent the world the way their targets are put together, and they lean on correlations far removed from the causal relationships responsible for the data. A model can predict everything and still work nothing like the thing it predicts. And since the true function behind a phenomenon is exactly what nobody knows, agreement with outcomes is the only test on offer, and it is a test a structurally wrong model can pass in full.

Sharpening faithfulness does not change this. A circuit-level account of a network is more informative than a saliency map, but it is more informative about the network.

Scientific practice quietly concedes the point. No researcher who trains a network and produces a clean feature attribution publishes, on that basis alone, a claim that the feature is mechanistically responsible for the phenomenon. They check the literature, compare against systems whose mechanisms are already known, and design a follow-up experiment. What looks like methodological caution is a tacit admission that reliability and faithfulness alone do not settle what a mechanistic claim requires. The missing premise, that model and phenomenon share structure, always arrives from outside the chain.

So what survives? The paper separates two ambitions. The strong reading treats the chain as establishing how nature actually works. If the model's computation runs through some feature, then nature must run through it too. That reading fails, and it fails at the limit of perfect reliability and faithfulness as firmly as in any real case.

The weak reading treats the chain as a source of candidates, possible structures worth investigating and hypotheses worth testing against the phenomenon. This reading survives, with a condition. A faithful description of a reliable model is, on its own, a description of an impressive instrument. Nothing inside it marks which recovered features bear on nature and which are artefacts of the training data. A description becomes a candidate hypothesis only when something external supports it, whether prior theory it fits, convergent evidence from systems already understood, or an experiment it could fail.

None of this makes explanation methods useless in science. It relocates their contribution. They are not the end of an inquiry but the beginning of one, generators of questions that the phenomenon itself must answer. Explanations propose. The world disposes.

Built with carefully drawn limits at socius: Experimental Intelligence Lab.