I very much enjoyed Andrew Gelman's post "Bayesian statistical pragmatism" (4/15/2011) on his blog Statistical Modeling, Causal Inference, and Social Science. And one aspect of that post struck me as especially relevant to some recent LL discussions:
I am surprised to see Kass write that scientists believe that the theoretical and real worlds are aligned. It is from acknowledging the discrepancies between these worlds that we can (a) feel free to make assumptions without being paralyzed by fear of making mistakes, and (b) feel free to check the fit of our models (those hypothesis tests again! Although I prefer graphical model checks, supplanted by p-values as necessary). All models are false, etc.
I assume that Kass is using the word "aligned" in a loose sense, to imply that scientists believe that their models are appropriate to reality even if not fully correct. But I would not even want to go that far. Often in my own applied work I have used models that have clear flaws, models that are at best "phenomenological" in the sense of fitting the data rather than corresponding to underlying processes of interest–and often such models don't fit the data so well either. But these models can still be useful: they are still a part of statistics and even a part of science (to the extent that science includes data collection and description as well as deep theories).
I wrote the passage above nearly two months ago, in the middle of April, as the beginning of a post that I never finished. The "recent LL discussions" that I had in mind then were "Word order 'universals' are lineage-specific?", 4/15/2011, and "Phonemic diversity decays 'out of Africa'?", 4/16/2011; but Gelman's post also obviously applies to "Norvig channels Shannon contra Chomsky", 5/31/2011, and "Straw men and bee science", 6/4/2011.
And all of this reminds me of something I wrote seven years ago about an earlier controversy in computational phylogeny — "More on Gray and Atkinson", 4/28/2004:
In thinking about the general problem, an analogy with physics may be helpful. If we assume that the sun, planets and other heavenly bodies are point masses in calculating their orbital dynamics, our model is obviously false to fact. But does this simplification invalidate our conclusions? Well, it might or might not, depending on what calculations we do and what conclusions we want to draw. Any model of orbital dynamics will be simplified — and therefore false — to one extent or another. The question is whether this matters with respect to some specific quantitative or qualitative prediction. Giving a correct answer to that question requires a mixture of detailed mathematical reasoning, relevant empirical testing and luck.
One of Russell Gray's slides made this point by quoting the well-known scientific proverb that "A model is a lie that leads us to the truth". I believe that this was originally adapted (by whom?) from a remark made by Picasso:
"We all know that art is not truth. Art is a lie that makes us realize truth, at least the truth that is given us to understand. The artist must know the manner whereby to convince others of the truthfulness of his lies." (The Arts, Picasso Speaks, 1923)
Yesterday Russell Gray made considerable headway in convincing me of the validity of his approach. His talk, and the discussion around it, clarified for me the nature of the simplifying assumptions that he's making, and the (empirical and logical) questions to be addressed in determining whether those simplifications invalidate his conclusions about the dating of Indo-European. He also convinced me that he's continuing a serious program of efforts to test the effects of his assumptions, and that he's serious about understanding and addressing objections. In other words, he's doing science.