(or Piantadosi and MLMs again (II)—continuation of this post)
In my critique of Prof. Piantadosi’s manuscript “Modern language models refute Chomsky’s approach to language,” I point out that regardless of the respective empirical results of Generative Linguistics and MLMs, the latter does not supersede the former because the two have fundamentally different goals. Generative Linguistics aims to provide a rational explanation of a natural phenomenon while MLM are designed to simulate human language use. Piantadosi does not dispute this, but rather states that
… there is an interesting debate about the nature of science lurking here. The critics’ position seems to be that in order for something to be a scientific theory, it must be intuitively comprehensible to us. I disagree because there are many phenomena in nature which probably will never admit ap37 of v7 (emphasis in original)
simple enough description for us to comprehend. We cannot just exclude these things from scientific inquiry.
Being one of the “critics” referred to here, I can grant the professor’s description of my position as basically accurate if a bit glib. But what is his position? He doesn’t say precisely, but we can make some inferences. In lieu of a clear statement of his position, for instance, Piantadosi follows the above quote with this:
There probably is no simple theory of a stock market (why IBM takes on a particular value) or dynamics in complex systems (why an O2 molecule hits a particular place on my eyeball). Certainly there are local, proximate causes (Tom Jones bid $142 for IBM; the O2 molecule was bumped by another), but when you start to trace these causes back into the complex system, you will quickly exceed our ability to understand the complex network of interactions.p37 of v7
These are slightly bizarre comments, as we do have comprehensible (i.e., simple) theories of stock markets—the efficient markets hypothesis, for instanceThis should not be taken as an endorsement of the efficient markets hypothesis—or any part of (neo)classical economics—as correct. A theory’s scientific-ness is no guarantee of its … Continue reading—and gases—the kinetic theory, for instance—which can give approximate predictions regarding real life events like the examples given. The professor’s view can be narrowed down slightly based onhis assertion that Rawski & Baumont (2023) “seem to misunderstand the linkage between experiment and theory” (p34 of v7)This is a bold claim for Piantadosi to make given that he is a psychologist, while Lucie Baumont—the latter half of Rawski & Baumont—is an empirical astrophysicist. when they state that “Explanatory power, not predictive adequacy, forms the core of physics and ultimately all modern science.” (Rawski & Baumont 2023) It would seem clear, then, that, for Piantadosi at least, that a “theory” is scientific only insofar as it has predictive power.
This may seem like a reasonable characterization—despite myriad insinuations to the contrary, virtually no one believes that predictive power is unimportant—but as soon as one attempts to develop that characterization things get dicey. What, for instance, is the required level of accuracy and precision for science? And What sort of things should a true science be able to predict? To use one of Piantadosi’s examples, individual molecules are the primitives of the kinetic theory of gasses, and the theory makes precise predictions about the behaviour of a gas—i.e., gas molecules in aggregate—but it is highly doubtful that it would make predictions about the actual motion of a particular molecule in any situation. Surely, this would be too much to ask of any theory of physics, yet Piantadosi seems to believe it is within the realm of scientific inquiry.
There’s also a question of what it means to “predict” something. Piantadosi’s argument boils down to “MLMs are better than Chomsky’s
approach theory, because they make more correct predictions,” yet nowhere does he explicitly say what those predictions are, nor does he document any tests of those predictions. Instead, we are treated to his prompts to a chatbot followed by the chatbot’s response. Perhaps these are the predictions. Perhaps they predict how a human would respond to such prompts. If so, then so much the worse for MLMs qua scientific theories because, even if MLMs were indistinguishable from humans, the odds of any two humans answering a single question the same way is vanishingly slim, and any way to determine a general similarity between utterances would almost certainly be either arbitrary or dependent on some theoretical framework. At best, MLMs simulate human language use, meaning they no more predict facts of language than a compass predicts facts of geometry.
approach to theories of language, on the other hand makes clear predictions if one bothers to engage with it. The predictions are of the form “Given theoretical statement T, a competent speaker of language L will judge expression S as (un)acceptable in context C.” This is exactly the sort of prediction that one finds in other sciences—”if one performs precisely this action under precisely these conditions, one will observe precisely this reaction”—and the sort of prediction that is absent in Piantadosi’s paper.
Indeed these predictions seem to be absent in the entire contemporary “AI” discourse, and with good reason—”AI” is not a scientific enterprise. It’s an engineering project. A fact that is immediately obvious when one considers how it measures success—against a battery of predetermined arbitrary tests. MLM researchers, then, aren’t discovering truths, they’re building tools to spec, like good engineers.
This is not to cast aspersions on engineers, but it does raise a question—the core question: How exactly can an engineering project like MLMs refute a scientific theory like Generative Grammar?
|This should not be taken as an endorsement of the efficient markets hypothesis—or any part of (neo)classical economics—as correct. A theory’s scientific-ness is no guarantee of its correctness.
|This is a bold claim for Piantadosi to make given that he is a psychologist, while Lucie Baumont—the latter half of Rawski & Baumont—is an empirical astrophysicist.