LLMs don’t know anything about anything and they simply can’t

In 2024, The California legislature passed SB-1047—a bill intended to reduce so-called catastrophic risk of AI. The companies that make and market LLMs, being private capitalist corporations, hated it—likely because it sought to impose real accountability and real transparency on them—and successfully lobbied Governor Gavin Newsom to veto it. Then last year, the legislature passed AB-1064—a much more modest bill that would require any company that marketed a chatbot to children to ensure that the chatbot “is not foreseeably capable of doing certain things that could harm a child, including encouraging the child to engage in self-harm, suicidal ideation, violence, consumption of drugs or alcohol, or disordered eating.” AI companies lobbied against it and Newsom vetoed it.

Why would a company be so resistant to this minimal regulation, especially when it was passed in the wake of Adam Raines’ death by suicide which seems to have been encouraged by ChatGPT? I don’t think it’s just a knee-jerk reaction against any regulation, because doing so in this case amounts to telling the public “we will not ensure that our product is safe for your children”—bad PR in an industry that has and needs excellent PR.

No, the reason is far simpler. The companies are resistant because they know that they could clear even the low bar that the passed bill sets forth. They know that they can’t make chatbots that adhere to any standard of behaviour, because in order to adhere to a standard you need to know what that standard is. But LLMs can’t know anything and their creators must know this at least subconsciously. And what’s worse, we all should have known this because it was spelled out in what might be considered the founding work of modern philosophy the 1781 tome The Critique of Pure Reason by Immanuel Kant.

Kant was not some futurist who predicted LLMs. Rather he was addressing an active philosophical debate of his time namely a debate between Empiricists like Hume and Locke and Rationalists like Decartes and Leibniz around the problem of objective knowledge.^[1]Disclaimer: I’m not a Kant scholar. I’m a theoretical linguist who challenged himself to reading Critique. Based on secondary material, namely a helpful series of videos, I know that Kant … Continue reading The problem was that we are able to have objective knowledge—i.e. knowledge of objects—that must come from experience, yet the sense-data which is the source of that experience is insufficient for such objective knowledge. To give an example, as I write this I am sitting in a cafe at a table and there is an empty ceramic mug to the right of my laptop. In order to make such a descriptive judgment, I need objective knowledge of myself, the cafe, the table, the mug, an my laptop to say the least, and I have that knowledge because my senses tell me so. But what do my senses provide me?

In a given instant each of the millions and millions of your sense-receptor cells is sending a bit of information to your nervous system, reporting whether and it what way it is being stimulated.
And each subsequent moment those cells are reporting something new. The world as our senses report it is an ever-changing matrix of atomic bits of data, what Kant calls the manifold, yet we consciously experience it as consisting of unified objects which persevere in time and interact with each other in lawful ways. These are two seemingly incompatible descriptions of the world yet, as the debate between empiricists and rationalists shows, both seem roughly correct.

So, does the world consist of a manifold of appearances constantly in flux or a unified lawful domain of permanent objects? Kant answers “yes.” For Kant, the manifold of appearances and the unity of our experience reflect different aspects of the world, both of which are real.

But how can we take appearances from our senses and extract objects from them? We must carry with us the tools to do so, to go from sense to understanding. For Kant these tools are the very notions of space and time, and The Categories—a set of minimal concepts that we use to construct the concepts we use to identify objects. Without these, experience is impossible, and without experience, knowledge is impossible.

An important thing to note here is that these tools, being necessary for experience, cannot be given in experience. Take, for instance, our sense of space. Part of how I can recognize that myself, the table, the mug, and the laptop are distinct objects is because they, or perhaps more accurately their sensible appearances, occupy distinct regions of space. Restricting myself to the visual appearance of the mug and oversimplifying—there are is a set of photo-receptor cells that occupy a contiguous region in my retina, and each of those cells is reporting being stimulated by whitish light.The information that cells x, y, z, etc are being stimulated is certainly data we did not have before, but the fact that those cells represent a contiguous region must be something I “knew” beforehand, if only because the position of a photo-receptor in a retina is given by my very biology. Likewise, when we consider time, we find that our judgements of time cannot be given by external data, but have to come from our own internal sense of our sensory data being constantly updated.

The Large Language Models which are what we usually mean when we talk about AI these day do not have in-built categories nor nor do they have any sense of space and time. They simply have inputs and math that is performed on those inputs. Lacking the tools necessary for experience, they can’t have any and being unable to experience anything can’t know anything. And therein lies the impossible bind that any regulation would place AI companies in.

Can we build an LLM that will not encourage children to harm themselves or others? In order to do this, an LLM needs to be able to make a judgement that a statement is an encouragement to self-harm and such a judgement requires knowledge of objective concepts of “harm,” “self,” and even “statement” and an LLM cannot have such knowledge. And the same reasoning explains why we can’t build an LLM that will not generate CSAM, that will not cause psychosis, and so on.

Such a conclusion likely strikes folks as a bit far-fetch, perhaps because we are used to websites being able to filter out certain types of content by default—Google, for instance, doesn’t surface NSFW pages unless the user turns off safe search, and doesn’t surface illegal pages at all. Why then, can’t we do so with LLMs? If it’s because LLMs can’t know anything, does that mean I’m saying that Google’s search algorithm can know things? Certainly not, but Search engines and other traditional websites are based on a database and as such can at least represent knowledge and people can build algorithms that use that represented knowledge.

Suppose you are building a search engine. You’ve crawled the entire web and placed every page into a database including some metadata that you deem useful in finding the best pages.
You’ve also created an algorithm for analyzing a query and ranking all your pages by how well they match the query. You open it up to the public and it works! Users are generally happy with the results, but you start to get complaints that unwanted results are showing up—porn, hate sites, depictions of violence, or the like. The solution is relatively simple—add some metadata to your database that indicates whether each page belongs to a particular category of potentially unwanted results and adjust your algorithm to check for this metadata before serving these results.

The devil is in the details, of course, in that as simple as it is to add the relevant metadata to a database, correctly and reliably categorizing each page accordingly is thorny to say the least. It brings in a number of grey areas, depends on value judgments, and is difficult to scale so it is error-prone, but it is possible in principle because humans directly encode the possibility of representing objective knowledge into the software. An LLM, though, does not store responses or pages, or facts in a database that can be directly modified—LLMs can’t even represent knowledge.

This distinction between LLMs and classic web apps cannot be lost on any competent engineer that would be tasked with building “guardrails” for chatbots. So, it is very likely that companies that build chatbots know that we cannot build those guardrails yet even without familiarity with 18th century philosophy. But given that critical philosophy/thinking seems to be anathema to the ever more reactionary tech sector, they likely don’t know, or aren’t allowed to admit, that they will never be able to build such guardrails.

It’s no surprise, then, that LLM makers would put so much lobbying effort and, by extension, money into fighting even the most reasonable regulations like “don’t build a chatbot that will encourage child suicide,” because it would force them to admit that they cannot possibly clear so low a bar.

But as bad as it is that LLMs cannot be restrained from being positively harmful, the impossibility of them having objective knowledge is a death knell for any hopes for their benefits. If you asked a normal person what they would hope an AI or a Robot would do for them, likely they’d want a machine to do chores around the house, work dangerous jobs, and give them more time “to knit sweaters, play with their dogs, start a garage band, experiment with new recipes, or sit in cafés arguing about politics, and gossiping about their friends’ complex polyamorous love affairs” (to borrow a formulation from the late David Graeber). LLMs, however, will never be able to do this even if they are given physical bodies. Because how can you trust an AI to wash a dish if it cannot even conceive of the possibility of a dish, let alone know what a dish is?

But, of course, the billionaires who sell, fund, and promote LLMs don’t have to do household chores, or dangerous jobs, so they don’t care if their AIs can do those tasks. Instead, they are promoting LLMs ability to write, to code, to make images, and so on. On their face, these might seem like possibilities as an LLM does nothing but generate formally correct outputs—they can string together words into grammatically acceptable sentences, or symbols into syntactically correct Python expressions. But anything we create of any value, be it text, or code, or images, has not only form, but content and function, both of which implicate objective knowledge. To write well, you need to create well-formed sentences, but you also need to choose the right words based on knowledge of their semantics. To write a good program, you need to know what you want it to do—how you want it to affect the world. You need knowledge of the world that LLMs cannot have.

This, of course, hasn’t stopped them from degrading our working conditions and firing us. Your livelihood has no doubt been negatively affected by LLMs. Brian Merchant has been gathering stories of the real life effects of LLMs. And as Cory Doctorow puts it “even though an AI can’t do your job, an AI salesman can convince your boss to fire you and replace you with an AI that can’t do your job.” So what can we do with the understanding that LLMs will never ever be able to do our jobs? I’m not naive enough to believe that we can convince the bosses of this, of course. But there’s value in holding this knowledge ourselves as a form of intellectual self-defense. They desperately want us to believe that an AI can do our job, that there is no alternative, that we should gladly welcome our new chatbot overlords. They want this because if we believe the opposite, then we might get it in our heads to organize and resist.