Man and Machine—May the Lord Save Us: IV. Is Man a Machine (Column 697)

In the last three columns I discussed whether the machine is a person. I distinguished between two different questions: (1) Can a machine do everything a human can do? (2) Even if it can, should we treat it as a person? Up to now I have dealt mainly with the second question, and the conclusion was that regardless of the machine’s computational power, it is not a person—chiefly because it lacks thinking and understanding (it has syntax without semantics). This can be sharpened via a saying of a Chabad influencer named Chaim Shlomo Kasselman, brought by Shneor in a comment: A day will come when computers will replace all the scholars, but a computer that prays (serves God) will never exist. I replied there that with respect to prayer this depends on how one conceives of prayer and what a person is actually doing when praying (machines can read books too 😊). But one thing is clear to me: even if it can pray, it certainly cannot learn Torah and be scholarly. The reason is that even if a machine can perform all the actions we perform, it lacks the mental dimension that stands behind those actions. Learning involves understanding and thinking, and those two functions do not exist in machines. I couldn’t resist adding that of course it can recite Tikkun Leil Shavuot or chapters of Psalms at funerals and memorials (in those superfluous functions there are only mechanical actions with no human understanding behind them). Oh, and it cannot conduct “moving” challah-separation ceremonies either (because it doesn’t get moved). Well, I can’t do that either.

All the discussion so far assumed that machine and human are two different beings, and the question is whether and how to compare them. As noted, thus far I asked whether the machine is a person. By contrast, in this column I will deal with the opposite question that arises from the very same contexts: Is a human a machine? The achievements of the new AI machines raise the doubt whether our basic distinction between human and machine even exists. Perhaps the human is nothing but a sophisticated machine (a biological computer—that is, a computer whose hardware is made of flesh and blood).

However, here I will focus only on computational performance. The question I will discuss is not whether a human is a machine, but whether human thinking is mechanical. I am not touching here on whether we have a mental dimension—I have discussed that elsewhere—but only whether the mental dimension is relevant to our thinking. It may be that all the rest of our spirit and judgment are merely epiphenomena, and in fact we are dealing with purely mechanical computation to which mental experiences merely attach.

AI as a Researcher in the Life Sciences

I’ll open with a news item I read a few days ago titled: “Researchers were stunned: Artificial intelligence solved in two days a mystery that occupied scientists for a decade.” It reports on the difficulty of understanding the antibiotic resistance of superbugs. This problem engaged researchers for about a decade, and a model solved it in two days. A very odd headline, since any calculator can do in a tenth of a second computations that would take us humans, working with paper and pen, a week or a year. Any computer simply does things at a totally different pace from us. That’s exactly why we need it—because of its speed advantage over us. We saw in previous columns that the meaning of the results exists only in our heads. The computer arrives at no result or insight; it merely sends electrical signals from here to there. Its programmer is the one who solved the problem, not it. In short, the computer’s essential advantage over us is speed, and that advantage is not new. That alone is why computers have existed in the world long before AI. The miracle in the achievement of that AI model from the article is the very success in solving the problem, but certainly not the speed.

Narrowing our question: Can a human do everything a machine can do?

We must understand that one side of our question is already clear and simple today: in principle there is no problem under the sun that a computer knows how to solve and we do not.[1] In principle, we can do on paper everything the computer does, step by step (after all, we programmed it). The issue is only how long it will take us. Therefore there is no doubt that at least in principle we can do everything it can do. The only question worth discussing is the reverse: Can it do everything we can? Is there anything we can do that the computer cannot? Note that all this is true even for those who believe that the computer can solve every problem we can. They too will agree that the converse is certainly true, since, as noted, we can always follow, in writing, what it does, step by step (only the time constraint interferes). Not for nothing are problem difficulty classes in computer science determined by the time required to solve them (or the number of basic computations: polynomial, exponential, etc.).

The question I will address in this column concerns us, not the computer. Its essence is whether all our thinking is mechanical (and therefore can be mimicked by a computer) or whether it contains non-mechanical components. Clearly, a large part (indeed, the vast majority) of our thinking is mechanical; the debate is only whether there are components that are not. There is, however, an additional assumption here—that the computer certainly operates mechanically—and even that isn’t entirely simple regarding the new AI models. In other words, in the background lies another question: What is mechanical thinking in the first place? These are the questions I want to touch on here.

The Church–Turing Thesis

Alan Turing is one of the fathers of computer science. Among other things, he defined the Turing machine, a rather primitive computational model that can nevertheless be proven to do anything a computer can do (up to time constraints). Alonzo Church defined another computational model, the lambda calculus, which can do the same (there is a proof that the two models are equivalent).

The fact is that to this day, no device or calculating machine—physical or merely theoretical—has been presented that can compute a function not computable by a Turing machine. That is, for any conceivable computation, the simple Turing machine model suffices. Moreover, there are functions not computable by a Turing machine, but they are also not computable by any other machine known to us.

These surprising findings say that at least in principle, the Turing machine is the strongest computable model imaginable; it can do anything we can conceive. This is, essentially, the content of the Church–Turing thesis, which concerns the computational capability of different calculating machines. In the background stands the fact that we do not currently have a good definition of “computation” (this is one of the foundational problems in our discussion). In a somewhat simplistic formulation, this thesis states that any computation we can imagine can be carried out by a Turing machine or Church’s lambda calculus (which, as noted, are equivalent). The thesis is a conjecture, not a theorem, but it is widely accepted among computer scientists. The difficulty in proving it lies, inter alia, in the fact that the notion of computation is not well defined. For this reason, it’s customary to reverse things so that the common definition of “computation” today is: whatever a Turing machine can do.

Note the philosophical implication of this (unproven) thesis: if everything we treat as thinking is encompassed by the notion of computation, then, in particular, human thinking can be represented by a Turing machine—i.e., everything we do with our intellect can be done by a (plain) computer. Of course, there is an assumption here that we are not doing something beyond the (undefined) notion of “computation,” so this is not a proven claim but an articulation of a rather vague conjecture. In effect, this remark says that all of our thinking can be done by machines; it has a mechanical character. A human cannot do anything a Turing machine cannot do.

This is what several commenters on the previous columns meant when they assumed that the human being is a biological computer. Their intention was to say that none of our thinking abilities do anything a Turing machine cannot do, even though a Turing machine is an inanimate object and our brain is a biological organ. They claimed that our brain (more precisely, our intellect) is a computer made of tissues and cells, but logically it is a computer like any other. I stress again that I am not dealing here with whether there is something in the human beyond this mechanical computation (the mental functions). We are focusing here on human thinking, not on the whole that we call “a person.” The claim is that our thinking is just a computer made of living tissue.

Here it’s worth adding another distinction. So far I have presented the claim that our thought processes are mechanical, just like a computer—we are a biological computer. One can make a weaker claim: that everything we do can be carried out by a mechanical machine. That is not the same claim, since this second formulation allows us to say that a human does not operate mechanically (is not a biological computer), but still that this gives no advantage (because there can be a mechanical imitation that will reach all the same outcomes). I will return to this question below, but first I must preface a brief discussion of language models.

Language Models

There is a lacuna in the explanations I offered in previous columns about neural networks. It matters here for the discussion, so I’ll fill it in now. I began there by describing a machine designated to perform a particular task (e.g., face recognition). I explained that (supervised) training is done by feeding an example and giving feedback on the machine’s output, where the feedback is based on our knowledge (of the trainer). Essentially we tell the machine whether its answer is correct or not, and it updates its internal weights again and again. The assumption is that after enough examples, the weights will be set so that the result will come out correct even for other examples not included in the training.

I then widened the scope and spoke of a machine that would be able to do all tasks. That requires massive, varied training on all the information and tasks under the sun. How can one train a machine to do innumerable different tasks? After all, we ourselves don’t have a mapping of all existing tasks. I said, in general terms, that we stuff the machine with all the information currently available in the world—for the sake of argument, the entire internet. But I did not explain how this connects to the notion of training. The machine “reads” more and more text. What does it do with it? What exactly is the training we thereby do? What is the task and what is the feedback it receives? How does it update its weights?

The problem is that the training I described was geared to a specific task, which would mean that training a universal machine would require mapping all human tasks and carrying out separate trainings for each. That is impossible. But here enters a brilliant solution called a language model, and lately a “large language model” (LLM). Put simply, a language model is built so that the machine receives a text and is supposed to guess the next words (or next unit, the token) that will appear in it. The feedback is whether it was right or not (whether those are indeed the words that appear there). This is done across all internet text, and the final weights are the finished machine. It turns out that a machine trained this way succeeds at a great many tasks, with highly impressive results. We do not train the model to perform a particular task, but teach it to “understand” linguistic information that is given to it and to produce such information. Now the model can converse with us (as in the Chinese Room), on any subject in the world (since the texts it was stuffed with deal with everything). Such a model operates on the basis of linguistic correlations (a statistical relation between a token and the tokens that precede it). For a more detailed description of a language model, see Stephen Wolfram’s article mentioned in a comment to the previous column. Among other things, he explains there that the next token is not necessarily the one with the highest probability but something near it (this is the “temperature” notion). Such an algorithm yields results that are less mechanical and more creative, and it turns out to work better in many cases.

The genius of this solution is the understanding that the ability to converse (which is also what Turing’s test deals with) is essentially the master key to solving all tasks in the world. In such a conversation you can also ask the model what happens to a body moving down a frictionless incline, and how long it will take to reach the bottom (a mechanics question). You can ask its opinion on quantum gravity or on the antibiotic resistance of superbugs (the same life-sciences task discussed in the video above). The brilliance here is the understanding that the ability to converse actually encompasses the most general problem-solving ability. Whoever can converse on any topic will also know how to solve problems and carry out tasks. And indeed, language models manage to attain abilities for solving a very wide range of problems. To be sure, there are limitations—tasks in which language models do not do well (see, for example, the last paragraph around note 24 here). This is how the models you all know today were built—models that converse with you and give you any information you want in any field (and sometimes lie to you, as we saw in the previous column), write software, conduct scientific research, solve mathematical problems, write academic papers, and more.

These language models differ from the machines described in earlier columns. Those machines were trained to perform particular tasks (like face recognition) and ultimately accomplished them impressively. But their creators knew what they were creating and what they could expect. A language model, by contrast, is not trained for a specific task, nor does it undergo a set of several dedicated trainings, one for each task. As we saw, they are trained to converse, and suddenly we discover that they have boundless capabilities to solve a vast array of tasks. This greatly surprised the models’ creators and continues to surprise them to this day. They themselves did not expect such amazing outcomes and did not know to what extent those tasks are covered by the ability to converse. No wonder such achievements raise doubts about the difference between such a model and human beings. It really behaves like a person with varied, astonishing abilities of thinking and problem solving. You can just look at the first comments on the previous column that brought responses of different AI models to what I wrote. It is simply mind-boggling that a machine can produce such responses and grasp subtleties as one sees there. And this, of course, is a very simple example of their abilities.

As noted, here we are focusing on the more specific question: Is our thinking mechanical or at least mechanically imitable? Apparently, for the new language models at least, the answer to the second question is increasingly turning out to be “yes” (though, as noted, the models currently available have limitations relative to human activities). I dealt with this question also in columns 590–592. I should note that the picture there is optimistic (or pessimistic) compared to how it seems to me now. Back then I still thought it was quite clear there are tasks we can perform that the model cannot. Today I am much less sure of that. But what about the first question? Is our thinking indeed mechanical? Are we, in effect, a biological computer?

The MIU Puzzle

In column 695 I presented Hofstadter’s MIU puzzle, and I’ll return to it here for the sake of the discussion that follows. This time I’ll present it as it appears in the book (see also its entry in English Wikipedia), since I need the solution he presents there.

Briefly: we have a language whose alphabet contains three letters {M, I, U}. A word in this language is a string of letters from those three, like MMIMUU, and so on. But not every word is legal. The set of legal words are those that can be generated from the word MI by applying one or more of the following four rules:

	Formal rule			Informal explanation	Example
1.	xI	→	xIU	Add a U to the end of any string ending in I	MI	to	MIU
2.	Mx	→	Mxx	Double the string after the M	MIU	to	MIUIU
3.	xIIIy	→	xUy	Replace any III with a U	MUIIIU	to	MUUU
4.	xUUy	→	xy	Remove any UU	MUUU	to	MU

Rule 1 says that if a word ends with I, you may add a U afterward.

Rule 2 says that if a word begins with M, you may double everything after the M.

Rule 3 says that you may replace any triple III with a U.

Rule 4 says that if there is a pair UU you may delete it.

The puzzle is: Is the word MU legal in the language? Now try to solve it yourself for a moment.

Solution

If we try to solve this with an ordinary program, the method would be to generate all possible strings in the language and then check whether MU is among them. But this is impossible, since there are infinitely many legal words in this language. We must note that there is no simple way to generate them and order them by length (otherwise we could check all two-letter strings and see whether MU is among them). So what do we do? How does a person approach such a puzzle?

Hofstadter offers a simple solution that requires us to step outside the system and look at the language from the outside. The claim is that the number of I’s in any word in the language cannot be divisible by 3. From this it follows that the word MU is not legal in the language, since it contains 0 I’s, and 0 is divisible by 3. Hence MU cannot be derived from MI by the four rules. QED.

We still need to prove that the number of I’s in any legal word cannot be divisible by 3. Here is the proof:

Only two rules can change the number of I’s in the relevant word:

* Rule 2 doubles the number of I’s.

* Rule 3 removes three I’s.

The initial word contains a single I; 1 is not divisible by 3. Applying Rule 2 doubles the I-count in the string, but that still yields a number not divisible by 3 (a number not divisible by 3 will never become divisible by 3 when multiplied by 2). Removing a triple I (deleting III) also does not change the divisibility-by-3 property: if previously it wasn’t divisible by 3, it still won’t be after the removal. The conclusion is that any word containing a number of I’s divisible by 3 is not legal in the language. QED.

Can a computer reach this solution?

We saw that a classical computer cannot solve this puzzle, since it would have to generate all infinitely many legal words to verify that MU is not among them. The question is whether a language model can solve it, since it imitates our way of thinking. Can a language model also step outside the system and “think” about it from the outside, as we did? Note that on the face of it, there is no reason to think of divisibility by 3 of the number of I’s in the word. One might have thought of divisibility by 5 of the number of M’s, or of the number of U’s being an integer power of 7, and so on. How did it even occur to us to check precisely the number of I’s and precisely divisibility by 3? It seems there is creative thinking here, and apparently it is not mechanical.

To grasp the difficulty, note that a language model does not operate like classical computation, but it is still a mechanical process. It receives some input, applies a rigid mechanism of weighted sums to it, and outputs a result computed that way. The relation between input and output is mechanical. The output is some function—however complex—of the input. Hence the question arises: can a language model exercise thinking like that in the solution above? If so, then apparently the process we carried out was not truly creative, since the puzzle’s input somehow dictated the path to the solution.

In light of the above, we are essentially asking whether a language model can be creative. Alternatively: is there a mechanism (since, as we saw, a language model is a kind of mechanism) that can produce an outcome we reach via a non-mechanical, creative process? In other words: is a mechanical imitation of creative thinking possible? Of course, one can raise the opposite wonder: perhaps we ourselves operate mechanically, and it only seems to us that we acted creatively here?

Bounded and Infinite Problems

Suppose we added a rule that the length of words in the language may not exceed five letters. In that case, there are at most 5³ (i.e., 125) possible words, not all of which are legal. The question is whether a classical computer (not artificial intelligence and neural networks) could now solve the puzzle. This can be solved with a classical computer, since it is a finite problem: it will collect all the legal words and then check whether MU is in the resulting set. However, even this is not truly a finite problem, because the path to those short words can pass through very many—indeed, infinitely many—possible paths (increasing and decreasing the number of letters in the word, as long as one ends up at a word of fewer than six letters). But if we add a restriction that the four rules may operate only on legal words—that is, that even on the way to the target word one may pass only through other legal words (if the path includes illegal words, it is blocked)—then the problem is entirely bounded, and a classical computer (not AI) could do it in milliseconds.

Apparently, then, the conclusion is that for finite problems there is clearly no difference between human and machine. Finite problems a machine can always solve. But this answers the question whether there is a mechanical imitation of our thinking, not the question about the nature of our thinking itself. Imagine a person thinking about the finite version of the puzzle as I proposed. He won’t do it the way I described for the computer. He will use techniques like Hofstadter’s (which involve stepping outside the system). That is, the problem a mechanical computer can execute, the human will solve in a different way. In such a case the computer constitutes a mechanical imitation of what we do, but our own act is not necessarily mechanical. We solved it in a creative way. Hence the existence of a mechanical imitation does not necessarily say anything about our thinking. The question is whether an AI program can also solve that finite task creatively (not like a classical computer). If so, then there is nothing stopping it from also solving the unbounded problem (i.e., the original Hofstadter problem). But now I must recall that a language model is a machine that operates in a purely mechanical way. So in any case, even if the machine succeeds at the unbounded puzzle, it will be a mechanical imitation, not creative thinking. As noted, this raises the suspicion that maybe what appears to us as creative is not truly such.

In any event, this example illustrates the difference between the two questions posed above: whether there is a mechanical imitation of our thinking, and whether our thinking is mechanical. In the bounded (finite) problem there is a mechanical imitation that a classical computer can do, and yet a person who solves it likely exercises creative thinking. The question is whether an AI that solves it would do so mechanically or creatively. The implication bears on the unbounded problem (in which, apparently, one cannot succeed mechanically—at least not via classical computation). But here the foundational question returns by the back door: perhaps we too do not truly solve it creatively. Perhaps it is a mechanical process that we represent in a way that seems creative. If AI can do this—and we saw that the model certainly operates mechanically (it maps input to output via a (very complex) mathematical function)—then if an AI program succeeds at the unbounded problem (which classical computation cannot do), it would appear that there is mechanical computation that perfectly imitates a process that seemed to us manifestly creative. Does this mean that our solution is also achieved mechanically, and that the creativity we saw in it is an illusion? And perhaps the notion of “creativity” is not necessarily opposed to “mechanical thinking.” The question is whether everything that can be carried out by mechanical mathematical computation is necessarily non-creative.

Chess and the Four-Color Problem

In column 35 I discussed chess-playing programs. The number of chess board positions is finite but enormous. Classical computers struggle to cover so many possibilities, so a program that plays chess at a very high level requires AI. But already decades ago, the program Deep Blue defeated the world chess champion, Garry Kasparov. Today humans stand no chance against AI. Does that mean such a program is creative? Not entirely clear. First, software is always mechanical in the sense of input–output mapping. Beyond that, this is a bounded problem, so in this particular case mechanistic success may be due to the existence of a mechanical solution (albeit not within reach of a present-day classical computer that cannot traverse all possibilities on the board). A human player probably does play creatively—that is, he does not actually traverse all possibilities. But he does filter the relevant ones and, among the remainder, he does traverse them all. The creativity lies in the filtering, and the question is whether such filtering is creative, or whether it too hides behind it a mechanical computation. In the old chess programs, the programmers themselves inserted filtering processes (since the computer couldn’t traverse all possibilities). That is, it was not the computer that solved the problem, but the programmer who solved it by means of the computer (as we saw in the first column, 694).

But here the filtering concerned a finite problem—that is, reducing the number of possibilities to a size the program can examine directly. In column 35 I also discussed the solution of the four-color theorem, where we encountered filtering for an infinite problem. The problem concerns a political map, i.e., a division of a two-dimensional area into sub-areas (states within the overall map), and the question is how many colors are needed to color any possible map such that no boundary has states on both sides with the same color. The theorem asserts that four colors suffice. For many years the conjecture could not be proven, but it seemed quite clear that it was true. Note that this is an unbounded problem, since in principle there are infinitely many possible maps, and the proof requires traversing them all. At some stage it was proven that one can partition all possible maps into 1,936 types of maps and later into 633 types; now it remained to prove the theorem for a finite (but large) number of maps. The proof was obtained by a computer that went through all possibilities (back then it took hundreds and thousands of hours), and it sparked a philosophical debate (whether a computer-assisted proof is an acceptable proof in mathematics—the intuitionism debate).

For our purposes, note that this filtering transformed an unbounded problem into a bounded one. We began with a situation in which one must check infinitely many cases—which a mechanical computation cannot do—but a human succeeded. Now it became a problem solvable even without AI. Where did the need for creativity go? It seems it lies in the transition from the original problem to its bounded version. There, a filtering action was performed, and if there is creativity, it is required at that stage. Here the question arises: can an AI model also perform such filtering? That stage involves a non-mechanical procedure (one a human can do but a classical computer cannot), and the question is whether only a human can do it—or perhaps an AI program can as well. I think it is fairly clear that actions similar to this (which to us appear manifestly creative) are already being done today by AI models (e.g., conversing with us and passing the Turing test). But for the time being, not all kinds of such actions can be performed by AI models; the question is whether this is a temporary situation that will be resolved, or an essential limitation of mechanical machines.

A broader look at what we’ve seen: Is everything a machine solves “mechanical”?

We saw that bounded tasks can be accomplished by classical mechanical computation. We saw that humans can perform unbounded tasks as well (by stepping outside the system). At the same time, it is now clear that many tasks we perform are also accomplished by AI language models, including some that seem unbounded. On the other hand, as long as we are dealing with a machine, the operation in question is a mechanical computation (a function from input to output). But that does not mean those tasks are not creative. It certainly does not mean that we do not exercise creativity when we solve them (remember the bounded MIU puzzle, where we saw that at least for humans there is stepping outside the system—i.e., a creative solution). It may be that there is a mechanical imitation of a creative act—that an AI model can solve Hofstadter’s puzzle. And still, that would not mean that we did not solve it by creative methods.

To sharpen this, I’ll recall what I described in detail in the recent columns: the mechanical imitation by a language model is carried out by a neural network created through very intensive training by humans. That training embeds in the network our knowledge and our thinking skills, and these can indeed be products we accumulated thanks to our creative ability. That is, it may be that at the training stage the human trainer feeds into the model—via the examples and feedback—the creative dimensions we possess as human beings; and no wonder that from then on the model manages, mechanically, to solve even tasks that require creativity (including unbounded tasks). The model is not just a mechanical machine, full stop, but a machine created on the basis of human thinking and skills. Therefore there is no bar to its having creative capabilities that are executed mechanically.

In the four-color problem we saw that after humans found the partition into types and fed it into the computer, the (classical, non-AI) computer could work mechanically and prove the conjecture. Likewise, it may be that an AI model can mechanically solve a task that requires creativity because its programmer inserted into it, during training, human creative thought-structures. If so, then even the AI is not truly the one solving the problem; rather, we are doing so by means of it, just as we saw in the first column regarding classical computation. This applies even to tasks at which, if addressed directly to us, we would fail (because of our limitations in mechanical computation and in our computational speed). In other words, it may be that its advantage over us is only at the level of mechanical capabilities and speed; yet that enables it to solve tasks that require creativity which we cannot cope with. Of course, this is only if our difficulty stems from our limitations in mechanical computations and not from the essence of creativity.

So far we have reached no clear conclusions regarding the questions I raised, but I think the landscape is now clearer. At least the questions and possibilities are better defined, as are the ambiguities that accompany them (what is mechanical computation and what is creativity).

Reflections on Language

I now come to a discussion that unfolded in my email with my student David Madr (mentioned at the beginning of the first column). The picture I have sketched here raises puzzles about our language, most of which he flagged in his letters. On the face of it, the ability to use language somehow contains the whole of our cognitive abilities. Are the structures of language and its syntax not merely formal structures, but also based on logical principles and inclusive of the laws of nature (knowledge about the world)? How can it be that learning to use language encompasses all these skills—or at least successfully imitates them so completely?

Put differently: can and how can one use a linguistic skill to deal with synthetic problems? Analytic problems (in my Kant-derived terminology) are problems whose entire solution method is contained within the problem and its components. To solve such a problem there is no need to go “outside the box”; it suffices to process the input to produce the output. Synthetic problems are problems that require addition (a synthesis with information and skills beyond what is contained in the problem itself) in order to solve them. A priori I would have expected that a machine that uses mechanical computation could deal at most with analytic problems (and perhaps not all of them), but not with synthetic problems. The question is whether the problems that seem synthetic to us are in fact analytic, since AI also knows how to handle them. Needless to say, our language as such (grammar and syntax) is not truly analytic. Some claimed that all philosophical problems stem from linguistic failures (this is the assumption of many analytic philosophers), and therefore some tried to create analytic languages designed to be perfect—that is, free of paradoxes and philosophical pitfalls (like Russell with his theory of types, Leibniz, and others). I think Gödel’s theorem casts serious doubt on the possibility of creating such a language; but even if it were possible, it is doubtful such languages could express all the richness we express with our language. And still, it would seem that linguistic ability should not exhaust all our abilities across all domains. Hence the achievements of AI models are so surprising.

Similar reflections appear in Wolfram’s article mentioned above. His claim is that language has a semantic basis, and the contents it expresses also influence its structure. Language is not just syntax or arbitrary syntactic rules; behind it there is logic and scientific, factual knowledge. His claim is that the use of language is not as creative and complex as it seems to us on the one hand, and on the other hand it depends on the contents it expresses. The basic logic must be substantive as well, not merely formal. I think Wolfram’s work over many years assumes this and tries to realize it (achieving the ability to handle substantive arguments and problems by formal means).

Briefly, I told David that it’s quite clear to me that ideas lie behind language; yet the mechanical manner in which AI software operates does not mean it deals only with language. There too, language is only a representation. Moreover, what is represented by it is not present in the software’s “awareness” but in the programmer’s. Moreover, we saw that the software’s training is done via answers and feedback to cases presented to it, and these are not determined solely linguistically. For example, one trains software to recognize someone’s face. The training is through examples and by giving positive or negative feedback for correct and incorrect recognition. The feedback is determined by whether it recognized correctly or not—that is, by the content.

Even in a language model that is ostensibly trained only on syntax and linguistic skills, this is not really the case. We saw that it is trained on various texts that express ideas in different fields; therefore the continuations it produces are not merely the result of formal grammatical rules but of content. The training is not just the embedding of formal syntax, and that’s it. Therefore it is no wonder that this shapes the network in a way that will answer to contents and semantics. This training is not a purely linguistic operation, even though it is represented through language. For us too, ideas are represented by language; but at least for us it’s clear that behind it are the ideas it expresses. Think of the person in the Chinese Room. He receives feedback from Chinese speakers that is based on correct and incorrect contents, and the syntactic sentences he produces are evaluated by content, not merely by their grammar (even though he himself does not truly understand them).

To sharpen this further, think of a language model fed with all sentences that are grammatically correct—regardless of their truth and logic. For example, it would receive sentences like “All speakers of Plantish fly with their wings to gather clouds,” and other such sentences. Would such a model acquire problem-solving abilities? Certainly not. It is therefore clear that these wondrous skills do not arise from the syntax it learns but from the filtering done by its trainers between syntactically well-formed but nonsensical claims and other well-formed claims that represent correct, true content. That filtering—not syntax per se—is what is responsible for developing an AI model’s skills; and, as noted, that filtering is done by the trainer or by his lights, not by the machine. Let us not delude ourselves that we are only teaching it language. We use language, but we teach it, by means of language, facts, logic, and methods of thought. Hence it is not the language that is responsible for the skills of a language model, but the contents passed to it in training (which are indeed represented linguistically). Just as with us, language is the instrument through which we convey contents; we should not confuse the instrument with the contents themselves.

What has all this to do with Evolution?

Finally, one last reflection (also raised by David). If indeed there is in us something beyond thinking—yet mechanical thinking can perfectly imitate it—then it’s unclear why, evolutionarily, this additional component arose in us. If it gives us no evolutionary or other advantage (since the mechanical imitation can do everything at the same level), why would we need it at all? This indicates that we are not products of evolution alone (a soul was not generated evolutionarily), and it also explains why materialists assume that there is indeed a perfect mechanical imitation of our thinking—and hence that our thinking itself is mechanical.

A similar argument was already raised by the well-known Christian analytic philosopher Alvin Plantinga (see my book God Plays Dice, chapter 4, §b). He argued that there is no evolutionary justification for the existence of consciousness, since mimicking functions that would do everything without consciousness behind them would contribute equally to our survival. Think, for example, of fear of predators. Suppose there were a person who felt no fear of predators at all, but still, if a predator appeared nearby, he would run for his life—just like that. It would be an instinct in him, with no mental drive behind it. His behavior would be exactly that of creatures with fear, but only phenomenologically, without the mental dimension. Clearly, his survivability would be identical to that of a creature who fears, and therefore the feeling of fear is evolutionarily superfluous. So why did it arise?![2]

Here I make a similar argument. If indeed there exists a perfect mechanical imitation of our thinking, then the mental dimensions within us have no evolutionary advantage, and therefore they should not have existed. In the first column I explained that no one disputes that they do exist (see there the mention of my conversation with Prof. Yosef Ne’eman, of blessed memory); hence there is a scientific problem for the hypothesis that a perfect mechanical imitation of our thinking exists. But if such imitation does not exist, then there is something non-mechanical in our thinking—and that more than hints that there is in our intellect something beyond neurons, and perhaps that there is in us something beyond physiology (dualism).

The next column will likely conclude this series, where I will discuss the philosophical implication of the picture I have drawn here. I will address the question of the centrality of intellect versus will, which is also sharpened in light of the discussions we have held here about the character and nature of our intellect.

[1] It may be that in quantum computation the situation is different. I am not sure.

[2] One could reject this argument by claiming that the feeling of fear generates all the necessary behaviors in different situations; it is more economical than creating each behavior separately. This assumes that evolution indeed generates our mental dimension.