“And you shall be like God, knowing good and evil”: On Technological Phobias (Column 186)
B”H
Dedicated to my son Yossi and to the system who_is_next.com
Following the previous column, in which I discussed the government’s lack of policy in Gaza and more generally, I was asked by Moshe why I prefer policy, whatever it may be, to random conduct. I answered him that from a consequentialist standpoint this may not always be preferable, but we elect a government and pay it, and do not make do with a random generator—yet whoever actually governs us looks like a random generator. Isn’t that a waste of money?! This led me to begin thinking whether I have any principled objection to being governed by a machine rather than a human being, and I tried to understand why. This connected to another topic that has already come up here in the past: the attitude toward autonomous weapons and autonomous cars. These too make decisions in our place and on our behalf. Since the question intrigued me, I immediately sat down to write the following column, and so I hereby inflict it on you.
Technological phobias
From time to time, public discussion raises fears that technology will take over us and perhaps even exterminate us. These concerns become sharper when viewed through an evolutionary lens that sees us as just another species that may be wiped out by another species or by some natural disaster or other (like the dinosaurs). Sometimes the fears surrounding technology are moral rather than survival-related. Thus, for example, there are concerns about cloning and creating artificial beings through various technological means. Here the concern is about engaging in such activity as such, even when no concrete threat to human beings is involved.
All this gives us some healthy proportion regarding our status as the ‘crown of creation,’ and regarding whether, and to what extent, the human being is truly eternal rather than transient. But with all due respect to humility, which is of course positive in itself, these fears—and especially the way they are expressed—strike me as somewhat ridiculous. In many cases they look like childish phobias of the witch or dragon in the closet (recommended treatment: watch the masterpiece Monsters, Inc.). Technology is a kind of modern dragon that threatens even adults. Let me say already here that in my opinion as well there is room to ensure regulation—moral and survival-related—in these areas, but the deep fears and apocalyptic anxieties reflected in discussions of these subjects seem to me to point to something much more basic and profound than the concerns explicitly voiced there.
Questions of framing
One could enter here into the definition of a human being and into the question of why this is perceived as a greater threat than threats posed to us by other human beings. One could also discuss why it matters to us at all that our species survive (beyond our own personal survival, of course). What exactly is sacred about these particular collections of molecules? These concerns also raise the question of materialism—in other words, whether there really is any substantive difference between a machine and a human being (apart from the percentage of carbon in the body).
It seems to me that the fears surrounding cloning illustrate this absurdity well. Either way: if we are dualists, that is, if we hold that a human being is not merely a body, then there is no fear that we will be able to clone human beings. The clone will be another person, perhaps similar in many respects to the original, but not essentially different from an identical twin. And if we are materialists who believe that a human being is only a body—then what is the fear of cloning? What is wrong with duplicating collections of molecules? In short, either cloning is impossible, or there is nothing especially problematic about it.
The pathos-filled speeches and primordial fear of communities of clones that will supposedly be created here really seem to me childish and absurd. Joining this trend are trendy rabbis who, out of that fear, explain to us that there is a prohibition in Jewish law here and not merely a moral problem: intervening in nature, ‘playing God.’ It hardly needs saying that this claim has no basis in Jewish law (see below). Incidentally, dragging God’s name into the matter is common among many of the worriers of all kinds, who claim that human beings are putting themselves in God’s place. This includes even complete atheists, because at the level of phobias primitive religious fears are apparently allowed to mix in even among enlightened people.
But that is not what I intend to discuss here. What I want to focus on below is the fear of technology itself, not an examination of the conceptual framework within which it is generated and conducted. If we focus on artificial intelligence, it seems to me that there are two clear examples in which one can see the phenomenon of fear of AI: autonomous vehicles and autonomous weapons.
Autonomous cars
In recent years quite a few companies have been engaged in developing autonomous vehicles, that is, vehicles that drive automatically without a driver. Beyond the technological difficulties, the question also arises how such a vehicle is supposed to make moral decisions. The examples usually used are various trolley dilemmas (see, for example, here and here). The original dilemma concerns a person standing at a junction where the railway tracks fork, with the switch lever in his hand. He now sees a train approaching, and farther down the track five sleeping workers lie on the rails. He must decide whether to divert the train onto the second track, on which one person lies, or let it continue and kill the five. Originally this question was presented by the philosopher Philippa Foot in order to examine whether our morality is instrumental (utilitarian-consequentialist) or deontological (a morality of intentions and motives). Later, a good deal of work was done by psychologists who examined the attitudes of different people and different societies to this question. It turns out that the answers are connected to quite a few of our cultural and sociological characteristics. At MIT, an internet study was conducted on millions of people, and different breakdowns of these answers were presented according to various measures. In light of this, the question naturally arose how we ought to program our autonomous car to respond in such situations. This is a moral decision, and we have a tendency to think that such decisions ought to be made by human beings.
Some have warned that a car cannot be trained to make correct decisions in every situation, since it is impossible to think of them all in advance. But this problem, difficult as it may seem, is mainly a technological problem. Those who raise it usually picture a machine that receives a set of instructions pertaining to all situations and telling it what to do in each one. From that perspective there really is room for concern that the car will encounter situations we did not foresee in advance. But in our era we are not dealing with such software, but rather with software based on a learning network. This is a technology in which the software is given various examples of situations and the correct responses to them, and from these it learns what it should do in other situations as well. It is then tested on additional examples, and every piece of feedback on an answer it gives is fed back into the system, which updates itself until it reaches the point where it gives correct answers in almost all situations. With this technology, there is no need for the machine to encounter all conceivable situations. After enough examples, it learns to respond correctly to every type of situation (it is basically imitating us). There will of course be misses, but humans have those as well (see the statistics on road fatalities). I assume that already at the technological level we have today, the number of victims on roads populated only by autonomous cars would be significantly lower than what we have now.
To understand this better, think about the way we ourselves make decisions. After all, we too cannot really think through every situation in advance and prepare ‘schoolbook answers’ for each of them. Who among you knows what he would do in the trolley dilemma? And even if, after reflection, he knows, is he sure that if he were to find himself in such a situation without prior thought he would make the correct decision? And in general, what is a correct decision in such a case? We too learn from examples that gradually build our thinking and our moral decision-making. The learning network tries to imitate the way human beings learn, and the assumption is that after enough examples it will reach the ability to make correct decisions in a large majority of situations, like a human brain. One must remember that human beings, too, do not always make correct decisions, and on the road there are millions of drivers with worldviews, abilities, and reaction speeds that differ enormously from one another. It is highly likely that a machine trained sufficiently by experts and broad oversight will make better and more correct decisions, and certainly much faster, than human beings—certainly than average human beings (like the reasonable driver). Therefore computerized decisions are expected to be better than the decisions we ourselves make in such extreme situations. As noted, the learning network is a mechanism that purports to imitate the way we ourselves learn to respond in different situations.
The arguments
The arguments raised include fears of incorrect answers in certain situations, fears of malfunctions in the car, which after all is only a machine and can of course break down. Beyond that, different people have different value systems, and the question is which value system will be installed in autonomous cars, and who will decide how they are to make decisions and which decisions are ‘correct’ for each situation (see here).
But these arguments seem to me unfounded, at least comparatively. Human beings suffer from more failures than autonomous cars. Human beings on the road also have a great many different value systems, and no one checks them before they are given a driver’s license. Human beings do not always make morally correct decisions, and who will decide what a correct decision is? Human beings are sometimes criminals, driven by impulses and passions, which is not supposed to occur in cars. Why is it so obvious to us that human beings are preferable to cars in such situations? At least from the consequentialist standpoint, I very much doubt that these arguments have any basis.
It is worth seeing here a report on an incident in which an autonomous car ran over a pedestrian because of a malfunction. The whole world erupted, and voices immediately arose saying that machines must not be allowed to make decisions in such situations. Does anyone seriously think that when a pedestrian darts into the road, a human driver would necessarily have managed to avoid her? If so, how is it that in the State of Israel alone there are hundreds of people killed every year in traffic accidents?
At least from the standpoint of these considerations, it seems to me that even today autonomous cars can be allowed on the road, provided that their decision-making system is trained in advance according to several value systems that pass the legislator or regulator. Each passenger can choose which of them will be installed in his car, and thus the matter would imitate the situation that exists today when the drivers are human, and I estimate that the results would be significantly better. In any event, the fears seem to me greatly exaggerated.
Autonomous weapons
Similar concerns arise with respect to autonomous weapons. A conventional weapon (by which I mean a non-autonomous one) currently serves as the extended arm of its operator, just like a conventional car. An autonomous weapon makes decisions on its own according to the situation in which it finds itself. And again, very similar arguments arise here (I get the impression that the moral arguments here occupy much more space than in relation to autonomous vehicles, and rightly so). About a year ago it was reported (see also here) that more than a hundred experts sent a letter to the UN demanding supervision of autonomous weapons or their outright prohibition. Ironically, quite a few of the signatories are themselves engaged in developing such weapons. At a conference that has just concluded in Geneva, agreement was reached on general and non-binding guidelines for the development and sale of such weapons, and many are protesting the feebleness of that agreement. See here a report on an appeal by experts and activists to the governments of Canada and Australia demanding that the development and sale of such weapons be prohibited; likewise regarding Google employees, and so on.
What exactly is the fear? If it concerns the scale of destruction and its indiscriminate nature, this is not essentially different from nuclear, biological, or chemical weapons. On the contrary, the damage here is far smaller. And what about malfunctions? Humans have those as well. The same goes for incorrect decisions. As we saw above, different value systems already exist today among different states that operate conventional weapons. Fears that such weapons will not distinguish well between combatants and non-combatants, or that they will decide to kill too many people, or carry out large-scale killing, also exist with respect to human beings and conventional weapons. If states develop nuclear weapons and the world has no ability to prevent that, why does anyone think it will be possible to control the development and use of autonomous weapons?! Autonomous decisions will be faster and of course also more efficient, sharper, and more precise than those of human beings, but that only means we have improved our weapons. As for their degree of morality, I do not see why there should be any difference between them and decisions made by human beings.
Moreover, even the definition of what counts as an autonomous weapon is not sharp: after all, a third-generation missile also makes decisions as it moves. Quite a few weapons carry out various calculations on their own, and therefore the definition of when a weapon may be called autonomous is not unequivocal.
Beyond all this, I greatly doubt how far such a global policy can be supervised at all. If some organization or state decides to develop such autonomous weaponry, will anyone be able to prevent them from doing so? Will anyone even know? Do we really trust that they will refrain because the UN has prohibited it? And in general, even if such a treaty exists, I assume that peace-loving and democratic nations will adhere to it, whereas terrorists or dictators pursuing power and rule will continue to develop such weapons to their heart’s content. So what have we gained by prohibiting it? Fine—we can at least console ourselves that we fell into the hands of robots operated by flesh-and-blood people, rather than our own robots having taken over us, God forbid (as Scripture says: Slaves rule over us, and there is none to rescue us from their hand.).
Preliminary conclusions
In terms of outcomes, I do not see a substantive difference between improvements in conventional weapons (in the sense of non-autonomous weapons) and autonomous weapons. If people fear the consequences of losing control over the operation of such a weapon, that is not essentially different from fear of a very powerful atomic bomb. Is there control over that? And if so—who is the controller: the UN, the great powers, or perhaps the terrorist who holds it? Therefore the discourse about limiting autonomous weapons should not be essentially different from the discourse about limiting weapons in general (and for that, of course, there is definitely room). So why is it nevertheless so different and so frightening?
My feeling is that the terror expressed in this discourse mainly reflects fear of handing our fate over to a nonhuman factor that makes decisions autonomously. The concern is not about the results, but primarily about the very control of golems over us. This is not an instrumental (consequentialist) concern, but a value-based deontological one. In fact, it would be more accurate to say that this is a phobia, that is, a modern terror of the dragon or sorcerer in the closet. The constant waving of terrifying, large-scale harms and loss of control—in other words, consequentialist considerations—is nothing but an attempt to use familiar and sensible arguments in order to express this irrational phobia. People are driven by a primordial fear of technological dinosaurs taking over, and they wrap these fears in rational arguments about catastrophic consequences. The calls to impose supervision on such weapons express a primordial fear, and therefore they should not be evaluated in terms of efficiency, feasibility, or logic.
My claim is that in both of the examples above, the fear concerns mainly the way the damage comes about, not its magnitude. We fear the very handing over of our fate into the hands of creatures of another type. This is an evolutionary fear of losing human sovereignty. It turns out that despite the bitter experience we have accumulated throughout history with various rulers, mass murderers, and wicked conquerors, we still have some basic trust in human beings, wicked as they may be. We prefer them to machines that make decisions in a cold and alienated rational manner. Perhaps we think that decisions made by human beings cannot be too harsh. At least they cannot get out of control and cause damage beyond their declared goals. But what if the declared goals themselves are problematic? If there were a human murderer whose goal was the destruction of the world or of some continent, then even if it were carried out without deviation from the original plan, we would be in trouble, would we not? True, but those are human troubles: they may be psychopaths, but not demons and mechanical zombies without restraint. We prefer the familiar kind of wicked people and murderers. Nice familiar Nazis, Stalins, or Pol Pots. Thus even if we die in agony, at least it will happen when everything is clear to us. We will die and scream with a smile of familiarity on our lips.
In short, what we have here is mainly a phobia, far beyond the rational concerns that supposedly explain it.
Fettering discretion
In their article, Eliav Lieblich and Eyal Benvenisti, ‘Robotic Warfare and the Problem of Fettering Discretion’ (Iyunei Mishpat 39, 2016, pp. 567-106), they survey the subject from its various angles and address most of the arguments I have raised here, along with many others. I will not be able to go through the whole thing, but I will describe its main lines, because one can indeed get the impression from the overall structure that this is a phobia.
After describing the problem and defining the concepts, they divide the discussion and the various arguments on the issue of autonomous weapons into two planes: the instrumental (consequentialist) and the deontological (intentional-value based). Their claim is that on both planes the discussion is circular and problematic and does not stand up to the test of logic. They point out that most of the considerations raised on the instrumental plane are also relevant to ordinary weapons, and the argument unique to autonomous weapons is arbitrary, speculative, and circular (it rests on optimistic assumptions about human nature. In other words: Pol Pot is preferable to autonomous weapons)[1]. From my perspective, this is of course an expression of the fact that these arguments do not reflect the real reasons for those fears. These are attempts to present phobias in rational terms, and therefore there is no point in judging them by the tools of logic.
The deontological discourse they describe is based on the assumption that even if, consequentially speaking, one can assume that autonomous weapons would actually improve the situation in terms of outcomes, still we must not place decisions about killing human beings in the hands of automatic and autonomous systems. A machine must not be allowed to make decisions about killing people. Why not? Just because. Why are these decisions themselves not regarded as decisions of human beings—the ones who created the machine and operate it—made through the mediation of the machine? Just because.
On the more substantive deontological plane, they bring two types of arguments. The first type is the claim that war is a human action, and as such it must not be conducted in a way that only one side bears risk (this is like the Jewish-law prohibition on interest, which applies only to an investment without risk). According to this argument, one-sided killing is immoral. It seems to me that here we have returned to the rules of the duel and to the wars of knights over the hearts of delicate ladies. This argument does not apply, of course, when both sides have autonomous weapons. Nor is it unique to autonomous machines. It is also true of ordinary powerful weapons (like an atomic bomb). Thus our war in Gaza is immoral, since they have no chance whatsoever (even with our idiotic leadership). It is not a fair duel. You may ask: after all, people also try to impose supervision on atomic weapons. A nuclear state is expected not to use nuclear weapons even if it will pay a heavy price. But that is an argument one can accept, since it concerns the proportionality of the damage you cause relative to the damage you prevent to yourself. One does not kill millions of people to save two. That is not an argument about fettering discretion but about proportionality. But if the argument is because of considerations of chivalry and fairness, then it is simply nonsense. If a state faces an existential threat, the demand that it not use nuclear weapons even in such a case is ridiculous (and of course will not be implemented in practice).
The second type concerns the distance between the person making the decision and its consequences. When that distance is too great, pangs of conscience are not to be expected in the operator. This too is of course relevant to many other weapons (that is what people always say about pilots as opposed to infantry soldiers). Beyond that, if the consequential difference is significant, does this abstract principle justify taking more lives? (Somewhat similar to the trolley dilemma.) Remember that the deontological discussion is conducted on the assumption that autonomous weapons would bring better results in terms of consequential harms. Beyond that, I would not build too much on the pangs of conscience of Kim Jong Un, or Pol Pot.
Some argue that in the case of autonomous weapons there is no one who will bear the consequences of the act. There is no one to put on trial, because it was the machine that made the decision. But this too seems to me a problematic argument. Whoever designed it and sent it into action will bear the consequences. Beyond that, again one must weigh this against the consequential difference. Is it not proper to save human lives at the price that perhaps we will not have anyone to put on trial?
The authors, as noted, criticize both types of argument, and in their place propose a third type: administrative law. In their view, combat is a governmental action, especially when it is conducted against individuals or non-state organizations (in such cases it is seen as the exercise of governmental force against private persons). As such it is subject to administrative considerations, which cannot be carried out by a machine. Therefore they propose prohibiting or limiting autonomous weapons by force of administrative law.
They explain that the rules of administrative law require taking the good of the other side into account as well.[2] Such a thing requires a type of discretion that autonomous and automatic weapons cannot exercise. Therefore, at minimum, after the machine’s autonomous decision there must be a possibility of ‘appeal’ before a human factor. Second, the use of autonomous weapons fetters discretion, and that is the main problem with them. They add that fettering discretion also leads to lower-quality decisions in the long term. Decisions that cannot be reviewed in real time may lead to more problematic conduct.
Here too, the implicit assumption throughout the discussion is that the computer cannot really imitate the human decision. It seems to me that they picture before them a program that receives a given algorithm and is therefore limited in its decisions and in the situations familiar to it. Like many others in these discussions, they are not aware of the difference between such classical software and learning networks. Therefore their assumption regarding long-term consequential superiority is unjustified (in my assessment, already with the technology that exists today). Beyond that, the question is whether such a formal legal limitation really justifies giving up weapons that would save human lives.
They further note that international humanitarian law requires constant caution during military action, and this does not exist when discretion is handed over to a machine (that is, fettered). But again, if the software is more cautious than human beings, then that itself is the best possible caution. Human beings err, and therefore oversight is required. Autonomous weapons operate precisely, and certainly do not need the oversight of inferior and fallible creatures like us. Autonomous weapons may in fact be the optimal way to achieve outcomes of minimal harm, far more than human beings supervised by other human beings. Again, my feeling is that these are arguments that come to rationalize, after the fact, non-rational feelings.
Interim summary
Reading the article as a whole clearly leaves the impression that what we have here is an attempt to scrape up arguments from every possible source, but the truth is that at bottom this is a phobia, an irrational fear of being handed over to the decisions of a machine. The psychological fear of a situation in which machines will gain control over human beings is covered by a cloud of supposedly rational arguments.
I should note that I understand the feeling and the fears very well, and I too feel them, but I find it hard to locate any real rational basis in them. In light of the variety of arguments on the one hand, and their weakness on the other, my conclusion is that this is mainly a psychological phobia. Let me repeat that I too support some degree of regulation over such weapons, insofar as possible, as with any other weapon. But the excess terror cast here, as compared with weapons, cars, and human drivers, seems incomprehensible.
Intervening in the work of Creation: ‘playing God’
I mentioned above the fears surrounding cloning (see Professor Steinberg’s comprehensive survey here). There are real and serious concerns, but I still get the impression that the phobia lies behind them. People speak about the fear that they will clone children for us, or blond people. So what? What exactly could happen? Why is it permissible for us to shape their education and values, but forbidden to shape their bodies? In my estimation, here too the same primordial terror sits behind the supposedly rational discourse: the terror of human beings becoming God and engineering creation. The hidden assumption (which in fact is also sometimes said openly) is that only He is permitted to create things and change His creation and our universe. But are we not constantly changing creation and intervening in it? Is surgery or healing in various ways not intervention? Why is that different from healing using genetic tools? People speak of fears of producing battalions of zombie soldiers with superior physical fitness, abilities, and resilience. How is that different from a soldier with technologically perfect armor or shield, or from an ordinary atomic bomb? It is not really different, but it is more frightening. The armored soldier is a normal human being in a wrapper, but the clone is a demon, a dragon—and go fight dragons. Therefore I get the impression that here too, as in the previous examples, this is a phobia presented and wrapped in terms of rational concerns.
Professor Steinberg indeed argues that in the accepted Jewish conception there is no impediment to such things (I think the concerns come mainly from Christian directions, including secularized ones). And yet I get the impression that in recent years there has also been a drift in the Jewish world after the Christian mood (see an example of such concerns here). Usually these are based on the words of Nachmanides (Leviticus 19:19), who wrote concerning the prohibition of kilayim (mixed species):
And the reason for the prohibition of mixed species is that God created the species in the world—among all living beings, in plants and in animate creatures—and gave them the power of reproduction so that the species would endure forever,
for as long as He, may He be blessed, desires the world’s continued existence. And He commanded that by their power they bring forth according to their species and never change, for all eternity.…
And one who grafts two species together alters and contradicts the work of Creation… and among living creatures, different species will not produce offspring from one another, and even those close in nature that do produce offspring from them, such as mules, their seed will be cut off, for they cannot reproduce…
Needless to say, Nachmanides is merely speaking about the rationale of the verse regarding kilayim, and yet he has, through no fault of his own, been turned into a binding source in Jewish law that innovates a sweeping legal ruling. One can raise various concerns and say that this is undesirable, but we also have a strange tendency to turn everything into Jewish law. It sounds stronger that way. And besides, that way one can show that we were ahead of everyone, that we are the most enlightened, and that all the worries and phobias originate in the Torah. Turn it over and over, for everything is in it. (turn it over and over, for everything is in it)—and therefore, of course, nothing is in it. A text in which one can find everything says nothing.
Back to the absence of policy
To conclude, I return to the starting point, that is, Moshe’s question about the previous column. I can now say that in principle I have no problem with being governed by a machine. On the contrary, it seems that it would make better decisions than what is being made today (more accurately: than what is not being made today). It would be very worthwhile for the machine to be based on learning software. It would be less advisable to use a classical machine fed by a decision algorithm (because then there is concern that in new situations it will make bad decisions), and worst of all is of course a random generator.
Ah—but actually no. On second thought, a random generator is not the worst. What is happening here today is worse than all the possibilities I have listed so far. In the previous column I described how decisions are currently made in a rigid and fixed manner with no connection to circumstances or to any goals whatsoever (I mean the substantive circumstances, not accidental events and results). That is the worst option. I estimate that even a random generator would be better than that (because at least in some cases it would accidentally make the correct decision. Even a stopped clock…), and certainly software fed by a reasonable algorithm would be better, and of course the best would be learning software. Incidentally, even the defect we discussed here in mechanical decisions—that there is no human oversight of the machine’s autonomous decision—exists in our reality as well. The government draws no lessons and changes no policy. So how is it different from a machine? In our case too there is no oversight. One could say that we operate under the fettering of all of our discretion.
So why is no one up in arms? Simple: because the phobia is absent here. The machine that governs us looks like a human being with two legs and two ears (just without a head). That shows you that appearances can deceive, and that it is not worthwhile to act on the basis of phobias.
By the way: who_is_next.com
All this reminds me of a proposal by my son Yossi, may he live long, for solving the missile problem from Gaza. He proposes a system, who_is_next.com, built as follows: a website on the internet (or a Facebook page) that at any given moment announces which five houses in the Gaza Strip will go up in smoke after the next missile. When a missile is fired at Israel, the moment it is launched (and regardless of the results of the launch), automatically and without human involvement, five bombs are fired with force sufficient to destroy utterly five houses in the Gaza Strip. The residents of those houses are informed in advance, so that no one is harmed unnecessarily (just think about their lives before the missile is fired). One could in fact keep a numbered and orderly list of all the houses in Gaza, and determine the order in advance according to missile numbers. Public institutions, military and industrial facilities, power and water stations, etc., can and should of course be included. It seems to me that yesterday 400 missiles were fired; by simple arithmetic that means Gaza now numbers fewer than two thousand houses. A few more days like that and our problem will be solved to everyone’s satisfaction.
Since this is done autonomously and without a human decision, then the one who destroyed the houses was the missile shooter, not the State of Israel. Every missile shooter knows that he is destroying five houses in the Gaza Strip and ruining the lives of dozens of families (but not harming the life of any person). Now the residents of Gaza and Hamas will be able to direct their complaints to the missile shooters, and perhaps they will even trouble themselves to prevent their blessed activity. No one will be able to claim against the State of Israel that it is acting violently or problematically. It did nothing. Hamas or Islamic Jihad destroyed those houses.
Admit that this is a refreshing and unhidebound thought (as befits Yossi, may he live long), although it requires legal clarification (especially in light of the principles mentioned in this column) and other kinds as well (there are of course other aspects, and this is not the place). That in itself is an unprecedented vision in these parts. In any case, at first glance I am entirely in favor. Didn’t someone here say that I only complain and never proposed an alternative?…
After writing this, my son Shlomo sent me a passage in which Israel Aumann speaks about this very same idea:
Q. Not long ago you gave a lecture in which you proposed developing an automatic system, without human involvement, that launches a missile immediately in response to every rocket launch from the Gaza Strip.
A. "I am working on it. The goal really is that the system be without human involvement, without human control. This is very important, because if there is control they will say that we are criminals and murderers and cruel. When there is no control, it is as though they take their women and children and force them through a dangerous minefield. There will presumably be global criticism against us then too, but at least we will be at peace with ourselves. If it is automatic return fire, the threat is persuasive: Hamas fires a rocket at Gaza when it fires a rocket at Israel".
And here are the main points of Yossi’s response to the plagiarism:
I am not familiar with the details of Aumann’s project, but there are several significant components in Yossi’s proposal that must not be ignored:
- A list of houses that will be hit in the future must be published in advance, for the whole world and his wife to know, and of course for the residents of the houses themselves.
- People must be given advance warning in order to prevent loss of life (both morally and for legal purposes).
These two components are important because that way there are specific interested parties who will see to neutralizing the missile fire. And thus we will not be accused of war crimes either (at least not of harming innocents). It is important to understand that without them there is no significant innovation here, and no alternative solution. As I wrote in the previous column, automatic fire does not change the responsibility of those who built the system. It is autonomous weaponry (not really), but the responsibility lies entirely on whoever operates it and built it. What difference does it make whether he pulls the trigger or builds a system that pulls the trigger by itself?!
[1] It is worth recalling here Moshe’s question cited at the beginning of the column.
[2] To my mind this is truly a bizarre discussion. After all, the enemy is acting to destroy the state, so why should administrative law be concerned for his rights? Particularly if he is not a citizen of our state. They discuss this toward the end of the article (p. 103), but this is not the place to elaborate.
Discussion
An interesting column. I’ll allow myself to raise here a few preliminary and tentative reflections.
It seems to me that the great difference between a machine and a human being is emotions. One can argue that the absence of emotions in a machine is its advantage over a human being, and one can argue the opposite. But one cannot ignore this difference.
In a famous midrash, Hazal describe the Holy One’s hesitation before creating the world, and say that it first arose in His thought to create it with the attribute of justice, but the Holy One saw that it could not endure, and so He combined it with the attribute of mercy. In light of this column—perhaps the attribute of justice is conduct according to rigid, pre-programmed rules, in the sense of “let the law pierce the mountain,” whereas the attribute of mercy is the possibility of occasionally deviating from the rules and acting “beyond the letter of the law.” If so, perhaps this is the human being’s advantage (the “image of God”) over the machine.
If that is correct, then there may be some justification for these “phobias.” It would seem that every person would prefer that if he is already sitting on the defendant’s bench, for example, the one on the judicial bench be a flesh-and-blood judge who also has emotions (provided he is not thoroughly wicked), and not an autonomous “judge-bot.” Similarly, it may be that we prefer a government of human beings who also have emotions and can (if they wish) also act beyond the letter of the law, for “Rabbi Yohanan said: Jerusalem was destroyed only… because they based their judgments strictly on Torah law and did not go beyond the letter of the law” (Bava Metzia 30b); and see Ramban on the Torah (Deuteronomy 6:18), but this is not the place to elaborate. According to this, there is no obstacle to creating autonomous cars, but we would not want to hand over every question—and certainly not the running of the state, etc.—to the decision of robots.
By the way, I now recall that many years ago I saw some horror film depicting a takeover by intelligent robots, etc., and if I remember correctly, the filmmakers built the sense of horror on the fact that the robots were utterly emotionless.
But as stated, all these are only preliminary reflections and still require further study. (For example, one must define what an “emotion” is; is it really impossible to program a machine “to be moved”? And other questions that I do not know how to answer and that are far from my field of expertise.)
The link to the article by Lieblich and Benvenisti doesn’t work.
Apparently one of the reasons for the opposition to autonomous weapons is the inability to assign real blame in borderline cases.
For example—if a terrorist is shot and wounded while charging, and afterward a soldier shoots him while he is lying on the ground, the soldier can be jailed on the grounds that he did not really fear a hidden explosive but acted out of vengeful motives. But if an autonomous weapon does this, it will be possible to inspect the logs that were recorded and prove that the machine really concluded that the terrorist still posed a threat. Then whom could one blame? Certainly not the programmer, because after all one cannot prove that the machine’s conclusion was unreasonable.
(This is of course a weak argument, but I wouldn’t be surprised if it has a psychological effect.)
Obviously. It’s just a paraphrase.
Indeed, as you wrote, extremely weak. Clearly it is possible, but the question is whether it’s worth risking soldiers and innocent people (assuming the machine gives better results) just so there will be someone to blame in case of an error.
I fixed the link.
A paraphrase of what?
Moshe, with respect, this is not very complicated.
I wanted to say that people have a tendency to show that everything is found in the Torah. For that purpose they use the expression “Turn it over, for everything is in it,” which indeed is its accepted meaning (though by no means universally agreed upon). By way of a sharpened formulation, I said that in my view everything is indeed in it—but only in the sense that one can derive anything from it, which means that one can derive nothing from it.
I am certain that machines cannot be programmed to feel emotion, but they can be programmed to act with mercy. That is a behavioral requirement, and after all that is what matters.
I understood perfectly. In other words, you see the mishnah’s statement—or at least the words of the Rishonim such as Rabbenu Yonah—as a cynical witticism. And since I have not found other interpretations, I ask again: how do you interpret the mishnah?
It seems to me that you still haven’t understood. Who spoke about the Rishonim or about the mishnah?! I used the mishnah’s terminology as a paraphrase to reiterate my approach that one should not extract non-halakhic principles from the Torah.
Hi Dr. Abraham, I follow your posts semi-regularly, and as always I enjoyed reading you. However, I have a number of challenging questions that I would like to add.
As someone who works with technology daily for a living (though not in the field of artificial intelligence), I feel perfectly comfortable saying that I admire technological progress and am a big fan of gadgets. (In short, I see no reason to suspect me of bias against technology.)
Even so, in my opinion there are fundamental problems with artificial intelligence that people are missing.
A. In the car example, precisely my concern is the inability to understand the “thinking” / analysis processes of artificial intelligence. There is an entire academic field trying to understand how exactly the final AI works after training. (I am of course referring to the field of deep learning.)
What worries me is the entry of such systems into fields like medicine. (It has already been shown that AI succeeds in making better diagnoses than doctors in certain cases.)
My fear is not of using them, but of over-reliance on them.
I think it is reasonable to assume that in another 200 years, almost every stage of medical diagnosis will be done by machines, with no human oversight at all. The advantage of machines over human beings will only keep growing, until it will no longer be worthwhile to employ human doctors for this purpose.
What worries me is what will happen if one day, after 150 years of continuous flawless operation, these systems suddenly begin malfunctioning without warning. That’s what frightens me.
1. The people who will be harmed before anyone realizes that the fault lies in the AI itself and not in mistakes around it.
2. The fear that we may not be able to go back, after our whole lives are built on smart technologies that think for us.
In my opinion this is an important consideration that must be taken into account. Isn’t there a legitimate fear here that is unique to this field?
B. Regarding weapons, this is somewhat an extension of A. Unlike one person, or a group of people, whose attention is limited, who need to eat and sleep, who are relatively predictable and with whom one can negotiate—machines can be too fixed on their goal for you to be able to negotiate with them at all; that is what makes them reliable.
What happens when a weapons system holds an arsenal of game-changing weapons? What happens when over decades such systems lose more and more expensive control mechanisms (because they have proven their reliability), and then suddenly they lose it? Isn’t there a difference here in that this is the first time a weapon can carry out an entire mission by itself? A bomb will not explode itself, but a weapons system can theoretically operate entirely on its own, and this can happen unexpectedly, without early warning signs. Isn’t that a justified concern?
Perhaps I didn’t understand. I am now asking a new question: in your opinion, what is the meaning of this mishnah?
And again, sorry for getting hung up on a marginal point in this wonderful and important article. You can move our entire correspondence to the Q&A page.
Hello Ben.
I completely accept that. These are substantive arguments and they should be considered. I was speaking about the phobias hidden behind the substantive arguments. One certainly must take real risks into account. As with any weapon or vehicle. Even with game-changing weapons, I am not at all sure that putting them in human hands is safer.
By the way, these phobias also exist among clear technology enthusiasts. In fact, sometimes they exist even more strongly among them, because in artificial intelligence there is a tendency to see robots as people and vice versa. Therefore the fact that you like gadgets and the like is irrelevant to the question of phobias. There is another phobia (such as one I have), namely an aversion to using technology and an inability to manage with it. That’s something else.
One can interpret it broadly, that the Torah contains everything. And in my opinion that is plainly incorrect. And one can interpret it more narrowly: that it contains everything needed in order to serve God.
I thought of something else. Pay them 15 million dollars and then fine them for every unusual incident:
An incident at the fence 100,000
Cutting the fence 500,000
A missile 1,000,000
Destruction of a tunnel 5,000,000
Three months of quiet bonus 2,000,000
You’re on the wrong article. This belongs to the previous column.
Sorry. I was mistaken. It actually does belong here.
They have an obsession with digging tunnels; we have an obsession with producing countless Shabbat pamphlets. We’ll reach an arrangement with them whereby their tunnels will be used to store worn-out Shabbat pamphlets awaiting burial. They will receive payment for each bag: 5 NIS for a small bag; 8 NIS for a medium one; 15 NIS for a large one9. Valuable real estate will be freed up for us, and everyone will be happy.
Regards, S.Z. L.Tz., extraordinary adviser to the Minister of Peace
Rabbi, I understand that Yossi’s proposal is brought mainly as an anecdote for the article, but I still have a question that bothered me a bit about it—the Rabbi said that this is thinking outside the box, and I didn’t understand why. Isn’t the proposal simply the usual proposal to intensify the response that is given today in order to create deterrence (a weak policy, etc.)? It may be that I didn’t understand the point about shifting responsibility, but since the State of Israel is the one operating the system and has the ability to stop it, it is obviously the one responsible for the destruction of the houses. (The fact that this is done automatically doesn’t seem to me to change the issue of responsibility…) And the point about publishing the list would indeed create pressure on the residents and on Hamas and would frighten those living in the house, but presumably the residents are afraid in any case and simply don’t really have control over Hamas, no? And Hamas in any event already knows that Israel will respond with bombings to every missile attack (let us say…), so again, isn’t the proposal basically just to intensify the response that exists today?
Would you prefer your investment money to be managed randomly? I think yes, because it would yield a higher return over time. That is despite the fact that you are paying for their foolish management. Of course, if it is random already, it would be better not to charge money, but I am willing to give up my feeling for a higher return. Until not many years ago there really was no possibility of investing without human judgment, even though studies showed that monkeys (as a paraphrase, I think) do it better.
First of all, thank you for the response and for addressing the point; I am glad to hear I am not alone in my criticism.
First, a brief comment regarding game-changing weapons and weapons of mass destruction. I assume you are familiar with the doctrine of Mutual Assured Destruction. It always makes me think that perhaps there is some value in certain human beings holding such weapons.
As for the phobias, interesting. Until not long ago I was surrounded by people who are enthusiastic about artificial intelligence and busy thinking about what problems can be solved with it and how it will make the world more efficient. They dismissed my arguments for various reasons.
Perhaps I should conduct a parallel survey in my new environment as well. (Which is also full of technological people.)
Good evening, Rabbi.
Do you have a more substantive refutation of Ramban’s words than simply saying that this is just the reason of the verse and one cannot derive law from it?
After all, if this is indeed the reason for the commandment of kilayim, extending the prohibition to the creation of additional creatures would be reasonable.
R. Michael, thank you for a fascinating article.
Are you really prepared for countries that have the technical capability to install similar autonomous defense systems, which would be aimed among other things at Israel, and which would be activated against Israeli civilian targets in response to Israeli fire (or the fire of any other country), which in practice might actually be launched by a hacker (Iranian or otherwise) sitting somewhere else?
Do you not see the dangers involved in a total relinquishing of human agency in warfare? And do you not understand that such relinquishment will arrive sooner or later in civilian life as well, that is, in the way the police operate? Are you ready for robots to arrest you in the street and take you to a robot court that will decide your fate (by means of algorithms based on neural networks, of course)? Please do not dismiss this with the non-substantive argument of ‘phobia.’ A considerable part of these things are existing technologies, or will mature in the coming years.
In my opinion it has an entirely different effect. See the addition I just added at the end of the column. And the residents definitely do have influence on Hamas, despite everyone’s trying to sell us the story that they are its captives.
Absolutely not true. Random management is a sure loss. Perhaps you mean index funds without management or something like that. That is not random at all.
Indeed, I am familiar with it. When it comes to game-changing weapons, reciprocity really is a very powerful mechanism—assuming, of course, that the players are rational on the one hand (they do not use the weapons in a situation where they too will be destroyed) and brave and resolute on the other (that is, that if it really becomes necessary, one can rely on them to use the weapon. Hesitant democracies that will use the weapon only in extreme situations and after endless hesitation may absorb a severe blow in the hope that they will not use it or will not manage to use it in time). These two are conditions for a balance of mutual threats.
First of all, why do I need to provide a refutation? I can simply disagree because it does not seem to me to be the explanation. Beyond that, in halakhah we do not derive law from the reason of the verse, especially from such a dubious reason. Third, a commentary on the Torah is not a halakhic source. Fourth, where do you stop with the prohibition on intervening? Is there a prohibition on healing oneself? On having surgery? Genetic therapy? In short, this is nonsense (not necessarily the reason itself, but the halakhic implications people derive from it).
See in the link to Steinberg’s article many sources that run counter to this idea.
And if I’m not prepared, that won’t happen? In your opinion, if there is a treaty against such weapons, Hamas will not try to reach a situation where it has such weapons? We do not intentionally fire missiles at civilians. So Hamas doesn’t do that either? You are illustrating exactly my point about these restrictions, which only democracies will impose on themselves, and thus they will finally lose the struggle.
These apocalypses (which are of course completely fictional: why, if the State of Israel builds such a weapon against Hamas, would it also be able to equip the police with such a weapon, and vice versa? This is nonsense, with all due respect) need to be examined against the alternatives.
By the way, when I speak of phobia, that has nothing to do with whether the technology exists. In my remarks I am speaking about existing technologies, and my claim is still that the fear of them is not rational. Not because there is no such technology, but because even if it exists, the fear of it is unjustified, or at least disproportionate.
These jurists reminded me in their distinctions of the distinction between monetary law and prohibition law. They tend toward the monetary side, and you toward the prohibition side.
The idea of index funds is based on the studies on random management. Random management is, in expectation, buying the whole market (with the unnecessary payment of buying and selling commissions), and therefore it will yield exactly the same return as the whole market. Simple mathematics, which a physicist ought to know.
One really only needs to add that the choice is not made with the same weight for every company, but according to its size (even randomness needs to be defined).
If you define randomness, of course you can arrive at almost any result. When people say randomness they mean minimum guidance (random selection of a security with a uniform distribution).
That is indeed random selection of a security with a uniform distribution.
There is no reason in the world to define it according to the list of companies. I have 100 NIS and I don’t know what to do with it, so I go to the stock exchange, see a huge pile of papers each worth one shekel, and choose 100 of them randomly with a uniform distribution among all the papers.
The problem is that the ‘randomness’ you are speaking about in the column is not really randomness. After all, you don’t really think that the cabinet chooses at random, with equal probability, a response from the set of all possible responses. You are merely trying to say that the decisions are foolish and not to your liking, and you want to say it in an intelligent way and call it randomness, and then philosophize about it; and when someone brings you a case of actual randomness, you accuse דווקא that one of not really being random.
I suggest that first you try to define the randomness you claim exists in the cabinet’s decisions.
By the way, even a uniform distribution over the list of companies—I am very doubtful that it would lead to a sure loss, as you claim.
If you had bothered to read, there would have been no need for this discussion. At the end of the column I explicitly explained the relation between this and randomness.
Additional explanation: although a stopped clock is not random, the relation (correlation) between a stopped clock’s reading and the actual time is similar to the correlation of a random clock to the actual time. A fixed policy for changing circumstances is like randomness for all practical purposes.
But in our case it is even worse, because a stopped clock shows the correct time twice a day, whereas a slow clock (with a fixed lag) never shows the correct time at all. A fixed policy that does not work and yet continues all the time in the same way is like a slow clock and not like a stopped clock. This is a randomness that optimizes failure (guaranteed failure). About this, R. Albert Einstein already said that only an idiot thinks that if he applies the same failing policy over and over, in the end it will succeed.
And autonomous weapons have existed for many years; their name is: a mine!
Regards, Yoqish’s nephew
Is it obvious to you that machines cannot be programmed to feel emotions because you are not a materialist? Seemingly, if the materialist psychiatrists and brain scientists are right in claiming that emotion too is nothing but a sequence of biochemical or bioelectrical reactions in the brain, then why should it not be possible (at least theoretically) to program machines to feel emotion?
As for the attribute of mercy—how can one program a machine to activate the behavioral function of “mercy”? After all, we would not want to activate the attribute of mercy in every situation, because then there is no justice; nor would we want to activate the attribute of justice in every situation—because then there is no mercy; nor would we want a robot that is merciful to the cruel and cruel to the merciful, etc. It is not clear to me how one can replace human (or divine) judgment here with robotic programming.
I am not expressing opinions here (because I am not an expert in these fields), but only raising questions. The point that troubles me is: what, after all, is man’s advantage over the machine in the realm of judgment? Or perhaps man’s advantage over the machine is nothing, for all is vanity?
As for randomness—an amusing anecdote. Many years ago (when I was still a student) I read about a “business game” experiment conducted in Japan. (A business game is an exercise conducted in business-school departments. Each student receives X virtual dollars and is supposed to invest in the capital market according to his own judgment. Afterwards they analyze the portfolios the students built and compare return, standard deviation, etc.) In that experiment, in addition to the students there also participated a monkey that received a treat every time it “invested” the “dollars” allocated to it. (“A mere monkey act,” in the plain sense.) It turned out that the portfolio “managed” by the monkey yielded the highest return (I do not remember whether it also had the lowest standard deviation). Would you appoint a monkey to manage your pension portfolio?…
By all views, emotion is not a chemical reaction. Anyone who says so, even if he is a materialist, is confused. At most, emotion is a result of chemical processes. Emotion is a mental event, except that according to materialists the mental is merely an outcome of the physiological and does not require positing the existence of another substance (a spiritual one).
As for mercy, of course one does not apply mercy in every situation, but one applies it in light of various considerations (which are not necessarily considerations of law). Those very considerations can in principle be imitated by a machine.
Obviously I would appoint a monkey, and people indeed do this (they manage their investments by software or by some kind of random lottery. See my exchange with Yishai). In a place where randomness yields good results, why not entrust my fate to it? It is well known that portfolio managers do not beat the market, and therefore entrusting your fate to the market is a perfectly sensible and reasonable policy. A properly designed lottery does this.
By the way, such an experiment of course proves nothing. The question is, if many such experiments were done, what would the monkey’s expected result be? That is what matters.
Hello!
As far as I am concerned, the fear regarding misuse of artificial intelligence is misuse by criminals, terrorists, and totalitarian governments.
A similar process happened, for example, with missile technology. Once, only states developed missiles as weapons, because development and production were expensive. But today this technology is cheap and widely known, and every insignificant liberation movement produces missiles for itself and uses them for extortion and threats (we saw this this week: give us money, Hamas said, otherwise we will fire missiles, and it will be very hard for you to stop this).
Now imagine what will happen when a drone or a swarm of grenade-carrying drones can attack targets autonomously. It will be very hard to stop them (if they are autonomous, harming those who sent them during the act will not help). This will not be difficult, because the technology is becoming more and more accessible.
This will open the door to a new tyranny by terrorist organizations, or even by ordinary criminals who will use the technology for extortion.
The autonomous drone is only one example. And another example: cyberattacks, which already occur today, are also a kind of autonomous weapon, already causing damage and liable to cost us dearly in the future.
It isn’t all that far away…
I do not understand where I was unclear. I did not say that the problem is that there is no danger or that the technology does not exist. I argued that the fears are exaggerated and are not specific to autonomous weapons. If nuclear weapons fall into their hands, is that fine?! I further argued that such international agreements cannot prevent organizations and states from developing such weapons, and the result will be that only the bad guys will have them and the good guys will not.
Terrorism is costly and dangerous. That is not a great innovation. The question is whether that justifies the absurd arguments regarding autonomous weapons.
Regarding weapons—many game-changing weapons were created in order to save human lives or out of mercy: atomic warfare (to decide the Second World War—it succeeded), chemical warfare (to decide the First World War—it did not succeed), the machine gun (by an American military doctor who was frustrated by the need for soldiers to expose themselves to fire).
Is there no concern that a game-changing weapon will only increase the risk?
Admittedly, here we are dealing with a weapon that is not only stronger but also fairer, because it does not make human errors (in addition to being more efficient)
And regarding the fears—in one of Douglas Adams’s books there is a scene (as part of an animated film) of a pilot who activates the automatic pilot in a plane that is about to crash, and then a robot comes out, takes the last parachute, and jumps out—it is basically doing the reasonable thing, but not for the benefit of the person who activated it. With ordinary machines this cannot happen, because they are programmed according to Asimov’s laws of robotics to prefer human life. But is there not some risk that learning machines will also learn things that we do not want them to know?
With God’s help, on the eve of holy Shabbat, Parashat Vayera: “Jacob saw Laban’s face”
Although a human being too can err, and err greatly, the human being has an advantage in that he has intuition and not only rigid rules, and therefore he can quickly identify errors and quickly find a creative way to correct them, or at least to minimize damage.
For example, in one of the wars our soldiers found themselves in a situation where soldiers from another unit thought they were enemy forces and began shooting at them. One of the soldiers stood up, took off his tallit katan, and waved it high in the air, and thus the soldiers in the firing force understood that he was a Jew and stopped shooting.
That is man’s advantage: he has intuition that enables him to recognize situations that were not foreseen in advance, and to find creative solutions for them.
Regards, S.Z. Levinger
It is possible to propose a combination of the advantages of autonomous weapons and of human beings together:
On the battlefield the autonomous vehicle would stand and thereby spare the life of the soldier operating it from danger, but far away in a protected place a human being would stand and supervise the actions of the autonomous vehicle, and when necessary he would intervene and send the appropriate instructions so that the autonomous vehicle would not cause a disaster.
After all, this is how the Holy One conducts His world: on the one hand He allows the laws of nature and human beings to act, and on the other hand “His eyes are open to all the ways of man,” and He guards lest the “freedom granted” bring about the corruption and distortion of the world.
Regards and Shabbat shalom, S.Z. Levinger
By the way, speaking of a model series, I don’t know whether the Rabbi has heard of a series called Black Mirror, in which all the possible scenes and situations of the harms of future technology are presented. One of the recurring motifs there is the possibility of copying a person’s D.N.A into some electronic/technological device and playing with it as one wishes, including horrifying abuses. The creator of the series argued that every addiction has a side effect, and there must also be bad consequences here. Again, one can always claim that this is a phobia, but the matter certainly does provoke thought and should be considered.
An article in Calcalist on the subject:
The decision machine: who wants an AI-based politician?
https://www.calcalist.co.il/internet/articles/0,7340,L-3743193,00.html
It seems to me that here too one sees the author’s misunderstanding of the nature of AI. He thinks this is software that implements considerations programmed into it by human beings, and he does not have before his eyes a learning neural network. (True, even in a learning system, the feedback that teaches it is created by human beings.)
Sorry for digressing from the main point of the article. Still, you often repeat that a text from which one can derive anything has no meaning at all. And this time you even cited the mishnah in Avot, “Turn it over and over, for everything is in it.” The commentators understand this literally, that all forms of wisdom are found in it. Do you regard the mishnah’s statement as a non-binding aggadah, or do you interpret it differently?