On Retroactive Proofs: The Writing on the Wall and After-the-Fact Wisdom (Column 87)

With God's help

A few days ago I saw an article reporting that the imam who led the terror cell that carried out the Barcelona attack had been imprisoned for four years (for smuggling hashish from Morocco to Spain—an act of obvious religious significance), and afterward (in 2014) faced deportation from Spain (apparently there is a law there that an immigrant who has served at least a year in criminal prison is deported). He appealed before a court there, and his appeal was allowed on the grounds that deportation would restrict his freedom of movement (?).

From the bizarre description in the article, more is hidden than revealed. As usual, it is unclear how much of this reflects the reporter's eccentricity and how much the eccentricity of the legal system. And, as usual, I did not check the facts, because it does not really matter what actually happened there. For our purposes here, what matters is the headline given to the article on the homepage, which reflects a feeling shared by many (see also the comments, especially regarding the immediate implications for us here in Israel): "The writing was on the wall." Embedded in the subtext of the headline and of the article itself are two claims: 1. The outcome (the attack) proves retroactively that the court that allowed the appeal was mistaken. 2. Lessons should be drawn for similar future cases not to allow such appeals.[1]

Against claim 1, many say that this is hindsight. It could not have been known in advance. In response, others argue that this is precisely why he was about to be deported, because there was a tangible concern that he might commit such acts. So there was already a dispute from the outset: the writing was on the wall. Against claim 2, some invoke the law of small numbers (see here). One case proves nothing. Statistical deviations can occur. Therefore, even if retroactively it turns out that in this case the judgment was mistaken, this is one case and it may be exceptional. That does not give us the ability to draw lessons for other cases. There are two questions here concerning the relation between the writing on the wall and hindsight, and I will touch on them here briefly.

Rabbi Shach on Entebbe

I once read in Mikhtavim U'Ma'amarim by Rabbi Shach (I think in Letter 2) that Rabbi Shach opposed Operation Entebbe because of the high expected risk to the soldiers and the hostages, and the low chances of success. His argument was that it is wrong to take an almost certain danger for the sake of uncertain success. After the operation succeeded beyond all expectations, people came back to him and told him that the operation's success proved that he had been mistaken. Rabbi Shach answered with a very sensible argument: I too did not say that the operation would certainly fail. The claim was that there was a high probability of failure, and it was not worth a high risk to the soldiers' lives. Even when there is a high probability of failure, success is still possible at some low probability. A 95% chance of failure means a 5% chance of success. So now the 5% materialized—what does that prove?! In other words, the decision must be made according to the data available to the decision-makers at the time of the decision. Rabbi Shach claimed that according to those data it was wrong to launch the operation. Its actual success does not prove that the original decision was correct. This is hindsight, and in fact there is here an expression of the law of small numbers. A single case proves nothing on the statistical plane.

I am not entering here into the question of what expertise a Jew like Rabbi Shach has in this matter, and whether he has authority in such areas. My discussion here concerns not authority and expertise but the question of retroactive proofs—namely, whether he was right in his argument or whether his critics were.

Distinction Between Two Situations

In such a context, it is important to distinguish between two different situations:

1. Everyone thought from the outset that it would not succeed, and the dispute was whether it should nevertheless be carried out (despite the slight chance), and now the success is brought as an argument proving that it indeed should have been carried out.
2. There was a dispute between two camps: some thought it had a reasonable chance of success (one that justified the risk), and some thought it did not. Retroactively, the success is brought as proof against Rabbi Shach and in favor of the optimists.

In situation 1, Rabbi Shach is clearly right. The success really proves nothing, since no one claimed that the chance of success was high. But the dispute described there apparently concerned a situation of type 2: those making the claim wanted to adduce proof that the chance was higher than Rabbi Shach had estimated; that, after all, was the point under dispute. At first glance, the arguments in his favor still stand: this is a matter of small numbers and hindsight. Is there nevertheless room here for the critical claim?

Conditional Probability

There is a very common and confusing inversion of probabilistic conditionals. The dispute over the success of Operation Entebbe was really about the a priori question, prior to the event: what were the chances that, if they decided to carry out the operation, it would succeed? The question we are now asking is an almost opposite one (not exactly): drawing conclusions from an event that occurred—how likely is it that, if the operation succeeded, then the original chances of success were high? The relation between these questions is not simple, and certainly not linear, but there is some relation between them.

In fact, the successful outcome of the operation shows that, to some extent, those who argued that the chance of success was high were more right than those who argued it was low. Think of a situation in which we repeated this operation thousands of times and each time it succeeded, or alternatively it succeeded in 80% of the cases. I assume we would all agree that this is a fairly good indication that those claiming such an operation would succeed are probably more likely to be right than their colleagues. This is also how scientific theories and hypotheses are confirmed in light of data. True, in this case we performed only one "experiment," and true, one experiment is not a representative sample and its result may be accidental; still, it gives some support to the view that assigns higher chances of success. If a large number of cases would prove this clearly, then every single case confirms it to some degree (probably a rather small one).

Hence the argument of those who supported the operation against Rabbi Shach has some merit. It is not very strong, but it is not groundless either. Assuming we have only one example (we have no way to repeat the experiment and carry out more similar operations), this is the only datum and information we have. If, unfortunately, we must draw conclusions on the basis of this lone datum, that is the best conclusion that can be drawn from it (its evidential force is of course only slightly greater than that of the opposite conclusion, but there is still a small difference). If we nevertheless have to decide on the basis of this single experiment, this is the least mistaken conclusion that can be drawn from it. Even in Jewish law we follow the majority, even a slim one. The reason is that although it is preferable not to make decisions on the basis of limited or partial information, if we nevertheless must make such a decision, it is more correct to choose the option with the higher probability, even by a small margin. Relying on 51% is certainly better than relying on 49%.

An Example: The Raven Paradox

This reminds me of a question that came up here on the site a few days ago (see here at the beginning of the thread). In response to the question I wrote there, among other things:

This reminds me of Hempel's Raven Paradox (an attack on scientific induction). Take the scientific hypothesis that all ravens are black. Some have argued that examining a raven and verifying that it is black confirms that theory. Hempel replies that the claim "Everything that is not black is not a raven" is logically equivalent to that claim. Therefore, according to the inductivists, it follows that finding a white cloud or a pink table also confirms the theory that all ravens are black. A thought exercise: what is wrong here?

Yishai answered correctly there that in fact examining white clouds or pink horses also confirms the theory that all ravens are black, but because this concerns a large class of objects, the confirmation is rather small.

And if I may put it in my golden prose:

That is indeed the explanation of the raven paradox. Finding a white cloud really does confirm the thesis that all ravens are black, but because of the abundance of cases the confirmation is minute. Therefore Hempel was of course not right in his attack on induction.

Of course examining clouds can refute the claim that everything that is not black is not a raven. If we examine that white object in the sky and discover that it is a raven rather than a cloud, the thesis has been refuted. The examination under discussion is of course not an examination of the color of clouds (given that it is a cloud and I am asking what its color is) but an examination of the nature of white objects (given that it is white and I am asking whether it is a raven). Mark this well.

One can see that here too the reversal of direction is significant. If we were examining a cloud and asking what color it is, that would be meaningless for our purposes. But if we examine a white object and ask whether it is a raven (and discover that it is a cloud), that examination has confirmatory significance, though a rather small one (because there are very many non-black objects, far more than ravens. Therefore one case is a much smaller percentage of them).

As a side remark, I should note that in the discussion of Operation Entebbe it is not entirely clear what exactly the direction of the examination is. Did we take an operation and check it and discover that it succeeded, or did we take a successful operation and discover that it had precisely the characteristics that were under dispute (of course because it was Operation Entebbe itself)?

The Role of Basic Assumptions

In light of what we have said here, should Rabbi Shach really have abandoned his position because of the successful outcome? Not necessarily. If indeed we are in a situation in which, from our standpoint, the two possibilities are evenly balanced and we want to draw a conclusion from an empirical observation, then it is more correct to conclude that such an operation really does have good chances of success (say, above 50%). But if we have a clear a priori view that this is a hopeless operation, then the success will be interpreted by us as a mere fluke—and rightly so.

To be sure, this opens the door to conservatism and intellectual ossification. Everyone will entrench himself in his position and never be persuaded by what others say or by the facts (don't confuse him with the facts). Of course, it is important to maintain honesty and be willing to conclude that we were wrong and retract. It is not right to dig in stubbornly in our initial position. On the other hand, in my view this really is the required conclusion. So what does one do?

It seems to me that Rabbi Shach should have gone back to his original reasoning and honestly reexamined it: was it really correct to infer that the chances of success were low? If he still remained of that opinion, then from his standpoint it would be correct to say that this was a fluke. If he reached the a priori-theoretical conclusion that he had been mistaken, then he must retract—but then that follows even without the actual outcome (which merely provides the motivation to go back and reexamine his a priori considerations). But if he returns to his arguments and reaches a non-decisive conclusion—that is, if his position is that perhaps the a priori reasoning was mistaken and there are arguments in both directions—then the actual result has significance. It is what tips the scales (slightly) in favor of the side claiming that there really is a high chance of success.

Notes in Jewish Law

Even within Jewish law there is a similar phenomenon. When there is an initial presumption, not every argument will tip the scales. A migo (the argument that a litigant could have made a better claim had he wished to lie), though in itself strong evidence, is nevertheless not enough to extract money from someone in possession. Why? Because a migo is indeed evidence, but when there is an a priori argument against it, we regard it as though this were merely a fluke and nothing more. By contrast, two witnesses are excellent evidence, and therefore it is implausible to treat that as mere happenstance. That is why witnesses can also extract money from someone in possession, that is, they alter the a priori position.

Another example. There are presumptions that are created after three occurrences. An ox that gores three times becomes established as a habitual gorer. A woman who experiences bleeding three times on the same date or at the same interval acquires a presumption that this is her fixed pattern of bleeding (for the laws of menstrual cycles). There is a difference between a situation in which the woman or the ox has a prior presumption and one in which they do not. Sometimes one occurrence will suffice, at least to create concern. But if there is a prior presumption, we require the change to recur three times in order to alter it. The same applies when there is a majority: we must examine whether it stands against a state presumed to be otherwise or not. This is not the place to enter into detail about the laws of majority and the various presumptions in Jewish law, but for our purposes I wanted to show that an initial a priori position has standing against empirical arguments and evidence, depending of course on their strength.

Rationalism and Empiricism

The picture I have described here is a proper combination of a priori positions (rationalism) with facts and learning from experience (empiricism). The conclusion is that it is wrong to disparage either of these two components. Some think that the facts determine everything, and anyone who remains in his position is conservative and ossified. These are empiricists in the bad sense, and they are mistaken about this. One cannot rely on facts without intellectual and logical analysis (which may itself perhaps include elements learned indirectly from experience). By contrast, some think that the a priori position is immune to facts (don't confuse me with the facts). These are rationalists in the bad sense, and they too are mistaken. The conclusion is that we must not attach exaggerated importance either to the facts or to our assumptions. It is the combination, in the proper dosage, that yields the most correct result.

Let us now return to the case of Operation Entebbe and the like. Suppose there was a dispute and in the end the operation succeeded. After the success, one side (Rabbi Shach) reexamined his considerations. There are four kinds of situations, each of which requires us to take a different path and adopt a different balance between rationalism and empiricism:

• If my theoretical position is still solid, the required conclusion is that the result was merely a fluke. Here we must beware of the law of small numbers. In such a case it is wrong to accuse the person who reasons this way of conservatism and ossification, since from his standpoint this is indeed the required conclusion. On the contrary, jumping to a different position because of the practical result is hasty and implausible empiricism. This is an expression of balanced rationalism.
• If I discovered that I was mistaken in my theoretical reasoning, I will retract even if the success was only a fluke. This is still an application of rationalist thinking (because the decision is not connected to the facts).
• If I discovered that, theoretically, the matter seems doubtful to me, then the practical result should lead me to change my position (only slightly, of course). This is an expression of balanced empiricism.
• If I discovered that my theoretical and a priori position remains in force, but there is a large number of facts that contradict it (and not just one case), then I must nevertheless change my position. Facts have weight, and one must beware of unreasonable rationalist conservatism.

The dosage—how many contrary facts are needed to break an a priori thesis—of course cannot be fixed in general. It depends on how certain I am of my a priori position and on how significant the empirical findings are (this is precisely the picture of scientific paradigm shift in the thought of the sociologist of science Thomas Kuhn).

Conclusions and Implications

Let us now return to the case with which we opened, the imam from Barcelona. It indeed turned out retroactively that he was responsible for the attack. Does this mean that those who thought from the outset that the court should have rejected his appeal were right? Not necessarily. One must remember that the court operates on the basis of partial information. The attack had not yet occurred, and from its perspective it is making theoretical judgments (rationalist rather than empiricist; it does not yet have the outcome). Therefore it must decide according to what appears at that moment (in the language of the Sages: "according to his present state").

On the other hand, when people come after the attack and criticize the court by saying "the writing was on the wall," one should adopt one of the paths described above. One has to reexamine the considerations and arrive at one of the four conclusions mentioned above (the fourth possibility requires examining additional similar cases). The conclusion is that there is room for this criticism, though it is not absolute. One cannot automatically say that the writing was on the wall, but neither can one dismiss the claims as hindsight (especially if we are speaking about drawing conclusions going forward and not merely criticizing the specific judicial decision from the past).

One must remember that human and civil rights are also at stake here. Therefore, a decision that rejects the appeal carries a significant cost. Set against an assessment of what may happen in the future (whether he will be involved in an attack) are weighty value-based and evidentiary considerations, and therefore one should not hastily conclude that there was a mistake, and certainly not draw sweeping conclusions regarding future appeals. Each case must be judged on its own merits.

Similar questions arise with respect to the release and judicial handling of terrorists here in Israel, as well as in cases of domestic violence. Time and again, cases are publicized in which a woman turns to the police and the court asking that her spouse be removed from the home. The court refuses, and then an incident of violence occurs, sometimes even murder. Once again the headlines return: "the writing was on the wall." Others charge that this is hindsight. Here too one must beware of following slogans and "facts." The facts here are extremely partial (one or two cases say nothing about another case that may not resemble them), and set against them are values whose infringement is grave (property rights, especially if there is concern that the woman is lying, and so forth). In short, writing does not appear on walls quite so quickly. And in general, it is preferable to be a prophet in order to decipher it (cf. Mene, Mene, Tekel, Upharsin). But on the other hand, when all one has is a sage, that is what one has, and sometimes a sage is preferable to a prophet.

These matters raise questions about the meaning of predictions in general. Can the future be predicted, and when? Does what actually happens necessarily refute or confirm the prediction? I plan to enter into this topic in one of the forthcoming columns.

[1] There is, of course, a difficult problem here: what exactly counts as similar cases? Are we never to allow the appeals of anyone accused or suspected of such acts or intentions? That is not reasonable. How far must the suspicion go for the appeal not to be allowed? Any suspicion whatsoever? One must remember that on the other side stand human and civil rights, and concern about harming innocent people or causing disproportionate harm to the guilty. Therefore, inference from such a case to others is problematic in several respects. Our concern here is the discussion of the principled question of retroactive inference backward in time, and therefore I will not enter into the whole range of those questions.