Social-Political Polarization: B. A Mathematical View (Column 451)

Introduction

In the previous column I used an article by Elizabeth Kolbert in The New Yorker to describe the phenomenon of social polarization, where the main issue is not our drift toward extremes but the formation of “mega-identities,” that is, the emergence of correlations between views and traits that shouldn’t be related. In such a situation, a person’s political identity dictates their positions on all the questions on the agenda. We saw there that even taste in television shows differs radically between voters of the two parties in the U.S. They also tend not to date and form romantic ties with one another, and as time goes by their views on every question on the agenda become increasingly aligned with their party identity.

I noted there that two insights arose for me upon reading. The first concerned the knotty situation in which we are dealing with a truly wicked person, which I discussed in the previous column. The second insight concerns a possible mathematical explanation for the phenomenon of polarization. To understand it, I will preface with a review of the K-means algorithm, which I only now learned in a machine-learning course I took (you can watch the relevant lectures here).

A brief preface on machine learning

In machine learning, our goal is to teach the computer a skill or a way to make certain decisions. To illustrate, I’ll use the hackneyed problem of distinguishing between regular email and spam (junk email). If we want to teach the computer to distinguish between these two kinds of email so it can discard the junk, we have two principal ways to do this:

1. Supervised learning. We give the computer a set of emails that are already labeled as spam or regular email (we ourselves decided which is which), and from this the computer is supposed to learn how to distinguish between the two. After it has trained and learned, from then on the computer should filter the incoming messages. This is supervised learning because we control the data from which the computer learns before it starts operating.
2. Unsupervised learning. We give the computer an unlabeled set of emails, and it must decide which are spam and which are regular email. Here there is no training phase; the computer proceeds directly to the classification work. This is unsupervised learning because we are not involved in the data itself (we do not control its learning processes).

The second problem sounds like science fiction. How is the computer supposed to understand what junk email is without our teaching it?! Indeed, one could just as well ask how we ourselves do this. At least at the outset, we too have no data and must decide which email is junk to be ignored and which is not. We also learn in an unsupervised way from our experience, and seemingly the computer can do the same. But a computer is not a human being, and as we shall see, even in this case we help it by supplying certain features of junk email that can help it identify it.

In problems of the first kind, the computer is supposed to extract from the labeled examples the relevant parameters that characterize junk email and henceforth use them. Of course, any given email may have many features, some of which are irrelevant to the outcome. The computer is supposed to “understand” which features are relevant from the supervised labeling we present to it. For example, it may notice that every spam email has many recipients, or that each contains purchase offers (prices, sums of money appear), and so on. These are, of course, not decisive features, since they can also appear in regular email, but a weighting of several relevant features can give the computer a very respectable predictive ability (each of us knows from our own inbox how well computers do in this domain—quite well, actually).

But in problems of the second kind the computer lacks supervised guidance. So what can it do? This is what is called a clustering problem: dividing the emails into two groups that are similar within each group, and assuming that one is spam and the other not. For this we must feed the computer relevant features (here it cannot extract them by itself from the labeling, or even choose which of the features matter and which do not. Here we do the work for it). In the junk-email example, we instruct the computer to check, for each email, several parameters: the number of words, how many recipients, how many times a price appears, and so forth, and then to build a table of these features for every email. Suppose we consider four features relevant to distinguishing spam from regular email; then each email has a vector of four features. For example, email #1 has 90 words, 5 prices, 300 recipients, and 10 words that repeat more than twice. We do the same for the second email, the third, and so on. If we have a thousand emails to classify, we create a 4×1000 table where row n contains the four features of email n.

Our goal in unsupervised learning is to divide these thousand emails into two groups according to similarity across the four features so that we can identify the two groups as spam and regular email. For this task one sometimes uses the K-means algorithm.

The K-means algorithm (K-means algorithm)

I’ll preface by noting there is a similarly named algorithm (see about it here) whose goal is to find the locations of K distinct centers to allocate goods to stations scattered in space optimally. The aim is to find a spatial arrangement of centers such that the sum of distances to the stations is minimal. The problem I discuss here is very similar but older (from the 1960s) and concerns clustering in unsupervised machine learning. You can read about it here.

Returning to the email classification mentioned above, we have a clustering problem of dividing the thousand emails into two groups. The K-means algorithm says we should represent each row in this table as a point or a vector in four-dimensional space (whose axes are the four features). In our case, the first email is represented by the point (90, 5, 300, 10), and so forth. Any two locations in this space can be defined as centers (centroids), which we’ll call a and b, and each such pair of centers can define a partition of the emails (points) by their distance to the two. Points closer to center a will be classified into group A, and those closer to b into group B. However, there is no reason to assume that a random choice of positions for the pair of centers will yield a partition that reflects the division between spam and regular email, since this is an arbitrary split. There are infinitely many pairs of centers, and a random choice among them will yield nothing.

To arrive at the “right” partition, or at least close to it, we must see whether there is a partition where the similarity within each of the two groups is “natural” in some sense. We must choose a pair of centers in the four-dimensional space such that the partition they induce on the set of a thousand points is natural. The basic claim here is that the same pair of centers that yields the optimal partition of the thousand points is the one that produces their natural classification, and we identify this with spam versus regular email. What is that optimal partition? We seek a pair of centers such that the sum of distances of the points in the group belonging to each center is optimal, i.e., minimal compared to the sum of distances for any other pair of centers. Why are distances the important similarity parameter? Because we chose the axes (features) to describe the parameters relevant to characterizing spam. If there is proximity along these axes, that means the emails belong to the same group in terms of these parameters, and therefore it is likely that this partition is a good representation of the division between spam and regular emails.

How do we find this optimal pair of centers? For our purposes the key is not the problem and its solution but rather the algorithm I will now define. What we do is arbitrarily choose two points in the space as initial centers, call them a1 and b1. We now divide the thousand points into two groups according to proximity to these two centers. We now have two (not necessarily equal-sized) groups of points, denote them A1 and B1. In the next step we compute the center of each group (their mean position) and define the two points obtained as the next centers: a2 and b2. Now we repeat the process—divide the thousand points anew according to proximity to the new centers—and obtain two new groups (which are indeed expected to differ from the previous ones): A2 and B2. We now compute the mean position of each such group and define the next pair of centers: a3 and b3; then form two new groups of points according to proximity to those centers, A3 and B3; and so on ad nauseam.

In many cases we will get a convergent process, meaning the groups become increasingly optimal (the total distance from the points to their centers decreases). In that case, the process stops at step n, when we obtain the two centers, an and bn, and the two surrounding groups of points, An and Bn, which do not change in the next step. At that point we stop, and these are the optimal centers we sought.

It should be noted that even if the process ends, there is no guarantee we reached the optimal point (it could be a local minimum, i.e., just a fixed point of the transformation), and as far as I understand there is also no guarantee that the process will converge at all at some stage to a minimum (even a local one) and stop.[1] But as a matter of fact, in many cases it does work, and it turns out that in quite a few cases this is a reasonable way to predict the two desired clusters.

Interim summary

I described the process for the case of a split into two centers (K=2). Of course, such processes can be performed for any number of centers we define. For example, we can take height and weight data of T-shirt consumers and represent each customer as a point in a two-dimensional plane. We can then divide the points into three optimal groups to decide which sizes will define the three types of shirts we produce (small, medium, large). This is an application of the algorithm with K=3. In the email example I focused on the case K=2 because, for our purposes later, that will be the relevant example.

I recommend seeing here some simulations of this process in two-dimensional space (I suggest choosing two clusters for simplicity).[2] You can also watch a YouTube video that presents the topic simply and nicely for higher dimensions. If you follow these simulations, you will see the gradual process of forming the two centers. You’ll see that at each stage the centers move through space until their final positions, and of course at each stage the surrounding groups change and reorganize accordingly. It is important to remember that each such center point defines, at every stage, an approximation to the typical feature set of spam (the a-series) or regular email (the b-series). As the process advances, the combination of features at the center changes gradually, and of course so do the groups of points (emails) around it, defined by their similarity to it. Thus a1 is an arbitrary combination of features, but the next combination, a2, is already the beginning of forming the typical combination. The final combination, an, is our final prediction for the typical combination of spam features. The combination bn represents, of course, the typical combination of features of regular email. Around them are clustered the points similar to each (i.e., close to them in feature values), and this is the outcome of the process for the two corresponding groups of emails.

We can now move on to a proposed explanation for the political polarization processes described above.

Back to political polarization

As we saw in the previous column, the explanations offered for polarization processes usually invoke psychological planes, namely the movement of public opinion over time toward polarized directions. Various mechanisms and explanations were offered there (such as the impact of social networks), and I’m sure there are others. But in light of what we have seen thus far, I thought we could offer a mathematical explanation unrelated to psychology. To highlight its different nature, I will assume for the sake of discussion (an implausible assumption) that voters do not change their minds at all over the course of the process. We will see that nevertheless, along the way the groups organize themselves in such a way that polarization necessarily emerges. If I can convince you that this is indeed what happens, then this process is not psychological but mathematical-political.

Consider a democratic process of repeated elections every few years between two parties, as in the U.S. Suppose our voters are represented by a set of views (=features), and each voter is represented by a point in a high-dimensional space (whose axes are the features of political views: position between capitalism and socialism, hawkishness on security, attitude toward the judiciary, religion and state, and so forth). In the U.S. there are two dominant parties, each trying to recruit as many voters as possible. My assumption for this discussion is that each voter chooses the party whose platform is closest to them (not only do the inhabitants of our Utopia not change their views, they also act according to reason and values. They have no gut and no emotions. In fact, this island is not called “Utopia” but “Asperger Island”). You can see that we’ve created a two-center problem here (each representing the views offered by the Democratic or Republican parties, denoted a and b) that partitions the voters into two parties. Let us now look at the democratic process and see that it fully parallels the process of forming centers as described above.

At the first stage two parties arise with some platform. For our discussion we will assume this is an arbitrary beginning, i.e., it does not necessarily relate to the voters’ views (since they have not yet been tested at the polls). The two views/platforms are represented by two centers located at points a1 and b1 in the high-dimensional space. Each such platform recruits to its side the voters whose views are closest to it, and accordingly wins their votes in the first elections to be held there (after the American War of Independence, in the late 18th century). Now the two centers have formed around them two groups of voters, A1 and B1 respectively. But if you now check the mean location of each of the two groups, you will discover that it is not at a1 and b1 but elsewhere. We compute them and denote them a2 and b2. In such a situation, ahead of the second elections, it is clear that the views a1 and b1 are not optimal for the two parties, since they do not necessarily fit the views of most voters.

What happens now? In principle, a new party could arise, of course, but in the American system the mechanism is that a new candidate arises within each of the two parties who detects the potential and positions themselves (their platform) at the new locations that have emerged. At this stage there are primaries among the party’s voters, namely among the members of groups A1 and B1. Note that those participating are the members of the groups from the previous partition still.[3] They will of course choose the second candidate rather than the first (unless the first changed their positions as is customary in our regions, and then the first themselves will be the second), since the second is at a more optimal location for them. After the primaries there are elections, and the new candidate tries to recruit additional votes from the entire electorate.

What happens now? The voters, of course, choose between the two new candidates, and therefore the election results repartition all voters into two new groups, A2 and B2 respectively. These are the groups appropriate to the new centers a2 and b2. But now again there is a gap that requires a new positioning of the centers. New candidates choose platforms a3 and b3 and are elected in the primaries. In the general elections to be held for the new candidates, groups A3 and B3 will form. The process, of course, repeats itself over and over; at each stage the centers shift and the groups cluster around them in a slightly different way. All this continues until Bibi arrives. He is the fixed point of the process and remains forever (until a legal-political assassination of a sitting prime minister occurs).[4]

Note that at each stage of this process, greater internal cohesion emerges among views that more naturally fits the natural division of voters into two groups. The center (i.e., the party platform) shifts slightly each time until it reaches a state that gathers the citizens around it in a “natural” and optimal way. The process should end when optimal candidates emerge who manage to capture the votes of most voters, i.e., whose views fit most of their potential voters. This is the stable state and, in principle, it should not change. Needless to say, it never really stops because the picture is more complex. New axes and new challenges are constantly being added that require a position to be formed about them as well, and the space changes. But this mechanism is sufficient for me to provide a model that describes one of the vectors in the political process of polarization.

If at each stage of this process coherence increases within each of the two groups and they fit more optimally to their “natural” center, then we have precisely the process described above as a “mega-identity,” namely, correlation across different axes. Suddenly party membership, Democratic or Republican, radiates onto views across all spheres of life (across all the axes and features of socio-political agendas). In voter group An, which is close to the ideological pole an, there are now people who like the same movies, form romantic ties with the same people (and not with members of the other party), and think similarly about abortion, the economy, security, the judiciary, religion and state, and much more. As this process advances, the positions of all party members on all axes become more synchronized with one another (it is, of course, far from merging, but the correlation intensifies).

A bird’s-eye view of the process and its implications

Note that all this happens even if no voter has changed their position. The re-drawing of the front lines of party platforms and the voters themselves into two groups creates this polarization. This is how the two “mega-identities” described above arise. The front lines organize themselves to achieve maximal polarization, not in the sense of extreme and very different views, but in the sense of correlations across different fields of opinion (correlation across axes). That is, this is a process that is not psychological but mathematical-political. It is not based on dynamics in voters’ views but on dynamics in the partition of the given voters (even if they hold frozen views) into two groups.

Note that as a result of this mechanism, the two polarized views do not necessarily move farther apart (i.e., become more extreme). In fact, the exact opposite is expected. Since each party strives to maximize votes, it strives to be as close as possible to the majority of voters (the majority of points in the space). It is no wonder, then, that the positions of the centers (=party platforms) move closer to one another.[5] The two parties strive to position themselves at the mean position of all voters (not only of those who traditionally vote for them), meaning they circle around the very same place. This is a possible explanation for the familiar paradox: the closer the positions are, the more the attitudes polarize and the hostility grows. An inverse relation arises between the distances in views and the intensity of hostility and polarization between the groups of voters who hold them. The smaller the distance between them, d(an, bn), the greater the hostility between the groups An and Bn.

If you look at today’s political map in Israel, you will find it difficult to point to ideological differences between Likud and Labor, New Hope, Yesh Atid, Blue and White, Lieberman, Yamina, and the like. Almost all parties and Knesset members crowd together at an almost uniform center, and the differences between them are truly minor, if they exist at all. No one can point to the differences between them, simply because there are almost none. It is no wonder that Knesset members move freely between parties according to the advancement and electability offered to them. They don’t even need to change positions or platform to do so. It is common to blame such MKs for lacking ideology, but that is not correct. They do not need to change ideology to move to another party, since there is no ideological difference between the parties. And at the same time, as perceptual and ideological gaps narrow, the hostility between the sides reaches unprecedented heights. The enmity focuses on trifles belonging to the margins of the disputes, and these are magnified as if they were momentous disagreements. Each side sees the other as malicious, and in the absence of an ideological difference, the conflict and enmity focus mainly on the personal plane (pro- or anti-Bibi/Trump).

Incidentally, what is currently called, sometimes derisively, the “change government,” is a historic event from this vantage point. They essentially tried to break the dichotomous division into “mega-identities,” and to connect parties with different platforms (with the common denominator being that they do not want bn, or bb, of course). No wonder the dinosaurs built by polarization on both sides accuse them of leftism on the one hand, or capitulation to the right on the other. These accusations try to forcibly preserve the “mega-identity” and the polarization arising from it. This is similar to clinging to the horns of the altar of polarization between religious “mega-identities”: religious-Zionist and Haredi, which also no longer really exists. Its preservation is in the interest of those who feed off this polarization (and of course they always speak of “unity of the camp”). Instead of dividing along real poles—Haredi (Zionist or not) versus liberal—people prefer to divide along the axis of Zionism versus anti-Zionism, which today has no real content.

Returning to the process of political polarization, note that this entire process can and is expected to occur even if throughout it no one changes their position on any issue. The “mega-identity” arises due to the organization of voters into “natural” groups with higher internal coherence. This is an outcome of the democratic process itself. Therefore polarization, in this sense (not in the sense of extremism—since extremism actually disappears—but in the sense of the emergence of correlations—the “mega-identity”), is not despite democracy but because of it. Elections and the replacement of candidates (internal elections, primaries) create an organization and consolidation of centers in a way that maximizes polarization (creates a “mega-identity”), just like in the K-means algorithm.

The relationship between the two explanations

I do not intend to claim that the psychological explanations are incorrect. On the contrary, it is quite clear to me that they are correct. But to the same extent, the mechanistic-political-mathematical explanation is also correct. The conclusion is that the process of political polarization is a combination of these two mechanisms together. Polarization is created by the radicalization of views due to echoing within social-network bubbles, and at the same time by a process of center-organization that would occur even if there were no change at all in voters’ views.

The first insight seemed quite discouraging. If everyone is sure they are right and the other is wicked, there is no chance they will be willing to examine themselves and give the other credit. That is a psychological knot. The second insight seems, at first glance, even more discouraging, since ostensibly it is an almost deterministic process. We saw that it is inherent in the democratic structure and our form of government. It is no wonder that the author quotes Stephen Marche (a Canadian writer and journalist cited in the previous column) saying:

“No American president of any party, now or in the foreseeable future, can be a symbol of unity, only of division.” Unfortunately, too many of his declarations sound correct, like “When the crisis comes, the institutions will not be there.”

This is also the feeling regarding our situation here in Israel.

And yet, I thought that at the end of the tunnel of the second mechanism there may be light, since here we are not dealing with a psychological blockage but with a result of a structural-governmental mechanism. Therefore, precisely here I have some hope for improvement and repair, if we can think of a way to change the mechanism of center-organization and the groups around them. I will propose two lines of thought derived from the picture I have described, and due to the shortness of the platform I hope you will forgive the preliminary and non-concrete nature of what follows.

Two principal directions for change

The first direction is to periodically change axes, or add axes to the ideological map. How is this done? For example, by adopting new challenges and placing them before the public. Once there are additional issues that require attention, opinions divide differently, and the process of forming “mega-identities” can be temporarily broken (but of course begin again and strive to reconverge around new centers).

The author of the article also notes that finding a shared task that unites both sides can change the situation. This claim, too, is supported by experiments in social psychology. Incidentally, the Kodkod Games company has a whole series of non-zero-sum games, i.e., games whose success is based on cooperation rather than confrontation and competition (in game theory we distinguish between these two types). We all know the phenomenon that in situations of internal conflict and tension the ruler and/or the public find a common enemy on whom to vent their anger and thus try to achieve internal unity (some will say sweep the problem under the rug, but here we see there is something substantive to it). This is an application of this method, albeit at the expense of an external party. Finding a task that requires internal cooperation without directing it against an external enemy could be more successful and positive, but of course much harder.

The second direction is to think about changing the electoral system. We should seek a mechanism that neutralizes center-organization in a way that maximizes polarization. For example, perhaps we can define, each time, a certain set of questions on which the current elections are held. Perhaps we can add and remove centers, i.e., artificially change the number of parties (changing the electoral threshold is such a process). Although nowadays we have many parties, there are still two fairly distinct sides on our political map (at least until Bennett, Sa’ar, and Lieberman—those wicked traitors—broke the underlying “mega-identities”). Therefore, at least de facto, our political mechanism is the organization of two centers. One could also think about a system where the citizen votes for whom they do not want, or where each person ranks all the parties, and so on.

Can these directions for change be brought about by the politicians who themselves create these processes? Is this not a solution of the sort we saw above (“behave logically and nicely and then everything will be solved”)? At least in some situations, the answer may be negative. They will do so only if it suits their political interest. For example, Israel today suffers from political deadlock (though after the Netanyahu era, no one will remember that anyone here thought about a deadlock. The deadlock is not between right and left but between Bibi and anti-Bibi). In a political tie, perhaps even the politicians will conclude that the existing polarization does not necessarily serve their interest. Changing and reorganizing centers will not necessarily reduce your number of voters (=the points close to you). If you manage to define additional axes that bring most voters closer to your center, or position your center to be closer to more voters, you can gain politically from this process. And then, of course, it will begin anew, and so on and so forth…

[1] As far as I know, if similarity is defined via Euclidean distance between a point and its relevant center, I believe it does converge.

[2] You’ll see that if there are two centers (K=2), the separation between them is by a straight line (and in three dimensions or more we get a plane or a higher-dimensional hyperplane). If there are more centers we get polygonal shapes around each center. There are also techniques to create separations that are not straight (not polygons), like circles or any other shape, but I won’t get into that here.

[3] In the U.S., the primaries of each party are held among the general electorate and not only party members. I assume participation is indeed divided (voters of each party want to influence their party’s candidates), but for our purposes here it doesn’t matter. I’m merely presenting a model to explain the polarization process.

[4] Either because he changes his positions to match the electorate (as the saying goes: “These are my principles, but if you’d like I have others as well”), or because the voters emigrated from Asperger Island and moved to Utopia. Specifically regarding Bibi, I think he does more or less maintain his positions (because he is always elected regardless of his views and therefore does not need to change them to be elected). He may not act based on his views, but that’s another discussion. I am speaking about what the candidate says, not what he does.

[5] Incidentally, this does not happen in the algorithm itself. The tendency to approach all the points in the space characterizes the political game but not the algorithm. The algorithm only seeks an optimal partition and has no interest in maximizing the number of points in each group. If we add that constraint to the algorithm, we get the political game more fully. In that case the two centers will, of course, position themselves at the centroid of all the points.