Probabilistic expectation. My question is whether, mathematically speaking, the Gemara’s preference for one of the two possibilities is justified.

I don’t understand the question. The option in the Mishnah makes it possible to solve the problem completely. By contrast, in the option mentioned in the Gemara it is clear that some of the levirate marriages were not done with the correct yevamah. What does that have to do with a difference in probabilities? Probability of what?

There are two extreme possibilities:
A. One of the brothers “does levirate marriage” with each of the women, and the other brothers perform halitzah for each of the women, so that only one of the brothers (the one doing levirate marriage) will definitely fulfill the commandment of levirate marriage (with whichever woman is actually his yevamah).
B. As described in the Mishnah, so that it is possible that each of the brothers will fulfill the commandment of levirate marriage.

So one can calculate, for each of the possibilities, the expected value for fulfilling the commandment of levirate marriage, and then it will become clear whether the Mishnah’s preference for the second possibility is probabilistically justified.

I explained above why that is incorrect, and I’ll repeat it. The Gemara says explicitly what its consideration is in preferring the Mishnah’s approach, and it has nothing to do with probabilities. In the Mishnah’s approach, there is a possible situation in which they all do levirate marriage with the correct woman (and it is also possible that they won’t), regardless of the probability that this will happen. In the Gemara’s approach, that is impossible, and therefore it is rejected. No comparison is being made here between the expected numbers of correct levirate marriages in the two approaches. This is similar to what the Gemara says at the beginning of Bava Metzia: “the division can be true.”

Still, I was interested in knowing the expected value of each of the possibilities. I’d appreciate it if you could write out the calculation.

If we follow the Mishnah’s approach, then each brother does levirate marriage with a different woman. Now we need to decide what question is being asked: how many, on average, will do levirate marriage with the correct woman, or what is the probability that all of them will do it correctly. Those are different questions. The second question is not relevant to the Gemara’s proposal, because there the probability of that is 0.
As for the first question, in the Gemara’s framework the answer is of course 1. As for the Mishnah, the answer is not simple (it’s a fixed-point distribution). If you’re interested, you can look here:
https://he.wikipedia.org/wiki/%D7%91%D7%9C%D7%91%D7%95%D7%9C_(%D7%A7%D7%95%D7%9E%D7%91%D7%99%D7%A0%D7%98%D7%95%D7%A8%D7%99%D7%A7%D7%94)

Actually, regarding the expected number of hits, the answer is simple: 1.
See the explanation here: https://math.stackexchange.com/questions/783623/fixed-points-in-random-permutation
Note that this is also the result for the Gemara’s proposal. That of course strengthens the Gemara’s claim that it sees these two as equivalent possibilities, and the Mishnah’s preference is only because it can be fulfilled completely.

Continuation from the email:
Following up on the discussion on the site, attached is the graph.
The expected number of levirate marriages in both possibilities is 1. Do you have any understanding (beyond the Gemara’s reasoning) of what underlies the Sages’ preference for the uncertain possibility?

This is the graph:

https://drive.google.com/file/d/1EMdf2evwylsp4JNLMpFansrosyqag27a/view?usp=sharing

I didn’t look at the details, but why does that matter? The Gemara explained its preference well enough. Beyond that, the graph here isn’t the relevant graph, but rather its derivative (or the graph of differences, in the discrete case). The relevant graph is how many levirate marriages there are at least in each situation (which is the area under this graph). For example, the probability of four levirate marriages is 0, but not because there is no case of four levirate marriages; rather, because if four got it right, then the fifth necessarily also gets it right, so there is no case of exactly four. The relevant graph is of course monotonically decreasing.

Thanks for the comment. From the Sages’ choice of the uncertain possibility over the certain possibility, even though both have equal expectation, can one infer that they prefer risk-taking?

I didn’t understand. They chose the option that can be complete, but there is no certainty there.

Maybe you meant to say that it is preferable to act in a way that may achieve the full result, even though there is a concern that none of them will perform levirate marriage correctly, over the possibility that there will certainly be one correct levirate marriage but it is impossible for all of them to do it correctly. That certainly does exist here.

A more conservative approach would have been to settle for one certain levirate marriage, rather than the permissive approach in which in more than 1/3 of the cases there is no levirate marriage at all. That implies that the Sages’ approach is one of taking risks.

In order to verify your comment about the graph, are you claiming that it would be better to present the graph:

Σp(i)-Σp(i+1)
where:
i = 0, 1, …, N-1
where N is the number of women

That is too sweeping a conclusion.
As for the formula, I didn’t mean that. On the contrary, the probability that there will be at least 3 levirate marriages is the sum of the probabilities that there will be 3, 4, and 5. In the graph, p(n) describes the probability of exactly n, not the probability of at least n. That is not a relevant quantity for our discussion. The difference you wrote here, in my opinion, does not give anything relevant.