Disputes in the Mishnah against ancient law

In the SD
Greetings to the Honorable Rabbi,
I wanted to ask the rabbi a question regarding the Toshba – how were departments created in tradition and in the Sages.
I have always understood that disputes arose when there was no explicit tradition on how to act, or when a new doubt arose that had not been discussed before. This is implied by an explicit mishnah in Tractate Sanhedrin (11:2). There we see that when we have evidence of how the law acts, then we follow it – "…they come to the one who is at the door of the help, and say, 'Thus I have taught, and thus my friends have taught me, thus I have taught, and thus my friends have taught me.' If they have heard, they tell them; and if not, these and those come to the great court in the chamber of the scribes."
From these things we can conclude that disputes can only arise in matters where there is no tradition or explicit evidence of how the Halacha is practiced.
Thus we also see that the conditions were followed in practice, as is stated in Tractate Edoyot (1:3), where there was a dispute between Hillel and Shammai over the amount of water drawn to invalidate the mikvah. Until two guards from the Collection Gate in Jerusalem came and testified that from the office of Shemaiah and Abtalion, three quarts of water were drawn to invalidate the mikvah, and the sages upheld their words. (And later it is mentioned that they did not stand by their words and accepted the testimony.)
It follows from all of this that when we have a clear tradition in halakhic terms, then we do not dispute it.
 
But I have come across several cases where I do not see any explanation for how they could have disagreed with a ruling that is like the Torah, and it is not at all reasonable to assume that they forgot the tradition in question.
For example, in the Beatings (3:10), we have a generally accepted rule that one should be beaten with 39 lashes, but Rabbi Yehuda disagrees there and says that one should be beaten with forty lashes.
In Tractate Sanhedrin (1:6), Rabbi Yehuda disputes the quorum of the Sanhedrin and claims that the Sanhedrin was seventy, not seventy-one.
In Teruma (3:3), Rabbi Eliezer disagrees and believes that in exchange for guilt, etc., they will die, contrary to the rule given in the Gamma (Gamiri 3:1) that in all cases of sin, death is due to the guilt of a shepherd (Babli Tmura 18).
In the blessings (1,3) they disagreed on whether one should recite the Shema in the evening in a reclining position and in the morning in a standing position. (And it is not reasonable to assume that this was the way of the world because we saw the case of Dr. Tarfon who reclined to recite according to the words of Beit Shammai).
 
Furthermore, I saw a mishna that could serve as a wonderful argument to show a controversy with a huge impact when in that mishna itself, testimony was given about how it was done in the High Court.
In Pesachim (7:1) we find a disagreement between Rabbi Akiva and Rabbi Yossi of Galilee about how to roast the Passover sacrifice. It is important to emphasize that these scholars lived during the time of the High Court and sacrificed the Passover sacrifice in their time.
"How to roast the Passover: A pomegranate skewer, inserted from its mouth to its pointed part; and put its legs and entrails into it, says Rabbi Yossi of Galilee. Rabbi Akiva says, this is like a kind of cooking, but a skewer is cut through it. We do not roast the Passover without a skewer…. Rabbi Tzadok said, an incident in Rabbi Gamliel who said to his servant, "Come out and roast the Passover for us on the spit." Etc.
We encounter here a significant controversy over how the legs and intestines of the Passover sacrifice should be roasted, whether inside the sacrifice or on the skewer. What is interesting in this mishna is that we see that these two conditions certainly did not disagree on the halachic law that requires that the Passover sacrifice be roasted and not cooked. Nor do we see that they disagreed on the fact that the roasting should be done on a skewer made of pomegranate wood (which is dry). Later in the mishna we see that Rabbi Tzadok testified to how Rabban Gamliel used to roast the Passover sacrifice on the skewer!
If so, I would like to know how it is possible that the sages disagreed on a law that certainly had a tradition of how to apply it? And how did the disagreement arise?
I thought of making the excuse that those individual opinions that we see that were divided in the accepted halacha of their time wanted to re-examine the Scriptures against the accepted halacha. But it's hard for me, because if tradition says that this is the correct way, how can a sage come and want to innovate against the path of the Torah?! For example (in the last paragraph) if the accepted tradition is that the intestines of the Passover lamb are roasted during the Passover, how can someone come and claim that this is cooking? After all, God, the Holy One, explicitly approved this… (that this is not considered cooking).
I would be happy to answer or refer you to a place that elaborates on how the disputes arose in Torah law against accepted tradition. For example: how they acted until the days of Rabbi Yehuda regarding the Sanhedrin quorum… and how the second Tanna disagrees with tradition.
Sorry for the length.
Best regards,
Y. Weizmann.