Answer

You assume that we need a source that authorizes us to interpret the Torah. But I assume the opposite: if a text was given to us, especially one that definitely doesn’t tie up all the loose ends and leaves a lot of room for interpretation, then it assumes that we will interpret it. And if it wanted us not to do that, it should have said so.
I should clarify that I’m not talking about adding or subtracting, but about interpretation that uncovers what is already in the text itself. As for adding and subtracting: if you were right that the Sages engaged here in a conspiracy to add to the Torah, the first thing I would have done is not sharpen the issue of adding and subtracting and not define them as prohibitions. They were not afraid to highlight this and define two such prohibitions. The additions of the Sages are rabbinic laws, and indeed they are obligated to clarify that these are different laws and not Torah-level laws, so as not to violate the prohibition of adding. Adding rabbinic laws is not an addition to the Torah.
Beyond that, there can be an oral tradition that the possibility of interpretation was handed down to us, and there is no need דווקא for a verse. You can of course be suspicious of any such tradition, but I don’t see any necessity for that suspicion.
Third, if our commitment to the Torah is by virtue of the contract we signed—“We will do and we will hear”—then we are the ones who determine the boundaries of that commitment. Therefore there is no need for permission to interpret, since that was implicit in the very terms on which the contract was made.
Fourth, the Torah does in fact authorize the Sages to interpret the Torah—not in the verse you cited, “My Torot,” but in the verses that deal with the authority of the Sages: “If a matter is too difficult for you, between blood and blood, between judgment and judgment, between lesion and lesion… then you shall arise and go up to the place…” The Sages are the ones authorized to interpret the Torah we received.
After all this, it is clear that derivations such as “My Torot,” and even what I brought from “If a matter is too difficult for you,” are not the source that gives us permission to interpret. Perhaps the opposite: once it is already clear that there is permission, we anchor it in one supporting text or another. It doesn’t carry much significance.
I’ll just add that in every legal system there is a top point of the pyramid that determines its own powers. So too with the Knesset, and so too with the Supreme Court. This is built into such systems, and there is no other possibility. Therefore I do not see any principled problem in the Sages determining their own authority.
And finally, I don’t know what exactly you mean by the term “the logical stability of our Torah,” but if you mean something that is logically necessary, you’re in the wrong field. Nothing in the world—and certainly not legal and normative systems—is “logically stable” in that sense.