Question

With God’s help,
A preliminary note:
You wrote that substantive authority is not binding; rather, it simply makes no sense not to listen to it. I think the same can also be said about formal authority, because the reason we listen to formal authorities is that it simply makes no sense to challenge them—whether because of the punishment we would receive, or because of the logic that, after all, there has to be some authority to run the system.
Now to the matter itself. I do not disagree with you, but I would like to suggest a different approach.
You write that whereas the authority of the Sanhedrin is formal, the authority of the sages of each and every generation cannot be such, because who would determine who the true sages of the generation are.
I think that from this perspective the Sefer HaChinukh may be correct in what it says. It says this: certainly you are allowed to replace whoever, in your view, fills the slot of “the sages of the generation.” But once you identify someone as occupying that slot, you cannot disagree with him and deviate from his words, and even if he is mistaken—you are obligated to obey him, just as one obeys the Sanhedrin. It is not that the responsibility is on him; the responsibility is on you, because you are the one who decided to identify the person in question as standing in for the Sanhedrin in your time. And if you were wrong about that—you will bear the consequences. But if you were not wrong, and he truly is worthy of that mantle, then even if he erred and you followed him, no blame rests on you; quite the contrary.
That is to say: whereas in the time of the Sanhedrin the identity of those seated on it was not a matter of personal choice, and its sages were appointed through a kind of “member brings member” system, in our time each person must use his own mind and judgment to identify the true sages of the generation. And insofar as he determines regarding one or more people that they are those worthy of greatness, he is supposed, in a formal sense, to obey them.
Therefore, if, for example, you think that Rabbi Shach is worthy of the title “one of the sages of the generation,” there is no reason you could not defend yourself before the Heavenly Court by saying that you did what was incumbent upon you, since the one whom you regarded as standing in for the Sanhedrin instructed accordingly. (And if you knew that he was mistaken, it is explained that one need not—and indeed may not—listen to the Sanhedrin, except regarding the sanctification of the new month; but if you do not know how to decide, and he decided for you, there is no reason not to listen to him.)
I think this is also relevant to what you wrote at the end of your remarks about replacing the nasi.
I do not see any contradiction between “do not deviate,” the law of the rebellious elder, and replacing the nasi. For there is an obligation to obey the nasi so long as he is nasi; there is nothing preventing replacing the person who occupies that seat.
So too in our time, according to the Sefer HaChinukh, there is no problem in deciding that in this particular ruling it has become clear to me that this rabbi is completely distorting things and is not fit to serve as a stand-in for the Sanhedrin, and therefore “do not deviate” does not apply to his rulings. But so long as I think that in general he is indeed fit, only that it is possible that in this specific case he erred, he has formal authority.
And it seems to me that the same can be said regarding the local rabbi. Certainly a rabbi can be replaced; I do not think there is any halakhic problem with removing a rabbi from office because of dissatisfaction with his performance. But as long as he is the rabbi, he has formal authority and not merely substantive authority.
If we return for a moment to the authority of the Talmud, which, as you wrote, is accepted as formal, perhaps this is the meaning of the point that the Jewish people accepted it upon themselves. Not only can the collective decide that the sages of the Talmud are not fit to serve as an authority for them, but any individual can claim that he thinks he is more qualified to interpret the Torah than they were, and therefore he would not violate “do not deviate”—though he would be fulfilling in exemplary fashion the dictum “and you would be a fool.” For it is more reasonable to assume that they had access to more traditions than we are capable of attaining, and therefore even if they are mistaken, the formal authority still remains theirs.
As I wrote, I do not see in these points any necessary conclusion against your view; I am only suggesting a different interpretation of the Sefer HaChinukh and what follows from it. What do you think?