Answer

It seems to me that this is not from the Sages but from the medieval authorities (Rishonim) (see, for example, Rashi on Exodus 22:1).
If a group comes to kill me, I am permitted to kill all of them (and it seems to me that I am obligated to do so). There is no difference here between one person and many. After all, if they do not have the status of a pursuer, then I may not kill even one of them, because my blood is no redder than his. But if they are coming to kill me, then all of them have the status of a pursuer, and I must kill them all.
Even if they want to kill me because I am a sinner, I am permitted to kill them. A proof for this is from Zimri, about whom the Talmud says that he would have been permitted to kill Pinchas under the law of a pursuer (or at least he would not have been liable to death for doing so). So much so that the Mishneh LaMelekh on Maimonides, in the laws of murder, wonders whether, when an agent of a religious court comes in the course of his duty to execute me, I would be permitted to kill him under the law of a pursuer (his conclusion is no). And that is said where the killer is doing something he is permitted to do, or even something that is his job. You are asking about just some person who decides to kill me because I am a sinner. That is a full-fledged pursuer in every respect, and anyone who sees him must kill him (not only the one being pursued).