Answer

Maimonides himself says this, and so do the Geonim and others. But beyond that, it is a simple logical point. Aggadah usually deals with facts, and there is no authority with respect to facts.
Therefore, even if this were in the Jerusalem Talmud, it could not be correct and should be rejected. And indeed, the Korban HaEdah there wrote that “aggadah” here means a law given to Moses at Sinai (and apparently his intention is “haggadah,” meaning something transmitted orally from one person to another).
And in the commentary Alei Tamar there he elaborated further and wrote that the Babylonian Talmud disagrees:
“Matter” refers to aggadah. In the Babylonian Talmud, “matter” refers to Jewish law. See Mareh HaPanim, who suggests that here too the meaning may be a law given to Moses at Sinai that was transmitted orally from person to person. And indeed, in Gittin 2:4, Rabbi Oshaya Rabbah said to Rabbi Yosi Nesiah, ‘In your aggadah of your grandfather, who can judge for us?’—yet this is an outright legal rule regarding a bill of divorce: if he nullified it, it is nullified. I wrote there that perhaps they sometimes called a law that was not fully grounded ‘aggadah’ as a borrowed term, meaning something told from mouth to mouth. And we similarly find that sometimes an established aggadah is called a halakha, as in Sifrei, Beha’alotekha, section 69, cited by Rashi on Vayishlach: Rabbi Shimon bar Yochai said, ‘It is a known rule that Esau hates Jacob,’ etc. And the gaon Rabbi Tzvi Hirsch Chajes also noted this in his glosses to Berakhot 31, and also brought a proof from Sanhedrin 104: ‘A law was forgotten from him: there is one limb in a person…’ See there.