Answer

As is well known, the medieval authorities (Rishonim) disagreed regarding things from which benefit is prohibited, whether or not ownership applies to them.
According to the views that there is no ownership over something from which benefit is prohibited, then leavened food is not exceptional. The assumption here is that if you have no permitted way whatsoever to use it, then there is no ownership here, and de facto it lapses. Leavened food is exceptional only in the sense that the Torah defined it as being in my domain even though it is not mine, so that I violate the prohibitions of “it shall not be seen” and “it shall not be found.”
According to the views that there is ownership over things from which benefit is prohibited, then indeed with leavened food there is a novel rule that it is exceptional and not mine at all (except for the purpose of violating the prohibition). This can be understood in several ways: (a) with all things from which benefit is prohibited there is some dimension of limited ownership, and in the case of leavened food that is enough to violate “it shall not be seen”; (b) with leavened food there is a special innovation that it is not mine at all (like other stringencies that apply to leavened food). 
See, for example, a discussion of these topics here: http://asif.co.il/?wpfb_dl=911