Borrowing a Holy Book Without Permission

I study in a yeshiva and it has happened several times that one of the rabbis who doesn’t know me has taken a book from me without asking. A friend from the yeshiva told me that he heard that the Rosh Yeshiva said that in the Beit Midrash it is permissible to take unless it is explicitly stated not to take out of the yeshiva. This sounds like theft to me, and I don’t see any reason now to write a do not touch on every book I bring to the yeshiva, but my books are important to me, so I don’t want anyone who wants to touch them just like that.
I wanted to ask the rabbi what he thinks about the issue from a halachic perspective?