Why not Berkeley?

Hello Rabbi, I read your book Truth and Unstable and enjoyed it.
The book deals with the question of the validity of our axioms (just because this is our logic does not mean that the world is indeed like this – “The world owes you nothing, etc…”). As I understand it, your answer is that our axioms, through synthetic perception, come closer to the world of the senses because we supposedly see them, with the eyes of reason. At a certain point, you part ways with Berkeley, who even questioned the reliability of the senses, and say that most people are not bothered by such skepticism.
I wanted to ask what is the justification for basing our entire world of logic on the senses? After all, even the senses can seemingly be challenged and it can be said that our reliance on them stems from our logic and not from any justification.
And if so, I didn’t understand what the book’s progress was for if one finally has to admit that one has no justification for the beginning of our world of cognition?
And simply why doesn’t Berkeley’s skepticism bother you?
I hope I was clear.
Thank you very much.