Answer

How can you go to a gym that doesn’t exist? That reminds me of William James’s saying: philosophy is like a blind man’s search in a dark room for a black cat that isn’t there.
I assume you meant that there is no other gym. In my estimation, the thing is not intrinsically forbidden. If it arouses forbidden thoughts in you, it is proper to refrain from it (though even about that one could debate, on the grounds of “there is no other way”—see Pesachim 25).