Answer

Truth be told, I too have always wondered about the fact that in our society people see spousal betrayal as something normal (with a wink), even on the moral level, and certainly on the legal level. After all, there is a breach of contract here and harm to another person. Of course, if the other side agrees, then there is no issue from either a moral or a legal standpoint.
In my opinion, it should be possible to sue the unfaithful partner in court for breach of contract and to set an appropriate penalty for it. True, you cannot bind one spouse to the other by force, but this is a breach of contract and there is definitely room to impose a sanction for it.
In any case, unlike you, I do not see this as something like murder or theft, which are direct harms that I inflict on the other person’s body or property. Here I did my own act, and the other person was harmed as a consequence. In a certain sense this is harm by omission (not exactly, of course), and not by action. Beyond that, and perhaps more essentially, the harm here is the result of a breach of contract and not of a basic right. A person’s right to his property and his body is a basic right that is not based on a contract, and therefore the duty to respect it is incumbent upon every person in the world. By contrast, a spouse’s right to fidelity is founded on the contract between them (and therefore it is incumbent only on the spouse), and so I think these are matters of contract law and not something criminal.