I'm curious to see if there's any meaningful support on Atlas for the "yes" answer. Call me crazy, but I can see the logic behind it from a libertarian perspective. If we view marriage as a contract, and adultery as a violation of that contract, then perhaps it could be argued that it is the government's duty to enforce that contract. Perhaps under such a system, marriage contracts could choose to either include or omit this kind of clause.
That is the worst form of "libertarian" theory.
Breach of contract does not carry legal penalties to speak of. Now, if you're talking about a civil breach of contract, well, adultery can lead to grounds and be a factor at least in resolution of a divorce settlement in Most states already.