I think adultery should be treated as some manner of tort, personally. I don't think most of the adultery laws that are currently on the books in states that still have them are very practical or well-considered. I doubt Virginia is an exception.
Seriously? That seems completely crazy.
In a way it already is. Usually if you're divorcing your spouse because they were unfaithful you can get something out of it, or at least better terms than if no adultery had taken place.
A divorce is a stand-alone cause of action. This is another matter entirely. The whole case would be fairly ridiculous and generally a waste of judicial resources.
But, I looked it up on wikipedia and apparently there was a tort of adultery, called "Criminal conversation," way back when and it still exists in North Carolina.