Article III of the Constitution provides that the Justices, and all other federal judges, hold their offices "during good Behaviour." (and while they serve, their pay cannot be cut.) They may resign at any time, or retire when eligible. Once confirmed, however, they may be removed—in accordance with Article II—only by "Impeachment for, and Conviction of, Treason, Bribery, or other high Crimes and Misdemeanors." In effect they serve for life.
I'd say this is a high crime. They did violate their oath, afterall.