Fuller, Field, and Peckham (in Plessy and Lochner majority)
McReynolds (one of the Four Horsemen and extreme bigot)
Butler, Vandevanter, and Sutherland (the other members of the Four Horsemen)
Byrnes (Dixiecrat)
Pitney (anti-labor and upheld Leo Frank's wrongful conviction)
Clifford (neutered the Reconstruction amendments)
Rehnquist and Scalia (Rehnquist and Scalia had an abysmal civil rights record, both in Bush v. Gore majority; Scalia tried to strike down the Affordable Care Act, and was in the majority for Shelby County and Citizens United)
O'Connor (co-wrote Bush v. Gore and quit at the wrong time)
Kennedy (co-wrote Bush v. Gore, wrote Citizens United, anti-labor, in Shelby County majority, tried to strike down the Affordable Care Act quit at the wrong time and personally chose Kavanaugh for Trump to appoint)
If you wanted to say this, you should've put it in the OP.