White had an interesting socon side that made him something of a reverse Kennedy (but not quite a reverse Souter). Goldberg passed away in 1990, so he would have needed to retire under Carter to keep the seat left-leaning.
But the 3 most significant shifts are the Goldberg->Fortas->Blackmun mess, Marshall->Thomas, and Ginsburg->Barrett. So 2 older justices declining a chance to retire when they could be sure of an ideologically similar successor, and 1 really weird case of a young, healthy justice being pushed out by a president with the same ideology in favor of someone who was corrupt as heck.
White being replaced by RBG was a pretty strong shift in the makeup of the Court, too. Also, if it weren’t for the Goldberg resignation, we would probably not have seen
Roe v. Wade.
I should further note that I believe Byron White was the only one who found campaign donation limits constitutional.