This same topic has been covered a lot in this sub-forum, but the obvious flaw I see in your premise is the simplified assertion that (A) supporting abolition of slavery is INHERENTLY "progressive" in our modern political sense, and (B) it's even more ridiculous to act like supporting something as broad as "internal improvements" was inherently progressive - something Democrats at the time (understandably) derided as effectively corporate welfare, given American society at the time. This amounts to the thinking that anything that led to positive ~progress~ has to be "progressive" in a left/right sense. This is circular logic that just applies all of the good outcomes of history to the "progressives" and assigns anyone who opposed the "good" outcomes (which we view with 20/20 hindsight) as "conservative." There were obviously left-wing GOPers who supported both abolition and internal improvements for "progressive" reasons, but there were also right-wing GOPers supporting both for "conservative" reasons.
To give a more "good faith" (if overly brief) answer, the GOP descended from the Whigs for a reason. Once the big tent issue of opposing the expansion of slavery was out of the picture, the main two things tying the party together were nationalism and a generally pro-business attitude. While there were obviously extremely left wing people in the GOP because it opposed slavery, it would be a mistake to think that had set the party on a natural path to be some left wing party. The GOP was a unifying party of anti-slavery Democrats and anti-slavery Whigs ... and it seems clear that the Whig political influence was stronger. In fact, it seems patently obvious that once slavery and Reconstruction were no longer at the forefront, the GOP very naturally and rather quickly showed its coalition's true colors.
If people want to try to argue the Whigs were to the left of the Democrats before this ... eh, I find that to be a super weak argument that is of the intellectual quality of "I just saw Hamilton and since I liked Alexander the most when it comes to race issues, he was obviously the progressive in that play."
Plus, it's often left out by modern day progressives -- including ones that are people of color -- that thinking blacks should not be enslaved is NOT the same as thinking blacks should be equal.
Non-white academics also tend to deliberately make whites who opposed racism seem more numerous, substantial and effective than they actually were (notably in the recent critique of Dr. Seuss).
Such deliberate fabrications and exaggerations discredit attempts to draw attention to the real unpleasant facts of history.
As even TV Tropes notes, there were far more people that tried to be a Schindler than Schindler -- but they were caught early and they and the persecuted they were trying to save were killed -- and nobody makes a movie about *them*.
tvtropes.org/pmwiki/pmwiki.php/Fridge/SchindlersList
