Polk, barely. Van Buren did nothing to help the Mormons in Missouri, Buchanan did nothing as the Civil War erupted around him as well as trying to distract the nation by fighting the Mormons, Andrew Jackson was genocidal, and Pierce was a puppet of the slaveholders. At least Polk got most of the now-American West into the Union, even if the way he did it was terrible. I like both California and Utah.
That is one way of looking at it. And to be frank you probably understand History better than other people (ie, how the actions of politicians in the past would've affected somebody of your background).
No "buts" here, just an observation.