Learned recently that prior to being elected president in 1868, Ulysses Grant had only ever voted once beforehand, for James Buchanan in 1856.
Grant was the only Republican elected prior to 1896 to have not been a Whig Party member as far as I can recall. He was very apolitical and their might have some dislike for Fremont involved in his vote as both served in the Mexican War, that would tend to complicate matters. Worth looking into because there might have been a bad interaction between the two or just a general dislike for him.