I wonder if it had anything to do with the movement to repeal corn laws in Britain, since corn is yellow.
Corn means any grain in the British context.
As far as a Wikipedia dive is credible in this context, apparently it originates with revolutionary sympathizers in the British parliament during the American revolution, particularly the allies of Charles James Fox, who would dress in blue and buff on the parliament floor - the colors of the Continental Army. Evidently the buff stuck better than the blue, and eventually came to represent the whole Whig party, and then liberalism on the whole.
