War weariness was a major factor.
Also generational. Since this was the lowest turnout election until 2014, it was pretty much the case that only old Yankees were voting in many districts that had become majority ethnic or were in the process of being majority ethnic white.
In New England Democrats actually experienced a pretty large drop-off in the white ethnic, particularly Irish and German vote as well due to both groups tending to be anti-war at the time. I was recently reading a book (I can't remember the title) about ethnic voting trends in Boston and it demonstrated how at the federal level there was a mini-trend among Irish-Catholics towards the GOP between 1936 and 42 with Willkie actually winning the middle class Irish vote in 1940. It may initially have been influenced by the popularity of Father Coughlin and his break with FDR but then accelerated as the possibility of war increasingly threatened. It's been speculated that a young JFK may even have voted for Willkie.