Economic policies associated with Carter did the worst of the damage, probably. There's not much of a link between places where Anderson did well and where Carter lost lots of votes.
Anderson's vote was pretty evenly distributed across the state. I'm not sure how you can argue that Anderson didn't hurt Carter there. Of course, you're also right that the biggest swing against Carter was in Worcester County, where presumably it was about economics (although maybe not) and where Anderson performed somewhat below his statewide figures. But Carter could have held on to the state even with such a swing against him assuming Anderson's vote split even as close as 45-40-15 Carter-Reagan-Other/Not voting.
