I think the problem for anti-Trumpers is the fact that all the the people who hate Trump more and more, probably never voted for him in the first place.
Think about it, if you were that upset about the muslim ban or the transgendered ban, the odds that you ever voted for him in the first place are very very slim.
Honestly, several GOP senators called him out for it. Even ultra conservatives like Scott and Hatch. It suggests that he's getting a lot of flack from his own wing.
Yep, I actually know Trump voters who abandoned Trump after the trans ban in the military.