The assumption that this letter is "anti-trans" when it doesn't even mention gender issues is pretty telling. It seems like a lot of the criticism is operating off of the assumption that if we let people say what they're actually thinking, they'll be transphobic.
I disagree with the people saying that the letter is anti-trans, but the assumption that it's anti-trans comes from the fact that some known anti-trans people signed it (J.K. Rowling and Jesse Singal, not sure if there were any others). I wouldn't be surprised if, in their minds, the letter was about trans people, but I doubt that many other people were thinking that way.