But in the case of trans women they're typically disguising female secondary sex characteristics so that would also explain why heterosexual men my not be attracted to them.
That's just the ones you notice not passing. When you remember that around 1/300 to 1/200 of people are trans but way, way less than that fraction of people you see are obviously trans, it becomes clear that they're everywhere and you just don't realize it.
Every time you go to a grocery store or Walmart, there are probably more than 200 people in there, but I bet it's really rare if ever that you notice a trans person. Every time you're in a large crowd, say a football game, there are a bunch of trans people there that you never realize.