Okay, organizations with independent executive authority like marching band can basically determine their own membership standards as much as they want, right? I see no problem for them to exclude a person based on the fact that other members of the organization don't feel comfortable around him due to allegations made against him. It's their own call.
So if they (or some small set of them, or even just one of them) doesn't feel comfortable around the new AltoSax player because they're Muslim, they should have the ability to exclude them? That seems like a dangerous road to go down.
It's called freedom of association. You can't have freedom of association without having the freedom to not associate, and that also includes the freedom to disassociate with those a group had formerly welcomed. So yeah, freedom can be dangerous and it can cause problems, yet it is generally better than the alternative.
^^^
Also religion is a protected status whereas "has been accused of committing rape" is not