That's part of the reason, but the other part is that nobody really likes it as an anthem. Some people get annoyed by 'disrespect', but a non-negligible proportion of them would happily agree that Jerusalem or Land of Hope and Glory or whatever would be a better tune to use instead.
That's a good point that I always forget: the times when its more relevant (sports) its mainly represented England and not Britain and that means that the Scots (and the Welsh) won't treat it like their national anthem and a lot of the English are like "why are we using a British song to represent England?". And then you get the weirdos that seem to think that all four constituent countries should use it because "ITS THE ONLY NATIONAL ANTHEM" but they are a vast, vast minority.
Not being English I don't know whether I should comment on it but I think that Jerusalem is what you should use: its historically a radical song which I will always admire but its come to be accepted by everyone: its got that sing along value that is always good plus its a good song. Admittedly the wrestling fan in me associates the tune of "Land of Hope and Glory" to Randy Savage which inherently makes that a funnier song but not in a way that's appropriate for a national anthem.