I'm guessing it's partly the fact that it's isolated from metropolitan France, doesn't pay much attention to what happens over there and just does its own thing (see other overseas territories), and partly the phenomenon of super-religious super-right-wing peripherial regions like Brittany shifting left due to secularization, just turned up to 11.
The real question is how it would vote if it were a part of Canada.
(Yay 1000th post!)
Might well be Catholicism, though I would have expected that to hold up more in the voting. Also, it tended to prefer the RPR to the UDF (except with Giscard) when Catholics tended to be the other way around.
Looking on Wikipedia, I have found that the islands were taken over by Free France in 1941, which might explain the 83% vote for de Gaulle in 1965.