Nantucket and Martha's Vineyard are not especially wealthy because they're summer destinations. A lot of the wealth there is only visiting from New York, Boston, etc. Those who live there year-round are considerably more low-brow.
But, as mentioned, there are homogenously wealthy areas in just about any American metro. They're mostly small enclaves, but there are no poor people in Scarsdale, Weston, McLean, Buckhead, etc.
Add Short Hills, Winnetka, Potomac, the Park Cities, Medina, West University Place, Paradise Valley, Atherton, Mission Hills, Belle Meade, Fox Chapel and so many more. Pretty much every major metro area has some areas like this.