I don't think prestige is even necessarily the best word for it--it's more about a sense of distinctiveness. People are more likely to remember ancestry which is distinct from the default--i.e. "my grandmother always used to say Polish phrases" and not remember the normal English other grandparents. This is probably why German is the most common ancestry among White americans on census forms.
I think prestige can't be ignored - about twice as many Minnesotans claim Norwegian ancestry as Swedish, even though the number of Swedish/Norwegian immigrants was broadly similar. So we're dealing with the same number of grandmothers saying traditional phrases, but one has been remembered much more than the other.
Very Ignorant Question:
What really is the difference between Norwegian and Swedish culture?
Norwegian are more live hard, work hard, party hard, pray hard and die hard. Fundamental Norwegian were fishermen, which mean they dealt with high mortality rate, famines and natural disasters, but they also had long period where they couldn’t work, while the Swedes were marginal forest farmers so they were always looking for any opportunity to find a way to make their life less marginal and they could also work all the time, so they didn’t have the same forced break the Norwegians did, Swedes also value face more than Norwegians does.
Fundamental Norwegian culture are more individualistic, equal and extreme, while Swedish culture are more collective, hierarchical and consensus based. This doesn’t mean Norwegians doesn’t value consensus, but it’s just less important for them than it is for Swedes (and Danes).