I agree that cultural exchange is a great thing, generally, and that most of the recent controversy is overblown, but there needs to be a way to acknowledge that cultural exchange can't be divorced from global power structures. Western people do get to pick and choose what cool cultural traditions they want to adopt while American culture encroaches everywhere due to cultural hegemony. I don't think a white girl who wears a sari is a terrible person, but it's important to understand the context in which this is happening.
(Also the commodification of cultures is often gross and tacky so I understand why people are bothered).