I can't remember where exactly I read this, but I'm pretty sure that pot polls better among white people than it does among minorities, so I don't think this is necessarily just suburban panic (despite of course the fact that the effects of criminalizing weed fall heavier on those groups). I'm sure there's a larger-than-you-think cohort of older blacks and Hispanics that mostly just still consider all drugs to be a scourge to be fought, and don't make the distinction between all-but-harmless weed and actually-harmful harder stuff.
That's probably also true- although I suspect younger generations to be quite for it.