Let's say someone has aspergers or is socially awkward, but was not able to attend college for whatever reason. Most of the lower skill jobs require strong people skills, which people with aspergers do not have. I am applying this to any of the reasons that said person could not attend college (i.e. finances, poor grades, other disabilities, etc).
I didn't know you needed strong social skills to work in a factory or construction site.
Obviously, if someone has a debilitating disability, "get a job" is not an appropriate answer. But minor mental or physical disabilities do not prevent someone from working. By the way, I don't believe in telling everyone to get a job - working is a lifestyle choice and we should treat it as such. But if you have the ability to work and do not, you should bear the consequences of your lifestyle choice if you don't have other sources of income.