This entire premise assumes that libertarians have any real strength to begin with.
I think we do. Perhaps not the ideological hard-core, but certainly there is a huge segment of the population that leans libertarian, if it's broadly defined as being economically "conservative" and socially "liberal".
You vastly overestimate what percent of the population leans ideologically consistent. There are a lot of people who think the government should "justify its existence" and not intrude on their lives, but many of those people see social permissiveness as being an intrusion.
The number of people who believe in a strict doctrine of non-involvement, and happen to agree with you on which economic model supports greater liberty, and does all of this consistently enough to form a meaningful voting bloc, is probably negligible.