Self-identified libertarians are a weird and diverse bunch of people who often have very little in common. I know of at least 4 different types of libertarians:
Type A is really a mainstream conservative, but he's gay, or likes to smoke pot, or enjoys some other family-unfriendy activity and he's affraid someone will call him a hypocrite if he identifies publicly as a conservative. They are probably at least a 51% majority of the "movement".
Type B is an anti-Semite, conspiracy nutjob, doom-is-coming, beware of the black helicopter guy who is stashing food, water and guns on his ranch in Idaho, you know, just in case. These are really the most deplorable people who associate with libertarianism.
Type C is a disappointed paleoconservative, yearning for romantic days of 100% isolationist past. Sure, I don't agree with their foreign policy views, but this group is usually tolerable.
Type D is a libertarian leftist, they are like left-wing version of type A, liberals who like guns or are just a bit too mysanthropic to buy mainstream liberal Kumbaya mantra.
The second most annoying thing about libertarians, is their constant bickering between themselves about what is "true libertarianism" (of course, conservatives and liberals bicker too about ideological purity, but not so passionately as libertarians, which is natural because like I said, libertarians often come from at least 4 different mindsets). However, the most annoying thing about libertarians is their belief that there's some silent libertarian majority that will soon rise and that the majority of people are really unconscious libertarians.
This is a really good analysis.
I'll just add that, while I'll take your word that the type A and D libertarians are the majority of the movement, thanks in part to the rise of the tea partiers the selective small government hypocrites from type C and (especially) the wholly obnoxious type B's are by far the loudest and most public face of the movement--and hence most annoying--both in RL and on the forum.
Beyond that, though, I don't really have a problem with libertarianism (or libertarians).