PPP under fire from Cohn, Silver, etc.

[hopper]

If you're going to throw out the pretense of "science" from polling you might as well bring back the Literary Digest.

I think Gallup went down that route last year.

No, actually Gallup's "Registered Voter Model" was correct having Obama ahead 49-48% but its "Likely Voter Model" had Romney winning the election in terms of the popular vote(47-45%) I think.

[J. J.]

PPP arbitrarily tossed a poll because they didn't think the results looked right. How is that different from what J. J. and co. do? Also just like the right wing trolls they were proven wrong later since the actual results did match the poll. They're methodology is irrelevant, that's not what's being attacked.

I generally do not toss the results.  I tried to guess how much a poll may be off; obviously they are, or we'd get perfect results.

Some of Silver's methodology is off.  If a pollster gets the last poll right, that is good enough for him.  It's terrible of you are looking at polls in September and trying to determine if that will be the result in November. 

[Link]

Nate Silver can be a bit of a dick, but he's clearly correct. Passing over a poll that you had planned to release because your gut tells you that its result is wrong is, as Silver puts it, bad and unscientific. 

I don't know if Mr Silver can be a dick or not but he does seem to have some valid criticism here.  If your poll isn't scientific and reproducible that it automatically becomes suspect.

Haven't any of the forum adolescents seen commercials for money management companies?  They say past performance is not and indication of future performance.  If the methodology is sound then you can trust it.  But merely looking back and saying they were right 8 out of 10 times simply isn't good enough.

Considering how young this forum skews and how many people are still in school I'm shocked at the dearth of curiosity concerning where these numbers come from.  When I was a teenager I was constantly asking why, why why.  I didn't just accept some vague hand waved response.  If someone can't explain their model in clear scientific and consistent terms are you really going to trust it?  Even a coin can flip 8 heads in a row.  That doesn't mean the 9th time is going to be heads.

[True Federalist (진정한 연방 주의자)]

Some of Silver's methodology is off.  If a pollster gets the last poll right, that is good enough for him.  It's terrible of you are looking at polls in September and trying to determine if that will be the result in November. 

J.J. By the nature of the beast, it's impossible to say what the results would have been if the election had been held in Fructidore instead of Brumaire.  At most what analysis might reveal is if a polling firm that had been averaging say two points more in favor of the Jacobins than the consensus in its Fructidore polls, ended up being one point more in favor of the Girodins than the consensus by the time Brumaire came around.

[Beezer]

I'd say methodology definitely matters. If I call 10 random people, then have a look at RCP's poll of polls in order to tweak the #s before releasing my results (which I suppose could be quite accurate) I would not be surprised if people had a hard time considering me a respectable pollster.

[Alcon]

Some of Silver's methodology is off.  If a pollster gets the last poll right, that is good enough for him.  It's terrible of you are looking at polls in September and trying to determine if that will be the result in November. 

I don't understand what you mean.

[Lief 🗽]

Great article defending PPP against Nate Silver's nonsense:

You must be logged in to read this quote.

http://www.psmag.com/politics/use-abuse-polling-american-politics-66266/

[Landslide Lyndon]

From today's DKE Digest:

http://www.dailykos.com/story/2013/09/18/1239030/-Daily-Kos-Elections-Live-Digest-9-18#20130918102223

In an interesting op-ed in The Hill, pollster Mark Mellman takes polling firms to task for failing to distinguish between likely voters and what he calls the "likely electorate," which he considers to be far more important. He specifically criticizes polls from Siena and Marist that showed Eliot Spitzer with big leads in the Democratic primary for New York City comptroller.

By contrast, Mellman says his firm had Scott Stringer, their client, "ahead the entire time" because they assumed that a quarter of registered Democrats would turn out.

...

Mellman doesn't go into any detail about how he constructed his "likely electorate," but Mark Blumenthal suggested on Twitter that it involves "lists and vote history," to which Mellman replied "yep" and intimated that he wasn't about to share what someone else called his "secret sauce."


So let me get this straight. PPP is crucified for by the likes of Cohn and Silver for not being transparent enough, even though they cooperated with them and Cohn had a lengthy correspondence with Tom Jensen about the company's methodology.

But Mellman outright refuses to divulge any details about his own polling methodology and the polling nerds are nowhere to be found crying about "transparency" and "questionable scientific methods".

FTR, I'm with Mellman on that. If you have a "secret sauce" that helps you getting accurate results then why should you make it public and give a helping hand to your flailing competitors?

[True Federalist (진정한 연방 주의자)]

FTR, I'm with Mellman on that. If you have a "secret sauce" that helps you getting accurate results then why should you make it public and give a helping hand to your flailing competitors?

The problem is that this secret sauce, while apparently allowing PPP to get better topline numbers is also causing their crosstabs to go wacky at times.

[Landslide Lyndon]

FTR, I'm with Mellman on that. If you have a "secret sauce" that helps you getting accurate results then why should you make it public and give a helping hand to your flailing competitors?

The problem is that this secret sauce, while apparently allowing PPP to get better topline numbers is also causing their crosstabs to go wacky at times.

Not really. Survey USA was the company where that phenomenon was most frequent.