Frank Pastore on How NOT to take a Survey

Frank Pastore has an article called "Who Speaks for Evangelicals?". Granted, not the most interesting topic, but it seems whenever the pious take pen to parchment, entertaining foolishness is sure to follow. Frank did not disappoint.

The article is about whether Evangelicals need to follow the liberal conspiracy and modernize, or whether they should follow the more conservative leaders. I know [yawn]. But what made me sit bolt upright in my chair was this little jewel of a pollster's nightmare:

"And, as far as answering the question, 'Who speaks for America’s evangelicals?' I asked listeners to my daily radio show on KKLA to name the Christian leaders they most respect.

It was no surprise to me the most common names were the national ministries we carry on our station. Why? Because these guys know how to teach the Bible."

Poor George Gallup (who I believe was evangelical himself) must be rolling over in his grave. Of course people polled on a radio station preferred the ministers that were broadcast on that station. That's part of what had them listening to Pastore in the first place! I think we should send Frank Pastore to a McDonalds and poll everyone in there as to their favorite fast food hamburger, and see what sorts of results he gets. Then let's send him to the teen pregnancy clinic and take a poll of how many of them are virgins.

One of the things that made George Gallup a pioneer was his understanding of the importance of an unbiased sample, and many pollsters still battle with that problem today. It is very easy to unwittingly choose the people you poll in a way that will skew your results. Pastore gaffe and the other examples above are silly extremes to make the point. But there are much more subtle ways this happens. A poll by phone is going to be biased toward the wealthy. A cell phone poll will be biased towards youth. There are many examples.

So when a poll is taken, the first question you should ask yourself is "Who was polled?" It might make all the difference.