I have only recently started to question quantitative research findings, since we began INF1240. How many times have we come across so-called 'research findings' in the news, newspapers, magazines, and actually asked ourselves, "How did they come up with this data? What questions did they ask? How many people did they sample? What was their margin of error?" Anyone reading this blog entry can easily come up with ten more questions.
The poll O'Reilly was referring to is an opinion research poll conducted by CNN:
CNN interviewed 1009 people over the phone. They felt that number was sufficient enough to issue a definitive statement that 70% of over 300,000,000 Americans opposed the mosque. Should the survey have asked them to list reasons for their answer? Does it matter what their reasons are? How much time did the respondents take to actually think about what the 41 questions were asking? Is it possible the questions were designed to illicit negative responses? Personally, I know if I'm forced to take a phone survey characterized by such long-winded questions, I respond mindlessly without thinking about what the question is really asking. I think, "Who cares? It's just a survey. I just want to get off the phone."
But to Bill O'Reilly and (judging by the thunderous applause he received and the video comments on YouTube) to many others, it is not just a survey. It amounts to rhetorical ammunition.
Guess the moral of the story is: never underestimate the power of a CNN poll.
Lana... I think you hit the nail on the head when you state that "...it is not just a survey. It amounts to rhetorical ammunition."
ReplyDeleteThe survey question was loaded. CNN/Opinion Research Corporation didn't ask whether respondents opposed the building of a "community center" or an "Islamic education facility" when they posed it. They used inflammatory language that, coincidentally, fits what they've been pushing in their coverage (especially since 2001).
"As you may know, a group of Muslims in the U.S. plan to build a mosque two blocks from the site in New York City where the World Trade Center used to stand. Do you favor or oppose this plan?"
We can pick apart the entire question. A "group of Muslims" - not a "community organization" or "religious organization" is building a "mosque" (even though it has been emphatically reiterated, time and again, that it is not a mosque they are building). Where the "WTC used to stand" is pretty loaded as well. They could have said "near the WTC site" or something along those lines, but it's an underhanded reminder that a group of Muslims blew up the WTC.
What can we take from this, as researchers? The impact of persuasive language often goes unnoticed. Good research, and bad, can live and die by its presentation. O'Reilly, that blowhard, knew exactly who we was speaking to, and what ammunition he was loaded with. He's been trumpeting this "ground zero mosque" nonsense as long as anybody, to the point that it's firmly encoded in network media broadcast language. The term "ground zero mosque" is now so deeply engrained in the American psyche that even leftists use it. That said, O'Reilly shows how constant repetition of hot points, often at the expense of more valid ones, can serve the act of persuasion. Remember "Sore Loserman" from the Gore-Bush election? Old red-baiting tactics ("he's a communist") have been built upon: Obama's a "Muslim" or "socialist" or was "born in Africa" etc. Repeat it enough, it becomes truth. Adam Curtis's BBC documentary, The Power of Nightmares, does a pretty incredible job of colouring in the lines here. Highly recommended for anyone interested in how the mainstream American media have been complicit in government-driven creation and manipulation of fear, post-911.
I know it has come up in class, but I'm reminded of the "race" scientists. Arthur Jensen, Philippe Rushton, and others of their ilk, know how to present their material, how to code it in language that masks the holes in their research... and they know their audiences. That doesn't mean they are impervious to criticism, though. As mentioned, Stephen Jay Gould's book, The Mismeasure of Man, excoriates these guys. If anybody wants to read a little primer, the last few chapters in his book Ever Since Darwin are a good introduction to Gould vs. the Race Scientists. But Gould, when he wrote this stuff, was up against the novelty of concepts like the bell curve, and the mainstream media's seemingly blind acceptance. (I still remember the Time and Newsweek cover stories in the mid-90s, and my classmates' willingness to side with Herrnstein and Murray because they were supported by the editors).
There are so many times that we have seen bias distorted messages narrated in the media. Many more marketing research reports, academic or not, have used precentages to exaggerate their "research findings". What's the difference between 300 and 3 million? If we think about the population in a national scale, as long as the survey is not covered on a national scale, those % are merely persuasive figures. As Gabby mentioned, the language used is for its target audience. CNN is more like a representative of some social trend, be it moral or not. For them, the clicking rate is the most important thing. Behind it is a purely commercialized rule.
ReplyDeleteI came to the blog tonight to write about Luker's section on operationalization (what is that 8 syllables?) -- which I actually found really interesting and which provided me with word that articulates a problem I foresee in my own research -- but I quickly became more interested in this post and Gabby's comments, which is probably for the best because typing operationalization is an effort in itself.
ReplyDeleteI took a course in logic and critical thinking during the first year of my undergrad and was probably the most useful course I've ever taken. I try to refrain from making assertions such as "everyone should be forced to take such and such course" but in this case I really do believe that it would be to the benefit of most. We spent some time specifically on polling techniques and the kind of logical and rhetorical fallacies that polls are susceptible to and I've been generally skeptical of polls ever since (I won't even consider a poll's results without knowing the sample size).
That said, there really is an elegance to polls that I think redeems them from the status of rhetorical ammunition. I completely agree with Gabby's analysis of the question -- it was clearly flawed. Which brings me to my point about operationalization... I kid. Any pollster with even the most minimal training would realize that the question was flawed when they posed it, which suggests to me that it was at least a conscious decision. There's no doubt then that this poll reflects poorly on CNN, both for their decision to run the poll in the first place and then to broadcast it. But then it's no secret that the mainstream media are equally adept at spinning both qualitative and quantitative data. In other words, what information isn't rhetorical ammunition when it's being hawked by CNN or Fox News? Nevertheless, barring a revolution in journalistic ethics or information literacy rates, there is hope for the poll yet...
Lana, you point about not taking polls seriously when they call you is illustrative. I get called fairly frequently and I always take the poll seriously, no matter how busy I am. So, if you and I participated in the same poll your possibly un-serious answer would be diluted by my serious answer. Multiply that by 1000 (and yes, 1000 has been statistically proven to be an accurate sample size for populations in the hundreds of millions, though randomization is another issue) and suddenly your un-serious answers become minuscule. But let's assume that there are a bunch of people that didn't take it seriously and a bunch of outright liars. Maybe 1% of responses are faulty. Well, that contributes to the polls degree of accuracy, which as Prof. Grimes mentioned in class is usually a minimum of +/- 2.5%. And that's the elegance of it: things average out.
[sorry, I guess this went a bit long, I had to break it into two posts]
ReplyDeleteAnd if a poll happens to capture a truly misrepresentative sample, which is statistically possible, the wildly divergent results of that poll should eventually be averaged out by another set of polls that test the same questions. That's the elegance of polls: they are fundamentally cumulative. If you regard them as averaging mechanisms, they're pretty accurate. If you regard any one poll as a perfect and precise representation of the things its apparently testing then it's a problem (I love the concept of synecdoche that Luker mentions, there's a great movie by Charlie Kaufmann called Synecdoche, New York that plays with this concept). What will go unreported in the mainstream media is that someone else did a poll -- hopefully with a less loaded question -- calling into question the results of the CNN poll, which would in effect average out the results. Again though, this isn't so much a problem with the media, not the polling.
I think this was your conclusion as well Lana -- that as rhetorical ammunition polls were in some sense being misused, and not that they were problematic in themselves -- but I thought it was worth stating anyway. In the wrong hands they can certainly be dangerous. Looking at the rest of the polls in the document you posted leads me to believe that CNN could at least put some effort into making it look like they're actually conducting scientific polls before they spin the results (the gay marriage polls seem really bad).
Another interesting and related concept from that critical thinking class so long ago: confirmation bias (http://en.wikipedia.org/wiki/Confirmation_bias). If polls are averaging mechanisms then the American mainstream media are confirmation bias mechanisms.
Lana, I think you make an excellent point. While I agree with what Gabby and Brian are saying to a large degree, we can't underestimate the need to generalize and structure our research questions in a way that provides us with the answer we want to know. When you add an issue so that is also emotionally charged then the findings are ripe for debate and analysis across the board. When we were talking about generalizability a couple of weeks ago, we didn't talk necessarily about the ethical implications of taking findings and sweeping them across all people. While they may seem statistically relevant and there is an element of yes it "can" be done, we didn't factor in the question of "should" it be done. Polls are based on opinion and as we know from watching the election polls, opinions changed quickly from day to day, minute to minute and one slip of the tongue by some candidate. They should not be used to make sweeping generalizations to a larger population. On the flip side however, for every generalization made, it will create a new research project to refute it, making the research world go round and round.
ReplyDeleteSuzanne