This semester, the Center offered a new course, Behind the Numbers: Polling and American Elections. In addition to learning about survey methodology and the role of polling in recent elections, students had the opportunity to write and administer original questions for a class survey administered via Mechanical Turk (MTurk). Through the use of creative and scientifically-informed questions, the students gained new insight into Americans’ opinions on timely policy issues, including defense spending, drone use, gay marriage, gun control, U.N. resolutions and immigration.
Purpose and Methodology
The purpose of the survey was to threefold: (1) allow students to apply the skills they learned in class to the practice of survey administration, (2) give students the opportunity to collect original, empirical data for use in their theses or other research projects and (3) help students learn about public opinion on timely issues. Students worked together to develop and revise their questions. Several students constructed survey experiments, in which half the sample received one version of a question and the other half received an alternative version. The survey was built in Survey Monkey and administered in MTurk, an online labor market run by Amazon that has gained increasing popularity among scholars of public opinion in recent years. The survey was administered from March 9, 2013 – March 28, 2013; 305 U.S. residents completed the survey. Students were provided with the raw data and asked to analyze the results in light of existing research and recent political events.
Findings on Defense Spending
Robyn Russell developed a battery of questions about the public’s opinion on defense spending. Current research suggests that opinion on this topic is fairly volatile and quite split. In a February 2013 Gallup poll, 36% of respondents indicated that the U.S. was spending too much on defense, 35% said the U.S. was spending about the right amount and 26% said the U.S. was spending too little. Two years earlier, these numbers 22%, 35% and 39%.
Russell theorized that one reason for the apparent volatility and absence of consensus is a lack of information about the topic. To examine this issue, Russell devised a question that first informs respondents about the portion of the U.S. budget that is devoted to defense spending and then asks about their position on the issue. The results are presented in Figure 1. The top graph in Figure 1 displays the results for the Gallup question. The bottom graph displays the results from Russell’s question. Responses to this question, which provides respondents with information about the overall level of defense spending in the U.S., yielded a substantially different distribution. According to these results, 75% of respondents favor reducing military spending while only 7% support increasing military of spending. While the MTurk sample has limitations (most importantly, its generalizability to the U.S. population), Russell’s findings suggest that a lack of contextual information for thinking about proposed changes to defense spending accounts for the public’s seeming instability and disagreement on this issue.
Findings on Gun Control
Doug Andres designed a survey experiment to examine support for gun control reforms proposed by President Obama in his 2013 State of the Union Address. Andres was interested in examining whether respondents would be more likely to support gun reform laws if the response choices offered specific policy proposals. Approximately half the sample (146 respondents) received a question that asked about support for new gun reform laws; the response choices were “yes,” “no,” or “not sure.” The other half (158 respondents) were asked about their support for new laws that would require universal background checks, prevent the sale of guns to criminals, limit magazine capacities or “none of the above.” The results from both questions are presented in Table 1. Among those who received the first question, 28% indicated that they did not support new gun reform laws. Among those who received the second question, 90% expressed support for at least one of three gun reform laws. This suggests that, when asked about their support for specific laws, as opposed to their general support, far more respondent express a favorable opinion toward gun reform.
MTurk offers instructors of survey methodology an inexpensive tool for administering original surveys to large samples. As with all internet surveys, selection bias and data quality are of concern. Nonetheless, MTurk has been shown by other scholars to be particularly useful for survey experiments (see, for example, Berinsky, Huber and Lenz 2012). Further, the sample obtained from MTurk has been shown to be far more valid and reliable than a typical convenience or student sample. Most importantly, an MTurk survey is a feasible means of allowing students of public opinion to learn by doing; they can test new hypotheses using original data.
For more information about the course or the survey, please do not hesitate to contact Dr. Bachner (email@example.com).