|
"Measuring Media Exposure in the 2006 ANES Pilot Study and Beyond "
By Jason Barabas
Scott Althaus and David Tewksbury co-authored a report on the performance of a “new generation” of media use measures that they designed for the 2006 American National Election Study (ANES) pilot survey. In doing so, Althaus and Tewksbury have advanced our knowledge about what is thought to be an important source of information for citizens across the globe—the mass media—and it happens to be an area of research in which effects are sometimes hard to observe.
Overall, the findings in their report are extremely valuable and have implications for researchers throughout the social sciences (e.g., political science, communications, economics, and sociology) as well as for journalists and pollsters. However, I suspect that the report may stimulate as many questions as it answers. In that regard, I offer (A) reactions specific to Althaus and Tewksbury’s analysis of the 2006 ANES pilot survey, (B) a comparison of the 2006 ANES measures with identical items that appeared in another national survey, and finally (C) some thoughts on unresolved methodological issues that go beyond the report and the new measures.
The 2006 ANES Pilot Study Report
Althaus and Tewksbury ask, provocatively, whether the traditional media exposure measures are still worth asking considering that some published studies that have detailed problems (e.g., Price and Zaller 1993; Bartels 1993). Based upon more than twenty-five pages of thoughtful commentary in their pilot study report, Althaus and Tewksbury seem to suggest that yes, media usage measures are indeed still worthwhile. Their position is buttressed by extensive analyses as well as a few sobering acknowledgments that some media exposure measures are not particularly useful. The report contains a lot of insightful passages and analyses, indeed too many to recount here, but some of the most noteworthy contributions are in the following areas
- Internet and Radio Usage. Althaus and Tewksbury included internet and radio usage in addition to the traditional newspapers and television exposure measures. If the media exposure items on the 2006 pilot study are remembered for anything decades from now, it is likely to be the addition of these long-overdue indicators. These are important media sources and unique ones at that, judging by the relatively low correlations between the various items and their empirical performance in models predicting various outcomes.
However, it was interesting to note that for the average respondent, television and newspapers are still the two most popular sources at 5.2 days per week and 3.7 days per week respectively. Radio and the internet are not far behind newspapers at 3.0 and 2.7 days per week each. It is still surprising that in 2006, despite all that has been said about the diffusion of the internet since the 1990s, that the internet is the least used source on a days per week basis in the ANES. All of these sample averages are statistically different from each other by my calculations using the 2006 data (p < .05). Thus, they help provide some baseline estimates of usage, although as I will discuss later, one might question the self-report measurement strategy in general. Nevertheless, the high correlations for the days per week measures with the minutes per day measures were reassuring. It is also nice to have a clear recommendation here: days per week measures are better than minutes per day. Yet, one wonders how well these results correspond to verified data on media usage (assuming such data exist)? Furthermore, are there any curious patterns that might lead one to question the validity of the exposure measures?
For example, some people might be surprised to learn that nearly 50% of the ANES sample claimed to have watched news on the TV every night during a typical week (that is, 185 respondents out of 371 gave an answer of 7 days of TV news for the typical week, although in the next section I report figures that are almost as large in a separate survey). Another point is that in the 2006 pilot study report media usage is given in (A) days per week, (B) minutes per day, and (C) minutes per week. I am not sure how the minutes per week measures were created, although it is entirely possible that I overlooked the relevant passage. One might expect the multiplicative term of the first two measures to be roughly equal to the third (i.e., A x B = C). The average news usage in days per week for the typical week is 3.58 in Table 1 and the average number of minutes per day is 31.82. The number of minutes per day multiplied by the number of days per week is 113.91, which is less than the reported 155.19 minutes per week in that same table. The differences were not as stark for the “typical week in the past year” results (138.8 vs. 162.8), but gaps remain for this medium as well as for the internet, radio, and television. Since the minutes per week measures do not appear in the 2006 ANES codebook (it has minutes per day), I am sure Althaus and Tewksbury will soon elaborate on how these measures were constructed.
Finally, there was one inconclusive result. Inspired by Chang and Krosnick’s (2003) research on “typical week” versus “typical week in the past year” self-reports, Althaus and Tewksbury tested these wording variations in the 2006 ANES. They conclude that the results “offer few definitive signs about the superiority of the ‘typical week’ reference over the ‘typical week in the past year’ prompt or vice versa.” However, Althaus and Tewksbury still seem to prefer the typical week variant compared to the typical week in the past year given the lack of strong evidence and on the assumption that “subjective reports of news exposure often overstate news use….” (p. 14-5). This may be true, but I thought there would be no clear recommendation in the typical week vs. typical week in the past year comparisons given the decision rendered in the days per week versus minutes per day experiments. To be fair, however, they conducted several auxiliary analyses and probably reached that judgment after careful consideration.
- Information Processing Variables. I was delighted to see some indicators for testing theories of information processing, particularly the decisiveness and closed-mindedness indicators of the need for closure concept. Althaus and Tewksbury write, “Both [decisiveness and closed-mindedness] represent defensive processing goals and should therefore be associated with increased polarization of perceptions of where candidates and parties stand. High levels of defensive processing goals should reliably predict greater perceived issued distances between candidates and parties” (page 3).
One question I had was this: why should decisiveness or closed-mindedness result in polarized judgments of party distances? Part of the logic could rest on the tendency for some respondents to default to the middle option, but party placements should depend on where the parties actually are located in the policy space or, perhaps even more importantly, on the communications about where the parties stand. In other words, I wonder whether it is possible to expect polarization or consensus independent of the messages that are being sent about where the parties stand. In some cases it could be possible to envision very consensual locations but in others the parties might be seen as different. I am not sure I would expect these outcomes in the absence of knowledge about what the messages or media were. I agree that being closed-minded can inhibit new information acquisition (Barabas 2004), but what respondents report as a candidate or party location will depend on what they believed prior to acquiring the new information. Also, need for cognition, which is another psychological measure that was available on the 2004 ANES, was negatively associated with perceived national political party issue distances, but it was unclear how this measure was related to the others.
Finally, while Althaus and Tewksbury should be congratulated for importing psychological concepts into the study of information processing, they rely heavily on the work of Kruglanski et al. as well as theories of elaboration likelihood. Both are worth considering, but there are other perspectives as well (e.g., motivated reasoning, online processing, Bayesian updating). I suspect resource constraints limited forced Althaus and Tewksbury to limit the scope of their inquiry, but it is worth noting that there are other theoretical models worthy of testing.
- Issue Distances vs. Issue Placement Accuracy. Althaus and Tewksbury include detailed Stata code in the appendices to their report. This is enormously helpful. Their analytical decisions are especially transparent for the 2004 ANES data analyses and it helps to make it possible for other researchers to replicate their findings or, as the case may be here, to engage in a constructive critique of particular choices. In that spirit, earlier I questioned the use of candidate and party issue distance. In Appendix C, the issue distance scores appear to be based upon the absolute value of the difference between two separate issue ratings (e.g., gen ideol_cand_dist=abs(V043087-V043088). The same computer code in Appendix C indicates that Althaus and Tewksbury also created a dichotomous measure of whether individuals were accurately placing one candidate or party to the right or left of another (e.g., ideol_cand=1 if V043087>V043088), but the accurate placement measure was not used in the analysis as far as I could tell. I view this operationalization of accuracy to be superior to the issue distance measures used in the report and to be more like the other political knowledge measures. What do the analyses reveal with the accuracy outcome? Also, based upon the coding decisions shown in Appendix C, the issue distance measures recode any missing data to zero (e.g., replace ideol_cand_distance=0 if ideol_cand_dist = =.), which imputes a value of no distance (or, the lack of polarization) for anyone who did not offer a rating on the items. Are the results sensitive to this coding decision?
Despite my questions, there are numerous bright spots and convincing analyses that I cannot detail here due to space limitations. I encourage others to review the entire report, or at the very least, the helpful executive summary. Moreover, some of the most important findings hold when compared with another survey as I will discuss in the next section.
Comparing the 2006 Media Exposure Measures to the Same Items in a 2007 Survey
The media usage questions that Althaus and Tewksbury proposed for the 2006 pilot survey were posted on the public Online Commons forum hosted on the ANES website (http://www.electionstudies.org/onlinecommons.htm). My colleague, Jennifer Jerit, and I were quite interested in the exposure measures so we employed several of the 2006 ANES pilot items in a 2007 survey. The results reported below are based upon a survey that was conducted during from March 1 to March 21, 2007 by Polimetrix, an internet survey research company (http://www.polimetrix.com/). A total of 1,500 respondents, with an oversample of adults age 55 and older, were selected from a larger pool based upon demographic characteristics (e.g., gender, age, race, education, party identification, and ideology) and the data were ultimately weighted to reflect known marginals for the general population of the United States from the 2005 American Community Survey. Although the methods of recruitment and modes of interviewing differ (ANES interviewed via the telephone), we can examine the degree to which the findings hold in another context.
Table 1 shows the average self-reported media exposure for the typical week.1 The 2006 ANES items are shown first (the table shows the mean, standard errors, and the number of cases), the 2007 Polimetrix data appear next (with the same set of statistics), and the difference between the two is shown last along with a t-test value on the difference. All the analyses employ the sampling weights, which should help make both surveys nationally representative from a demographic standpoint, so the numbers in Table 1 for the ANES 2006 may differ modestly from what Althaus and Tewksbury report in the first column of their Table 1. Reading across the first row, the average ANES respondent reports watching television news 5.29 days per week and the average Polimetrix respondent watches it a bit less at just under five days per week (4.61).2 This is the most common form of media exposure in both surveys, but the Polimetrix respondents watch television roughly one day a week less than the ANES respondents.
The estimates for print newspaper exposure and radio news exposure are even closer. In fact, there are no statistically significant differences between the respondents in the surveys for these two forms of media exposure (at the p < .05 level). The typical ANES respondent reads a print newspaper 3.32 days per week while the average Polimetrix respondent reads the paper 3.08 days per week. The typical ANES respondent listens to the radio 2.68 days per week as compared to the 3.02 days per week average for the Polimetrix respondents. The differences of -.24 and .34 respectively are not statistically significant, although the radio difference represents a borderline case (p < .10).
Finally, the largest differences that are observed are with the internet news usage. The ANES respondents use that medium the least, averaging 2.37 days per week. In contrast, the Polimetrix respondents used it 4.59 days per week, nearly as much as they did for television. The 2.22 day per week difference in internet usage might not be surprising. The Polimetrix respondents, by virtue of the way they are selected and interviewed, all have access to the internet. Indeed, these numbers might be closer to what one might expect given the mass diffusion of the internet.3
One might be tempted to attribute the internet usage differences to a descriptively unrepresentative internet sample, but the ANES and the Polimetrix surveys showed no statistically significant differences in weighted comparisons of gender, percentage African-American, income, education, age, and partisanship. Also, it is important to remember that there were differences in survey administration (ANES asked about the media late in module 18 of the test pilot survey, Polimetrix asked about the media early, but after some other media items). The order of the items also differed (according to the 2006 ANES codebook the order is as follows: internet [V06P664], newspaper [V06P666], tv [V06P668], and radio [V06P670]; for Polimetrix the order was tv, newspaper, radio, and then the internet). Furthermore, the ANES respondents are panelists who first participated in the 2004 ANES, although not all of them were re-interviewed in 2006. As such, these analyses use sampling weights for both ANES and Polimetrix in an effort to be nationally representative from a descriptive standpoint. However, the analyses are not sensitive to this decision.
Perhaps most importantly, the media exposure measures in the ANES and Polimetrix perform similarly in at least one key area: predicting political knowledge. In Table 3 of their report, Althaus and Tewksbury note that the typical week measures for newspapers, internet news, and radio news are positively associated with predicting general political knowledge (see column 1 of their Table 3). In their analyses, television news usage was unrelated to political knowledge for the typical week measures. The analyses in Table 2 with the 2007 Polimetrix survey confirm those same patterns, albeit using more detailed knowledge items. The Polimetrix survey included four items that correspond to general political knowledge and ten that are policy-specific in nature.4 In the first column of Table 2, an OLS regression shows that each added day of radio news and internet news usage increases the predicted level of general political knowledge by .03 and .06 respectively (standard errors of .01 for each; p < .05). Newspapers and television are not related to knowledge in this model, but newspapers do predict policy-specific knowledge in the second column of Table 2 (coeff.=.04; s.e.=.02; p < .05). Since the outcome in each case is a count variable (i.e., the number of correct answers), the third and fourth columns show that these results hold when the specification is changed to a Poisson regression. In addition, all of the models in Table 2 control for gender, income, education, race, age, and partisanship. Many of these are statistically significant predictors in ways that confirm prior research on political knowledge.5
Thus, three media exposure measures—newspapers, radio, and the internet—all appear to be associated with high levels of knowledge. Television news usage was not related to knowledge. Taken together, these patterns confirm what Althaus and Tewksbury found regarding the relationships between the media exposure items and political knowledge. However, the preliminary analyses are admittedly more tentative than what appear in the thorough report that Althaus and Tewksbury compiled. The results might change substantially once an attempt is made to recover some of the missing demographic data (especially for indicators like income) which regrettably leads to the omission of hundreds of cases from the analyses.6
Some Remaining Concerns in Media Exposure Research
Beyond concerns about social desirability (i.e., do respondents tend to over- or under-report their true media usage?), which Althaus and Tewksbury mention in their report, there are other persistent methodological challenges associated with media exposure analyses. Three of the most problematic are the following:
- Reverse causality. Employing a media usage variable as a predictor in a regression equation does not necessarily mean that it is a causal variable. That is, knowledge, or whatever outcome one is trying to explain, could be influencing media usage rather than the reverse. This is a long-standing concern (e.g., Clarke and Fredin 1978; Mondak 1995), and there have been some encouraging unidirectional results in some recent studies (Eveland et al. 2005) after earlier results pointing to reciprocal causation (Eveland, Shah, and Kwak 2003). Nevertheless, causality is hard to pin down in most cross-sectional studies and the evidentiary basis for media exposure exogeneity likely varies from case to case.
- Selection-bias. Usually media exposure is not randomly assigned. That means empirical patterns could be due to underlying differences between those who opt to use one medium compared to another. Analysts often include demographic variables to control for some of these differences, but it is hard to rule out selection threats entirely.
- Inconsistent scale usage. Generic media usage or attention variables are often subject to problems when it comes to making interpersonal comparisons. One person’s report that they use the media “often” could be the same as another person’s “somewhat” due to biases in the ways that they read the question (see King et al. 2004 for more). Creating scales that combine several measures may only serve to compound the problem. Fortunately, the measures Althaus and Tewksbury employ largely bypass these problems due to the common time-based metric (e.g., days per week), which should be the same across respondents. Whether or not these measures are used consistently remains to be seen, especially in light of the back-of-the-envelope calculations noted earlier in which days per week times minutes per day does not necessarily equal minutes per week.
These concerns are not new; some have been raised in the literature for decades. Nevertheless, media use variables are often plagued by methodological problems that cannot be corrected easily. In the face of these problems, researchers might be tempted to conduct experiments in which they simulate media coverage and randomly assign it in an experiment. While superior from the stand point of causality (i.e., internal validity), these studies might not be generalizable, both with respect to the subject pool and the media message treatments that are employed.
At an even more fundamental level, however, what do media use variables say about the media effects of the media? One response might be: not much. Media exposure terms cannot pinpoint what it is about the media that matters because most studies of media effects do not include media content. In other words, even if we know someone used a particular source, it is equally if not more helpful to know what was in the source. This can require augmenting survey data with media content data. Althaus and Tewksbury are no strangers to content analysis and using media data (Althaus 2003; Althaus and Kim 2006; Althaus, Edy, and Phalen 2001; Althaus and Tewksbury 2002), but blending surveys and media data is not as common as one might expect (cf. Price and Czilli 1996). My colleagues and I have tried to address this problem by merging media content with survey data in several studies (Jerit 2008; Jerit and Barabas 2006; Jerit, Barabas, and Bolsen 2006; Simon and Jerit 2007). Doing so often requires advanced statistical techniques such as multilevel modeling or related methods of acknowledging the clustered error structure. However, there might be other, even more straightforward ways of estimating the effects of media coverage that bypass the problematic aspects of media usage measures by conducting intra-individual comparisons on topics with multiple questions and varying levels of coverage (Barabas and Jerit 2008).
But even if one were to measure and include media content, it begs the question of whether the 2006 ANES media usage measures are specific enough. Some non-ANES surveys go beyond general categories such as television or newspapers to ask about particular sources. Unfortunately, the ANES measures are fairly general in nature (and ANES appears not to have asked some of the more detailed channel specific measures in Appendix B that appeared in the original Althaus and Tewksbury proposal). Are all television sources equal? Does every paper convey the same information? For some events, the answer might be yes, but for others there might be a lot of heterogeneity within particular media and days per week might be too rough to detect these variations. Contrary to what I stated earlier regarding the usefulness of the internet and radio usage items, content for these sources is hard to obtain. It might be that the generic internet and radio usage measures are the best we can do unless researchers are willing to ask additional follow-up questions about what sites or stations respondents use. Even then, coding them might not be possible, and it is in this manner that studies on the degree to which multiple media sources carry the same information become useful.
Conclusion
The title of the Althaus and Tewksbury report is “Toward a New Generation of Media Use Measures.” Have they delivered on that title? Yes. I believe they have moved us down the path and even helped clarify the value of some measurement variations (i.e., minutes per day measures). In addition, many of the measures they propose work similarly in different survey from 2007. Nevertheless, I think there are still questions about what those measures are measuring and what an ideal set of exposure items, if one exists, would look like.
We might be able to think of superior measures within a given context or to measure a particular event, but Althaus and Tewksbury set the bar high by seeking longitudinal measures. That is, Althaus and Tewksbury set out to explore media exposure measures that work across time, not for any particular administrations of the ANES. Information recall items that are election-specific or situation-specific (e.g., Price and Zaller 1993; Jerit, Barabas, and Bolsen 2006) are unlikely to be useful from one context to the next. So, while we might be able to design better and even find that specific surveillance measures outperform media use in a particular context, can we design something that is more general but still valid? Scholars constantly face tradeoffs when designing and conducting their research. The questions become, however, have we given up too much ground on the internal validity side (i.e., we abandon causal claims) and what other validity disadvantages (or advantages) have we created?
Althaus and Tewksbury note that ANES has used many different operationalizations of media exposure over the years, so standardization seems wise. In that sense, it is doubtful that Althaus and Tewksbury could or would even want to propose wholly different media use items on future ANES surveys if for no other reason than in trying to maintain some ability to compare measures across time. Hence, in this instance there is a good rationale for retaining some core measures, no matter how problematic they might be. Yet, Althaus and Tewksbury still find ways to innovate (e.g., radio and internet usage) and they should be commended for making realistic recommendations to the ANES board that do not increase the total number of new questions. In sum, then, the findings in the Althaus and Tewksbury report are extremely valuable. I am sure many will agree that the report has wide-spread implications, and the findings are likely to influence multiple generations of political communications researchers.
Jason Barabas is an Assistant Professor in the Department of Political Science, Florida State University.
ENDNOTES
1. The Polimetrix survey used the “typical week” question wording that appeared in the 2006 ANES, so these comparisons are limited to that subset of the ANES survey, not the respondents who were asked about the typical week in the past year. The specific wording for each of the Polimetrix (and ANES) items is as follows: TV News: “During a typical week, how many days do you watch news on TV, not including sports?”; Print Newspaper: “During a typical week, how many days do you read news in a printed newspaper, not including sports?”; Radio News: “During a typical week, how many days do you listen to news on the radio, not including sports?”; and Internet News: “During a typical week, how many days did you watch or read news on the Internet, not including sports?” Note, however, the internet question (and only this question) differed slightly from the ANES in that it used the verb “did” instead of “do.” Also, as was the case for the measures that Althaus and Tewksbury studied in the 2006 ANES (see footnote 5, page 12), all of the Polimetrix items include the phrase “not including sports.”
2. Like the self-reports on television news usage in the 2006 ANES, 42% (623 of 1,496) of the Polimetrix respondents reported watching television seven days a week.
3. In the Polimetrix survey, roughly 43% of the sample claims to use the internet every day while 15% uses the internet for news one day or less. In the 2006 ANES, 20% use the internet everyday and 54% use it one day or less.
4. The general knowledge measures were knowledge of which political party controls the US House, identification of Harry Reid as Senate Majority leader, knowledge of which branch of government declares laws unconstitutional, and the percentage needed in Congress to override a presidential veto. The policy-specific items all concerned details on the Social Security and Medicare programs (e.g., the relative financial strength of each program, Medicare spending vs. foreign aid, whether or not the President proposed negotiating with drug companies for Medicare prescription drug coverage, Social Security tax thresholds, and others). All of the items were scored as correct (=1) or incorrect/DK (=0) and then used to form separate additive indices of general and policy-specific knowledge.
5. Althaus and Tewksbury controlled for all of these factors, with some minor scaling differences, except for age. In addition, the models that they employ in Table 3 of their report include partisan extremity, but these models do not employ that variable. In addition, the knowledge measures they use are lagged (i.e., from the 2004 study), while these are contemporaneous.
6. If the demographic and control variables are omitted—and thus the listwise deletion is reduced—newspapers, radio, and the internet predict general knowledge (p < .05) and radio, the internet, and television are associated with policy-specific knowledge (p < .08). However, multiple imputation techniques can be used to recover the missing demographic data so that the analyses can include the same control variables which appeared in Table 2.
REFERENCES
Althaus, Scott L. 2003. Collective Preferences in Democratic Politics: Opinion Surveys and the Will of the People. Cambridge: Cambridge University Press.
Althaus, Scott L., Jill A. Edy, and Patricia F. Phelen. 2001. “Using Substitutes for Full-Text News Stories in Content Analysis: Which Text Is Best?” American Journal of Political Science 45 (July): 707-24.
Althaus, Scott L., and Young Mie Kim. 2006. “Priming Effects in Complex Information Environments: Reassessing the Impact of News Discourse on Presidential Approval.” Journal of Politics 68 (Nov.): 960-76.
Althaus, Scott L., and David Tewksbury. 2002. “Agenda Setting and the ‘New’ News: Patterns of Issue Importance Among Readers of the Paper and Online Versions of the New York Times.” Communication Research 29: 180-207.
Barabas, Jason. 2004. “How Deliberation Affects Policy Opinions.” American Political Science Review 98 (Nov.): 687-701.
Barabas, Jason, and Jennifer Jerit. 2008. “Estimating the Causal Effects of Media Coverage on Policy-Specific Knowledge.” Unpublished manuscript, Florida State University, Tallahassee, Florida.
Bartels, Larry M. 1993. “Messages Received: The Political Impact of Media Exposure.” American Political Science Review 87 (June): 267-85.
Chang, LinChiat, and Jon A. Krosnick. 2003. "Measuring the Frequency of Regular Behaviors: Comparing the 'Typical Week' to the 'Past Week.'" Sociological Methodology 33: 55-80.
Eveland, William P. Andrew F. Hayes, Dhavan V. Shah, and Nojin Kwak. 2005. "Understanding the Relationship Between Communication and Political Knowledge: A Model Comparison Approach Using Panel Data." Political Communication 22: 423-46.
Eveland, William P., Dhavan V. Shah, and Nojin Kwak. 2003. "Assessing Causality in the Cognitive Mediation Model: A Panel Study of Motivations, Information Processing, and Learning During Campaign 2000." Communication Research 30 (Aug.): 359-86.
Jerit, Jennifer. 2008. “Issue Framing and Engagement: Rhetorical Strategy in Public Policy Debates.” Political Behavior, forthcoming.
Jerit, Jennifer, and Jason Barabas. 2006. “Bankrupt Rhetoric: How Misleading Information Affects Knowledge about Social Security.” Public Opinion Quarterly 70 (Fall): 278-303.
Jerit, Jennifer, Jason Barabas, and Toby Bolsen. 2006. “Citizens, Knowledge, and the Information Environment.” American Journal of Political Science (Apr.) 50: 266-82.
King, Gary, Christopher J.L. Murray, Joshua A. Salomon, and Ajay Tandon. 2004 “Enhancing the Validity and Cross-cultural Comparability of Measurement in Survey Research .” American Political Science Review 98 (February): 191-207.
Mondak, Jeffrey J. Nothing to Read: Newspapers and Elections in a Social Experiment. Ann Arbor, MI: University of Michigan Press.
Price, Vincent, and Edward J. Czilli. 1996. "Modelling Patterns of News Recognition and Recall." Journal of Communication 42 (Spring): 55-78.
Price, Vincent, and John Zaller. 1993. “Who Gets The News? Alternative Measures of News Reception and Their Implications for Research.” Public Opinion Quarterly 57 (Summer): 133-64.
Simon, Adam F., and Jennifer Jerit. 2007. “Toward a Theory Relating Political Discourse, Media, and Public Opinion.” Journal of Communication 57: 254-71.
|
