This is a continuation of a series on how the "liberal media" myth is created. Previous installments covered myth-creation using "tone" of media coverage (Part 1), "catch-phrases" like 'right-wing extremist' v. 'left-wing extremist' (Part 2), "newspaper headlines" (Part 3) and "topics" covered (Part 4). This part highlights an unusual, indirect approach that uses "think-tank" citations. 

The focus of this post is a paper titled "A Measure of Media Bias" by Tim Groseclose (UCLA) and Jeff Milyo (Univ. of Chicago). I found this paper via Language Log (there has been some back and forth at Language Log between critic Geoffrey Nunberg and the paper's authors), where it was also noted that:

Groseclose and Milyo's study has been approvingly cited by Bruce Bartlett in National Review, by Linda Seebach in the Rocky Mountain News, and by Harvard economist Robert J. Barro in Business Week,  not to mention conservative bloggers like Instapundit, Andrew Sullivan, and Matt Drudge, among a number of others, who trumpet its "objectivity."

A single blog post, once again, is insufficient to provide a detailed critique of the paper. So, I'll refer readers who are more curious to my detailed critique over at ICM - Sec. 2.9. Here, I'll reproduce my summary (with links to details) showing why this paper's conclusions are wrong.

The Gloseclose-Milyo (G-M) paper attempts to assess media bias using an approach wherein adjusted ADA (Americans for Democratic Action) scores (0-to-100) are used to assess legislator ideology (archconservative-to-archliberal), and separately, the think-tank citations of the legislators are compared to the think-tank citations of the media outlet to then derive the media outlet's "bias". Based on their methodology (presented and discussed in this paper), they claim that:

...we find a significant liberal bias in our sample of media outlets.

They also claim, for instance, that their data shows the Brookings Institution to be left-leaning and not centrist. [NOTE: I use the word think-tank in the same sense in which the authors use it - to describe not just traditional think-tanks, but advocacy groups as well; this is a debatable definition but irrelevant to this critique.]

I examined the paper from three perspectives:
1. Is the methodology used for assessing the ideology of think-tanks correct and reliable?
2. Is the methodology used for assessing the ideology of the media correct and reliable?
3. Is the definition of media bias used by the authors correct and reliable?

The answers to each of those questions is NO. 

Why?

The methodology used by the authors for assessing think-tank ideology (i.e., based on the average adjusted ADA score of the legislators citing the think-tank) is deeply flawed because it omits public or private disagreements that legislators have with the same think-tank and it does not account for the fact that legislators may agree with a think-tank but not state it publicly for various reasons (e.g., they are unaware of the think-tank; they are aware of the think-tank but the latter may not be known well enough to cite, it may be a "controversial" think-tank, there may be no need to cite a think-tank, etc.). This can effectively skew their results in the wrong direction, to an unknown degree. Indeed, the fact that their methodology found the ACLU to be "conservative" was a result of the former flaw. To address this, they say in page 8: "If we omit ACLU citations that are due to [Republican Senator] McConnell, then the average score, weighted by sentences, increases to 70.12" [which is a "liberal" score]. Unfortunately, that is the wrong approach to fix this problem. The way to fix this problem is to actually ADD all those instances in which Republicans actually disagreed with ACLU, not incorrectly and artificially remove situations where *they agreed with ACLU* in order to get an average score that seems more in sync with a *separately established* reality. In other words, if we already knew ACLU is "liberal" and need to know that to "adjust the data", then what is the value or point of this study?

Additionally, a legislator may cite a think tank not because he or she mostly agrees with the think tank but because that think tank's view is closer to his or her view than any other think-tank the legislator is aware of or cares to cite. It is very unlikely that legislators who cite a think tank agree with everything the think tank says or stands for. For example, some legislators may cite it because their position is in agreement with, say, only one or two or three of the think tank's positions and they may cite it for that reason, repeatedly (like in the ACLU case). The bottom line is that their think-tank ideology ratings are unreliable and incorrect. [More details here.]

The methodology used by the authors for assessing media ideology is completely untenable. There are three principal reasons for this:

(a) The approach G-M use establishes media ideology indirectly, by using the media's think-tank citations and comparing those to think-tank citations by legislators in order to find the legislator whose citations are the closest match. Thus, if a legislator is liberal and the media's think-tank citations match that of the liberal legislator, they would declare the media to be liberal. Momentarily setting aside the fact that this definition of media bias is itself incorrect, their claim would make sense only if it can be independently proven that the think-tanks cited by the liberal legislator are actually liberal. Their study does not prove this at all, considering that their methodology to establish think-tank ideology is itself deficient. Thus, at a fundamental level, their entire conclusion on media bias breaks down. (NOTE:  It is not at all implausible that left-leaning legislators may cite more centrist think-tanks in public than progressive/liberal ones, especially considering how the liberal advocacy groups and think-tanks are tarred negatively by the GOP in the illiberal conservative media).

(b) G-M use the Median adjusted ADA score for the House to determine the "center" for defining who is liberal and who is conservative. The median approach is fatally flawed and can dramatically, and incorrectly, skew the derived ideology of the media in a direction that is opposed to the ideology of the majority party in Congress. Consider an example where there are 400 members in the House, with 210 arch-conservatives (ADA score = 0 for each) and 190 arch-liberals (ADA score = 100 for each). Say, there is a media outlet A which shares its ideology with one of the arch-conservative members (so media outlet ADA score = 0). Then:

• Median ADA score for House = 0
• Mean ADA score for House = 47.5.
• Per Median approach, Media Outlet A ADA score = 0 = "centrist"
• Per Mean approach, Media Outlet A ADA score = 0 = "conservative" [which is the reality]

[NOTE: If the House is comprised of majority liberals and minority conservatives (flip the numbers in the above example), the median approach would say that even if the media were aligned with one of the arch-liberals (ADA score = 100), it is "centrist".]

(c) The use of the Mean adjusted ADA score for the House is slightly more meaningful than the Median, but even this is completely deficient and incorrect because the ideological center is set not using an independent, objective measure of ideology but based on the (political) positions of the people in the House at a given point in time. Thus, their model simultaneously assumes that ADA scores can provide an absolute picture of a legislator's ideology, but that media- and think-tank ideology should be determined not using the same absolute reference but a relative, moving reference that is highly dependent on who's the majority in Congress and what their ideological position is relative to the minority (e.g., the mean ADA score of the House). This is not an acceptable model, for, if the minority party becomes the majority party in the next election, think-tanks or the media would suddenly appear to have switched ideologies even though their ideological positions underwent ZERO change. 

Alternately, if the Republican majority suddenly decides to become 100% conservative (median adjusted ADA score = 0), guess what happens. The mean ADA score would also drop, even if the liberals DID NOT change at all. So, imagine a scenario wherein the media is STATIC and the Republican legislators decide to take the country in a 100% conservative direction. Then, even though media's ideology has NOT changed, it's adjusted ADA score will artificially look more and more liberal compared to the House median or the mean! (BONUS FOR LEFTIES: This is right in line with one of the long-time Republican strategies of declaring the media (and Democrats) to be too "liberal" by moving the country to the Right)

The final, and perhaps most serious, problem with their analysis is their attempt to derive a conclusion of media bias using this study. Their confident conclusion that they have proven "liberal media" bias is simply wrong because the study does not examine whether the media's news reporting is accurate. Citing a think-tank says nothing about whether that think-tank is accurate or not. For example, you could have a "liberal think tank" that is 100% correct and cited 70% of the time and a "conservative think-tank" that is 10% correct and cited 30% of time (without corrections) and this would not make the media outlet automatically "liberal" - indeed, giving that much credence to a think-tank which lies or misleads most of the time could easily constitute conservative bias. Moreover, simply looking at think-tank citations certainly says nothing about what the media communicates to the viewers when it is not citing think-tanks, which is a big chunk of the time. 

When controlled for other factors, the more fundamental determinant of bias in news reporting is accuracy -- not whom the news reports cite. To the extent that news reporting could become inaccurate by citing certain think-tanks over others, one may have a case that think-tank citations could influence the accuracy of the reports. But, G-M have fallen into the trap of assuming that the part is the whole. Think-tank citations are merely one part of the whole - which is the media's accuracy in news reporting.