This is the continuation of a series on how the "liberal media" myth is created. Previous installments covered how this myth is created using "tone" of media coverage (Part 1), using "catch-phrases" like 'right-wing extremist' v. 'left-wing extremist' (Part 2), and using "newspaper headlines" (Part 3). In this part, I address a fourth (superficial) approach used for creating a myth of "liberal media" - "topics" covered. [A few more parts can be expected over the rest of this week].

The focus of this part is the 2004 paper, "Being the New York Times: The Political Behaviour of a Newspaper" by Riccardo Puglisi of the London School of Economics (LSE) (which I discovered via Marginal Revolution). I have provided a more systematic critique of this paper at Illiberal Conservative Media (ICM) - Sec. 2.10; here I will highlight some portions of that critique. 

The issue of topic choice is important in media bias analysis, but like everything else it has to be treated with some sophistication to eliminate false results/conclusions. As I have indicated before at ICM:

Topic choice is certainly a function of editorial bias, but it also a function of numerous other confounding factors - source credibility, events, circumstances, issues of public interest, issues of interest to politicians or policy-makers, issues of interest to the media outlet to ensure their revenues and profits in the markets they compete in, etc. So, it would be much more difficult to credibly demonstrate editorial bias on topic choice, by itself.

With that sentiment, let's look at Puglisi's paper, starting with his abstract (bold text is my emphasis):

I analyze a dataset of news from the New York Times, from 1946 to 1994. Controlling for the incumbent President’s activity across issues, I find that during the presidential campaign the New York Times gives more emphasis to topics that are owned by the Democratic party (civil rights, health care, labour and social welfare), when the incumbent President is a Republican. This is consistent with the hypothesis that the New York Times has a Democratic partisanship, with some "watchdog" aspects, in that it gives more emphasis to issues over which the (Republican) incumbent is weak. Moreover, out of the presidential campaign, there are more stories about Democratic topics when the incumbent President is a Democrat.

Now, you can call me silly, but when someone derives such a sweeping conclusion based purely on a study of "topics" covered, my GIGO* detector goes on Orange alert. As I read more of the paper, I sadly realized the alert level had to be upgraded to Red. In my detailed critique of this paper, I've pointed out what I believe are six major problems with this paper (I, II, III, IV, V, VI). I can't do full justice to all of those points here - so I am simply going to condense my points here and refer interested readers to the full critique for details.

First, here are some assumptions stated by Puglisi for his study:

As briefly anticipated in the introduction, the empirical analysis performed here and the interpretation of its findings are based on the following set of identifying assumptions:

(1) The issue ownership hypothesis holds.

(2) “All publicity is good publicity”.

(3) The relative share of Executive Orders about a subset of issues proxies the relative intensity of the activity of the incumbent President with respect to those issues.

The issue ownership hypothesis, which Puglisi bases on historical polling data and mentions throughout, is the following: 

Democratic topics comprise Civil Rights, Health Care, Labor & Employment and Social Welfare. Republican topics comprise Defense and Law & Crime.

Now, it may be convenient to assign such ownership because it helps make the analysis more interesting, but really, someone "owning" the issue often has little to do with whether the publicity/coverage that person gets on that issue is good or bad. Thus, the second assumption, that "All publicity is good publicity" (referring to "owned issue" coverage for the person who owns it) simply makes no sense. For example, was "Health Care" coverage always "good publicity" for Bill Clinton (Democrat)? Was "Defense" and "Law and Crime" coverage always "good publicity" for Richard Nixon (Republican) and Ronald Reagan (Republican)? Was "Employment" and "Social Security" coverage necessarily always "bad" publicity for the Reagan administration? In other words, the assumption that if a newspaper reports on topics "owned" by a party, it automatically means that party benefits, makes no sense because such an assumption fails to account for the fact that newspapers, can and do issue reports on "owned" topics that may not be positive at all to the "owning" party. (If this seems like an odd assumption, wait till you see the second one, which effectively scuttles the papers' conclusions.)

Second, consider these "definitions" offered from Puglisi:

Definition 1 A newspaper has a Democratic (Republican) partisanship if during the presidential campaign it devotes more space to issues owned by the Democratic (Republican) party, at the expense of neutral or Republican (Democratic) issues.
...
In fact, over and above the electoral partisanship of the newspaper, as described by definition 1, the political colour of the incumbent President could be given an interpretation within a lapdog/watchdog dichotomy. The idea is the following: if it turns out that -during the presidential campaign- the New York Times gives less emphasis to Democratic topics and/or more emphasis to Republican topics when the incumbent is a Democrat, over and above his Democratic or Republican or lack of partisanship, this is consistent with the fact that the newsaper [sic] acts as an electoral watchdog with respect to the incumbent President.
...

Definition 2 The newspaper is an electoral lapdog of the incumbent President if ceteris paribus during the presidential campaign it devotes more space to the issues over which the incumbent is strong, and/or less to issues over which the incumbent is weak.

Definition 3 The newspaper acts as an electoral watchdog if ceteris paribus during the presidential campaign it dedicates more space to the issues over which the incumbent is weak, and/or less space to the issues over which the incumbent President is strong.

Where do I begin?

I don't mean this insultingly at all, but these definitions are ridiculously wrong because not only are they inconsistent with each other, the latter definitions are wrong on their face. For example, Puglisi's definition of "watchdog" in Definition 3 is highly convenient (and reveals possible built-in bias in his thinking about this problem) because I can just as well argue based on his Definition 1 that the newspaper is no "watchdog" but just a shill for the candidate opposing the incumbent and is therefore displaying "partisanship" in favor of the challenger. In fact, let's ignore Definition 1 completely and consider Definition 3 on its own. It is Puglisi's *opinion* that the newspaper serves as a "watchdog" by focusing on the topics that supposedly favor the challenger. One can easily have a different *opinion* that a newspaper doing this is a partisan supporter of the challenger and not a "watchdog"! (Talk about pre-ordaining the results!)

This is the natural (and fully expected) problem with studies of this nature which don't bother to actually analyze the content of the news articles. Thus, Puglisi's assumptions and definitions are wrong because at a very fundamental level, they neglect the actual nature of the coverage (accurate or inaccurate). So, combining Problem I and Problem II, this study and the interpretation of its results totally break down even before we get to the actual data. 

Third, by Puglisi's own admission (Tables 2 and 3), when we look at "All stories" that appeared in the New York Times in the period 1946-1994, the so-called Republican topics and so-called Democratic topics were only 21.7% (8.37% + 13.36%) of the total. Thus, this study claims to show "Democratic partisanship" (or otherwise) based on a study that essentially ignores over 78% of all stories published in the New York Times. Stunning.

For example, "Banking, Finance and Dom. Commerce" (14.66% of all stories) and "International Affairs" (13.22% of all stories) are not part of Puglisi's model because they are not "owned" by Republicans or Democrats. What category would "taxes" or "spending" or "budget deficits" fall under? This is one of the most important topics in all Presidential campaigns - which often make or break campaigns - and there's no mention of it in the analysis. Also, what category would draft-avoidance or alleged extra-marital affairs fall in? Other? Or is it "Law and Crime?" There's a whole slew of topics relating to the individuals or their policies, that fall into the supposed "non-owned" issue category, which have a habit of coming up frequently during campaigns. It is extraordinarily odd that one would simply ignore all that and still feel comfortable reaching sweeping conclusions of the kind the author has.

Fourth, Puglisi's paper does not consider seriously the fact that major events happen which have nothing to do with the "strength" of Democrats or Republicans. For example, George Bush Sr. started significant cuts to defense spending at the end of the Cold War and Bill Clinton continued this effort. When there are no major wars and when there is no overarching concern about national defense, there is no reason for papers to simply keep writing more articles about "defense" just because a Democrat is in power. This same argument applies to every topic under the sun.

It is also obvious that many topics are raised, especially in electoral campaigns, by the politicians who are campaigning. Not to mention, one of Puglisi's "findings" is that the coverage of "Republican topics" actually goes up significantly in the campaign coverage when the challenger is a Republican. This takes us right back to Problem II. Either the NYT has "Democratic partisanship" or it doesn't. It makes no sense to claim that it has "Democratic partisanship" and simultaneously say that "...under a Democratic incumbent there are more stories about Republican topics when the presidential campaign kicks in. This effect is quite strong in magnitude...". Why is the latter considered a "watchdog" behavior rather than "Republican partisanship"? After all, if part of the "results" point one way, it is sufficient for Puglisi to label it "partisanship" of one kind; yet, when another part of the "results" points in another direction, it is not partisanship in the other direction - it is "watchdog"ism! If that isn't bias convenient, I don't know what is.

Fifth, when I look at Puglisi's basic data tables 3 and 4 (in his paper - see footnote), even if one makes the assumption that Executive Orders get proportional coverage in the NY Times (as he does), the numbers I derived suggests that even when the New York Times' topics-coverage is normalized to Executive Orders, it provided more coverage overall on the "Republican" topics than on the "Democratic" topics (I invite readers who are more statistics-aware to comment on whether I made any mistakes in my assumptions/calculations because I am not a statistics expert). This seems to partly contradict his main conclusions (even if you ignore the fundamental flaws I discussed above).

Sixth, Puglisi's study lacks any *real* control for comparison. Even if we assume that the results of this study are correct (which they are not), how can someone claim that a paper is partisan without even evaluating another paper - with an ideology known to be conservative - to see whether that paper's topic coverage was similar or the opposite?

All in all, this is a deeply flawed paper that certainly does NOT prove ANY liberal bias or Democratic "partisanship" on the part of the New York Times. But it helps us learn yet another way media bias myths are propagated.

P.S.: GIGO = Garbage In Garbage Out

*FOOTNOTE ADDED 4/4/05: It has been brought to my attention that the version of the paper I had originally linked to and analyzed is not the final version of Puglisi's paper. The latest version is available for download here. I apologize for this inadvertent /unintentional error. Given this, I have made appropriate (minor) modifications in my detailed analysis at ICM, and in the post above, to reflect the content and pagination in the final version of the paper. Having said that, Puglisi's conclusions or my critiques of his assumptions, data or conclusions have not changed with the latest version of his paper. Thus, the substance of my critique remains unchanged.