2. Conservative Books and "Studies" Alleging "Liberal Bias"

2.10 PAPERS: Other papers claiming the presence of a "liberal media"

2.10A PAPER: "Is Newspaper Coverage of Economic Events Politically Biased?" by John Lott and Kevin Hassett (of the American Enterprise Institute) (2004)

2.10B PAPER: "Being the New York Times: The Political Behaviour of a Newspaper" by Riccardo Puglisi (2004)

2.10A PAPER: "Is Newspaper Coverage of Economic Events Politically Biased?" by John Lott and Kevin Hassett (of the American Enterprise Institute) (2004)

A copy of this "paper" is here (a PowerPoint type summary from the authors is here).

The abstract of this paper says the following (bold text is eRiposte emphasis):

Accusations of political bias in the media are often made by members of both political parties, yet there have been few systematic studies of such bias to date. This paper develops an econometric technique to test for political bias in news reports that controls for the underlying character of the news reported. Our results suggest that American newspapers tend to give more positive news coverage to the same economic news when Democrats are in the Presidency than for Republicans. When all types of news are pooled into a single analysis, our results are highly significant. However, the results vary greatly depending upon which economic numbers are being reported. When GDP growth is reported, Republicans received between 16 and 24 percentage point fewer positive stories for the same economic numbers than Democrats. For durable goods for all newspapers, Republicans received between 15 and 25 percentage points fewer positive news stories than Democrats. For unemployment, the difference was between zero and 21 percentage points. Retail sales showed no difference. Among the Associated Press and the top 10 papers, the Washington Post, Chicago Tribune, Associated Press, and New York Times tend to be the least likely to report positive news during Republican administrations, while the Houston Chronicle slightly favors Republicans. Only one newspaper treated one Republican administration significantly more positively than the Clinton administration: the Los Angeles Times’ headlines were most favorable to the Reagan administration, but it still favored Clinton over either Bush administration. We also find that the media coverage affects people’s perceptions of the economy. Contrary to the typical impression that bad news sells, we find that good economic news generates more news coverage and that it is usually covered more prominently. We also present some evidence that media treats parties differently when they control both the presidency and the congress.

Why have I highlighted specific words? Well, when you read about the methodology they use, you'll understand (bold text is my emphasis):

In this paper, we attempt to overcome these problems by objectively categorizing newspaper headlines as either positive, negative, neutral or mixed and then comparing those headlines to the actual economic numbers that generated those news articles.

They study newspaper headlines and not the actual content of the articles (wow) - and I would bet that anyone who reads the abstract of the paper could easily miss this point, namely that their "study" is based on headlines - not "news reports", "news coverage", "stories", "news stories", etc. I imagine it must be particularly busy out there at AEI where both these authors work. 

Now, they do acknowledge this silliness (not in so many words of course), among other things (bold text is my emphasis):

We chose headlines because they create the strongest image of the news in readers’ minds, and because headlines are easier to objectively classify, though the headlines we examine may differ systematically from the stories they are associated with. While newspapers write other news stories on the economy that do not coincide with the specific release of economic data, one benefit of limiting ourselves to these announcement dates is that we can more directly link a specific set of economic data to how the media covers that data. It is possible that these other news stories are biased in ways that are different from stories released on announcement dates, and thus announcement date coverage might not give the complete picture of any partisan biases. The values for the different economic variables were those released at the time of the news reports.

So, let's recap.

• They only look at headlines; they don't look at the actual content of articles

• They only consider headlines associated with articles that coincide with the release of the economic data and not at any other articles that may be published about the same data subsequently

• They acknowledge that "It is possible that these other news stories are biased in ways that are different from stories released on announcement dates, and thus announcement date coverage might not give the complete picture of any partisan biases"

• And predictably, they make firm conclusions from their data anyway

Let's just say I didn't have it this easy in graduate school. And they actually get paid big bucks to write this stuff up, while I have to do this on my own dime. 

Now, I do concede that people who only skim through headlines may potentially get misled by badly written headlines. But, even if set aside the fact that the headlines, however "negative" they were, could have actually been accurate or appropriate, Lott and Hassett's study did not even bother to assess headlines on all articles about the same economic news. They certainly didn't look at the actual content either. And despite the ridiculously limited nature of their exercise, they arrive at firm conclusions anyway. (As a side note, as far as I know copy editors are the ones who usually write headlines, not reporters / journalists themselves, and there's quite a movement to teach them how to do it better, in general). 

At this point, people with a reasonable level of intelligence should be wondering: "What the heck is this paper worth?" Reasonable people already know the answer ("nothing"). But "research papers" get circulated regardless of how poor they are (especially ones that are not peer reviewed), so, unfortunately someone has to say "more" about it to sound credible these days.

It turns out, that whenever John Lott publishes something, a useful place to start is usually Tim Lambert's Deltoid blog. So I checked in and sure enough Lambert did cover this paper, here. Let me reproduce his post because it tells you "more" about why this paper is worthless:

Lott finds more bias

Lott has teamed up with Kevin Hassett to study whether economic reporting is biased. The paper, Is Newspaper Coverage of Economic Events Politically Biased?, concludes, surprise, surprise that the newspapers are biased against Republicans.

The trouble with their study is that the economy was stronger under Clinton than under either Bush, so of course the reporting of the economy under Clinton was more positive. Lott and Hassett claim to have controlled for this with a multivariate analysis but you should only find this persuasive if you have complete confidence in the competence and integrity of the authors. When building such models there are so many choices you can make that it easy to get the results that you want to see. In this case it is particularly easy, since all you have to do is leave out a relevant variable so that the state of the economy is not fully controlled for and you will get results like Lott and Hassett report.

We probably should not have complete confidence in Lott and Hassett. Lott’s previous attempt to show that the media was biased seems to have involved cherry picking of models, and careful selection of models and data to get a desired result seems to be a constant characteristic of his work. As for Hassett, I’ll let Lott tell you about him. This is Lott’s anonymous review of Hassett’s Bubbleology:

Despite Mr. Hassett’s track record with his previous book “Dow 36,000,” I saw him appear on CNBC during the early morning show and thought that he did well enough that I should buy the book. He promised that you could use his book to figure out what stocks were overvalued and which ones weren’t. A pretty important topic given the current market environment. However, after reading this short book I have no idea of how to actually rank stocks on the 1 to 6 scale that he uses. He doesn’t actually provide concrete examples, only that he says that he put together this ranking and it worked really well. My other problem is that if this approach works so well how come he didn’t use it when his “Dow 36,000” book came out when the stock market was at its peak. Some explanation would have been useful for why Hassett, who is marketing this book as a full proof approach to spotting bubbles, wasn’t able to use this approach himself over just the last couple of years to warn people and predict which stocks were going to crash, a period when he was supposedly writing this book. Claiming that you use a not clearly stated formula to identify overvalued stocks after they have already crashed seems like a scam to me.

“seems like a scam”, indeed.

The New York Times has published what Lott calls a “hit piece” on their study. The Times gives their study more credence than it deserves (Atrios is disgusted that they even did a story on him), but they do eventually mention Mary Rosh and the 36,000 Dow prediction and finish with an apposite quote from Brad DeLong:

To even base a story on Lott’s work at this point in time is to demonstrate a pronounced bias toward right-wing hacks

Update: The AEI is holding an event to promote the study and the official George W Bush blog is also pushing the study. The gang at Lawyers, Guns and Money get stuck into Lott and Hassett here and here.

Update 2: Brad DeLong points out that the NY Times reporter dropped the ball by letting Lott’s deceitful statement that “the things he had said in the guise of Ms. Rosh were, indeed, truthful” stand. You can see them here and count how many are plainly untrue.

Let's continue with a thread from the comments section of Lambert's post above, that by itself shows why this paper is complete nonsense and proves nothing.

Xrlq 13/9/2004 10:57:22

The trouble with their study is that the economy was stronger under Clinton than under either Bush...

Define "stronger." Today's unemployment rate is identical to that of 1996.

Tim Lambert 13/9/2004 12:14:29  
The unemployment rate decreased under Clinton and increased under Bush.
... 

jre 13/9/2004 22:59:44
...
Tim is right about unemployment; in fact he could have put it much more forcefully.
Have a look at the numbers.

During Clinton's two terms ('93 to '01), unemployment declined monotonically, while it rose monotonically under Bush I and II. (If you go to the BLS, get the PDF file. It's an eye-opener.)
Or check the figures for GDP.
Average annual GDP increase over the four years of Bush I was 1.9%. For the eight years of Clinton, it was 3.6%, and for the first three years of Bush II, it has been 2.8%.

And as for the deficit -- aaargh, don't get me started.

This case is clear-cut: the US economy did extremely well during the eight years of the Clinton presidency, so much so that when someone argues otherwise I have to wonder if that person is searching for any old stick, even a feeble twig, to beat Bill Clinton with.

Tim Lambert 14/9/2004 00:23:14  
All right then, this picture shows exactly what is wrong with Xrlq's comparison.

[eRiposte note: This last sarcastic comment limning Dick Cheney is perhaps the best part of this thread]  

Martin 14/9/2004 01:26:45 But what about those hundereds of thousands of uncounted people who make their living off Ebay?

Lambert posted an update here:

Now, here’s what Lott and Hassett say:

“In the case of unemployment, 44 percent of the headlines under the Clinton administration were positive while that same number was only 23 percent under Bush II. By comparison, the average unemployment rates were fairly similar, 5.2 percent under Clinton s eight years and 5.5 percent under Bush during the sample. There is also a great deal of overlap (3.9 to 7.1 percent under Clinton to 4.2 to 6.4 percent under Bush II).”

What they fail to mention and what is obvious from the graph is that under Clinton the unemployment rate decreased from 7.1% to 3.9%, while under Bush it increased from 4.2% to 6.4%. Maybe, just maybe, that’s why the headlines were more positive under Clinton. In fact, there seems to be evidence of bias against Clinton—why were only 44% of the headlines about unemployment positive when it just kept going down and down to the lowest levels in decades? Oh, and don’t expect to see a graph of the unemployment rate anywhere in their paper or presentation.

Now, they claim to have controlled for level and trends in unemployment in their analysis, but of course they have not. The only control they have for trend is the change since the previous quarter and it is obvious that changes over longer terms will affect the reporting. Do Lott and Hassett believe that no-one ever compares the unemployment rate with what it was a year or two before?

Another look at the graph of the unemployment rate will reveal the futility of the whole exercise. Is it not obvious that unemployment did something fundamentally different under Clinton than under either Bush? To get a meaningful comparison you would need to compare Clinton with a Republican administration where unemployment declined for eight years.

Lambert also links to this post at Lawyers, Guns and Money, that points out a few more obvious points:

Pay careful attention to how the [NYT] article is constructed. The goal of the piece is to demonstrate that economic reporters are all lefties who can't report economic matters straight (same goal as the crappy "research" it purports to report on), and to that end it contains howlers like this:

For instance, they said, the unemployment rate in the Clinton administration averaged 5.2 percent, only three-tenths of a percentage point less than it has under George W. Bush. But while 44 percent of Mr. Clinton's headlines on unemployment were positive, only 23 percent of President Bush's headlines on the subject have been upbeat.

Hmm. Why could that be. Must be bias. Or, it could be because the unemployment rates went down consistently during Clinton's tenure, which GWB has failed to do. There's a reason only one month with solid job growth data under Bush (March 2004) is cited--they're pretty rare--and in the context of Bush's own purported expectations of job growth, it falls pretty short. (By the way, what don't those liberal economic reporters harp on how far short of his own job growth goals team-Bush has fallen?)

Through a Google search I came across this post at Dead Parrots Society that explored the unemployment comparison further (not specifically in the context of the Lott/Hassett paper, but in the context of a similarly nonsensical "media bias" post by another blogger, using the employment figures):

Via Glenn Reynolds, I and many others have been reading this Tim Blair post about media framing of unemployment figures. The gist is that CNN described a 5.6% unemployment rate as "low" in 1996, when Clinton was in office, but describes a similar rate as a sign of problems for Bush. Tim's post is being widely cited as yet more proof of media bias; in Glenn's link, he encourages us to "Go figure." So I did.

[eRiposte note: chart similar to the one above is shown]

The graphic is courtesy of the BLS, and shows the unemployment rate charted over the past 15 years. Perhaps it offers a little insight into why 5.6% was considered "low" in early 1996, but not in 2001. Actually, the context was right there in the excerpts Blair chose from the 1996 CNN story:

Economists didn't expect June's unemployment rate to be much different from May's, which was an already-low 5.6 percent. But in fact, it did fall -- to 5.3 percent. The unemployment rate hasn't been that low since June 1990.

And from the 2001 CNN story:

The U.S. unemployment rate jumped to 5.7 percent in November - the highest in six years - as employers cut hundreds of thousands more jobs in response to the first recession in a decade in the world's largest economy.

Tim Blair was right: It is all relative. But not to who's in office, rather to where the unemployment rate has been in the past few years. So is it appropriate for journalists to speak in such relativistic terms? Are they at least consistent? What would be really helpful here is a Lexis-Nexis search of media reporting on the early part of Clinton's tenure, when unemployment was rising. Or on the situation in 1984, when unemployment was at 8%, down from 10%. But I don't have Lexis-Nexis access.

Still, there is some context that might be helpful. The first place we can look is right there in Tim Blair's 1996 story, to see how the Clinton administration and economic analysts felt about the numbers:

White House: But the Clinton administration was tickled about the increase in jobs, and took credit for the upturn. The president said the figures showed "the most solid American economy in a generation."
Analysts: In January, analysts were concerned that growth was so anemic that the nation was in danger of a recession. But five straight months of strong job gains now have analysts worried more about inflation. ... The Federal Reserve is almost guaranteed to push interest rates up to stave off inflation.

The second place we can look for context is in Tim Blair's 2001 story, to see how the Bush administration and economic analysts felt about that very similar unemployment figure:

White House: President Bush and his Labor Secretary, Elaine Chao, separately expressed alarm at the data and called for Congress to approve a package of economic stimulus. "Today's numbers are not good news, and I think it's a clear reflection that the attacks of Sept. 11 are still reverberating around our economy," Chao told CNNfn's Market Call program.
Analysts: To keep consumers spending despite mounting unemployment, the Federal Reserve has cut its target for short-term interest rates 10 times this year and is expected to do so again after its policy makers meet Tuesday. "Despite some better-than-expected data over the past two weeks, this report is sufficiently gloomy to force the Fed to ease next Tuesday and retain their bias toward further economic weakness," said Steven Wood, economist with FinancialOxygen.

Really, my point here doesn't have anything to do with whether a 5.6% unemployment rate is too hot, too cold or just right. Frankly, I don't have any idea. What I do know is that journalists weren't the only ones who looked at the unemployment figures in a different light between 1996 and 2001. The reality is, the media saw the data the same way as the White House, economic analysts and the Fed. (The question Tim didn't ask is why Bush considered it bad news when the unemployment rate hit 5.7% in 2001.) Perhaps everyone was too excited in 1996 and too glum in 2001, but the key word here is "everyone." This is not media bias, folks, and casting it as such is misleading at best.

...

Update II: Thanks to Tim Blair for linking to this post.

Let me note that the operating word is not [what] "everyone" [felt]. The operating word is whether the information portrayed in the article is "accurate". The whole world may think something is just dandy or something is just terrible, but that's different from saying that what the whole world thinks is "accurate".

One of the commenters to the post above also notes the following:

Barry Ritholtz writes ...

This is an interesting discussion -- but you kids need to learn to drill below the headline numbers to gather the most relevant data.

The headline is always "How many jobs per month is the economy creating?"

But that's a simplistic question which fails to address the underlying details of the ecoonmy's strengths and weaknesses. Forget framing and the media -- lets get to the actual DATA:

How do the jobs being created compare to the job lost, outsourced, or made obsolete in temrs of wages? Remember, 2/3rds of the US economy is based upon consumer spending, so wage growth/contraction has a major impact on the next part of the economic cycle.

How do the new jobs compare benefit wise with the lost positions? Again, like oil price increases, if employees are paying for halth costs enttirely, they have that much less money to discretionary income to spend on retail goods, autos, durable goods.

What is the spread between wage growth and CPI? At present, acording to Bear Stearns, CPI has risen 3% faster than Unit Labor Costs -- that makes those people with jobs "feel" like they are losing ground. Incidentally, Bear tracks this because this spread can be a significant danger to incumbents in re-election campaigns (see Carter, Bush I).

Next up: How many unemployed people have stopped looking for work? What is the actual percentage of unemployed -- as opposed to only the # wno are still eligible for Unemployment benefits. (Greenspan called it the "Augmented unemployment rate"). Its almost always much higher in Europe (12%+) than it is here -- about 8%.

Here's an odd data point: What is the average age of the labor pool? Are people working longer -- are the elderly coming back to work, as they did after the market crash?

Lastly, how many people have given up looking for work -- dropped out of the labor pool?

One of the great failings of economics is the way it allows school boys to grossly oversimply the complex, the intricate, the unseen, and believe they have "figured it out."
Its never that simple . . .

Posted by Barry Ritholtz at 08:21 PM May 27

The Times article also had some comments from others:

Alan S. Blinder, a former vice chairman of the Federal Reserve who also served as an economic adviser to Mr. Clinton, said that, if anything, current economic coverage favored Mr. Bush by letting the administration get away with blaming 9/11 for the economy's poor performance.

Jack Shafer, the media critic of Slate, the Web journal, was skeptical of the study, saying that it was based solely on headlines, not on an appraisal of actual news articles. "A headline is not coverage," he said.

While the researchers of the American Enterprise Institute claim to expose the political bias of the reporting, Mr. Carroll said, it was unlikely that they succeeded in stripping out other factors. He said the reporting of economic statistics depended on broad perceptions of the state of the economy, which are influenced by many variables. The fact that the economy did better under Mr. Clinton than either of the Bushes probably affected the coverage more than the researchers allowed for.

Moreover, Mr. Carroll pointed out that the results had large statistical margins of error. "I'm not persuaded that the results have any statistical significance," he said.

AND what of the researchers' own objectivity? Critics question both their scholarship and their motivations in releasing this research in the middle of a presidential campaign in which the economy is no small issue. Mr. Hassett was an adviser to Senator John McCain, Republican of Arizona, during his bid for the presidency in 2000, and a co-author of "Dow 36,000," a wildly bullish analysis of the stock market's prospects.

Mr. Lott's research supporting gun ownership as a crime deterrent has also come under criticism. He acknowledged that he assumed a pseudonym - Mary Rosh- to write his own praise and defend his positions in online debate on that subject from 2000 through January 2003.

Enough said. Let's move on from this junk paper.

2.10B PAPER: "Being the New York Times: The Political Behaviour of a Newspaper" by Riccardo Puglisi (2004)

Here is a copy of this paper written by Riccardo Puglisi at the London School of Economics (LSE), who incidentally acknowledges he has "greatly benefited from conversations" with a list of people that includes, you guessed it, John Lott. 

I decided to comment on it because I saw this paper being mentioned on the blog (sans analysis) of conservative economists. A cursory analysis of this paper shows six major problems (I, II, III, IV, V, VI) that, unfortunately, make its conclusions baseless and untenable.

Like other similar papers, the author believes his "results" are credible even though no serious attempt is made to confirm that the paper's fundamental assumptions (about the premise and model) are actually correct. While I certainly have no complaints against someone doing a lot of hard work to study the issue of media bias (which the author has clearly done), it is disappointing that he seems to have forgotten a basic fact about modeling (econometric modeling or otherwise): just because one has a model and churns out some numbers it doesn't mean that the model is correct or that the results are meaningful. (There's another, less politically correct, term to describe this phenomenon: GIGO - Garbage In, Garbage Out). 

To understand why this author let himself fall into the GIGO bin, let's start by reviewing a couple of extracts from the paper to get some quick background. 

The abstract (bold text is my emphasis):

I analyze a dataset of news from the New York Times, from 1946 to 1994. Controlling for the incumbent President’s activity across issues, I find that during the presidential campaign the New York Times gives more emphasis to topics that are owned by the Democratic party (civil rights, health care, labour and social welfare), when the incumbent President is a Republican. This is consistent with the hypothesis that the New York Times has a Democratic partisanship, with some "watchdog" aspects, in that it gives more emphasis to issues over which the (Republican) incumbent is weak. Moreover, out of the presidential campaign, there are more stories about Democratic  topics when the incumbent President is a Democrat.

A partial extract on the methodology used (bold text is my emphasis):

The empirical approach adopted by Lott and Hassett is very similar to the one I follow here [eRiposte note: uh-oh], because of this common focus on the time series behaviour of news providers. However, their analysis is chiefly based on the correlation between the political affiliation of the incumbent President and the average tone adopted by newspapers in the coverage of economic news [eRiposte emphasis; note how a paper that looked at "headlines" is being quoted as one that addressed news coverage]. On the other hand, presidential elections and campaigns play a very minor role in their econometric analysis, while they represent the central aspect of my identification strategy¹².

Let's now examine why, neither the assumptions, nor the conclusions of this paper, are correct or meaningful. 

MAJOR PROBLEM I: The first fatal problem with this paper is that one of its basic assumptions (#2 below) is wrong
[The other assumptions are questionable too, but let's not worry about that right now.]

Here are the assumptions stated by Puglisi:

As briefly anticipated in the introduction, the empirical analysis performed here and the interpretation of its findings are based on the following set of identifying assumptions:

(1) The issue ownership hypothesis holds.

(2) “All publicity is good publicity”.

(3) The relative share of Executive Orders about a subset of issues proxies the relative intensity of the activity of the incumbent President with respect to those issues.

The issue ownership hypothesis, which Puglisi bases on historical polling data and mentions throughout, is the following: 

Democratic topics comprise Civil Rights, Health Care, Labor & Employment and Social Welfare. Republican topics comprise Defense and Law & Crime.

Now, it may be convenient to assign such ownership because it helps make the analysis more interesting, but really, someone "owning" the issue often has little to do with whether the publicity/coverage that person gets on that issue is good or bad (even if one can be sure that the "issue ownership" actually holds). Thus, the second assumption, that "All publicity is good publicity" (referring to "owned issue" coverage for the person who owns it) simply makes no sense. For example, was "Health Care" coverage always "good publicity" for Bill Clinton (Democrat)? Was "Defense" and "Law and Crime" coverage always "good publicity" for Richard Nixon (Republican) and Ronald Reagan (Republican)? Was "Employment" and "Social Security" coverage necessarily always "bad" publicity for the Reagan administration? 

In other words, the assumption that if a newspaper reports on topics "owned" by a party, it automatically means that party benefits, makes no sense because such an assumption fails to account for the fact that newspapers, can and do issue reports on "owned" topics that may not be positive at all to the "owning" party.

(P.S. Consider another example. Clinton's term actually saw a drop in crime rates. Granted, a big part of it was outside this particular study's time period, but, if anything, stories on crime rates would have tended to benefit the Democrats during that time, not Republicans.)  

MAJOR PROBLEM II: The second, decidedly fatal problem with this paper is that the author's simplistic interpretation of his results is wrong

Let's consider these "definitions" from Puglisi:

Definition 1 A newspaper has a Democratic (Republican) partisanship if during the presidential campaign it devotes more space to issues owned by the Democratic (Republican) party, at the expense of neutral or Republican (Democratic) issues.

...

In fact, over and above the electoral partisanship of the newspaper, as described by definition 1, the political colour of the incumbent President could be given an interpretation within a lapdog/watchdog dichotomy. The idea is the following: if it turns out that -during the presidential campaign- the New York Times gives less emphasis to Democratic topics and/or more emphasis to Republican topics when the incumbent is a Democrat, over and above his Democratic or Republican or lack of partisanship, this is consistent with the fact that the newsaper [sic] acts as an electoral watchdog with respect to the incumbent President.
...

Definition 2 The newspaper is an electoral lapdog of the incumbent President if ceteris paribus during the presidential campaign it devotes more space to the issues over which the incumbent is strong, and/or less to issues over which the incumbent is weak.

Definition 3 The newspaper acts as an electoral watchdog if ceteris paribus during the presidential campaign it dedicates more space to the issues over which the incumbent is weak, and/or less space to the issues over which the incumbent President is strong.

Where do I begin?

I don't mean this insultingly at all, but these definitions are ridiculously wrong because not only are they inconsistent with each other, the latter definitions are wrong on their face. For example, Puglisi's definition of "watchdog" in Definition 3 is highly convenient (and reveals possible built-in bias in his thinking about this problem) because I can just as well argue based on his Definition 1 that the newspaper is no "watchdog" but just a shill for the candidate opposing the incumbent and is therefore displaying "partisanship" in favor of the challenger. In fact, let's ignore Definition 1 completely and consider Definition 3 on its own. It is Puglisi's *opinion* that the newspaper serves as a "watchdog" by focusing on the topics that supposedly favor the challenger. One can easily have a different *opinion* that a newspaper doing this is a partisan supporter of the challenger and not a "watchdog"! (Talk about pre-ordaining the results!)

This is the natural (and fully expected) problem with studies of this nature which don't bother to actually analyze the content of the news articles. Thus, Puglisi's assumptions and definitions are wrong because at a very fundamental level, they neglect the actual nature of the coverage (accurate or inaccurate). The "study" therefore fails catastrophically to even set up the basic question and data interpretation correctly. Incidentally, this is a standard problem with most of these supposed "econometric" media bias "studies" (especially those that, conveniently, keep finding "liberal bias"). Whether they look at "words" used in news reports (e.g., "right-wing" v. "left-wing") or "topics" covered or "tone" ("positive" v. "negative") or even "headlines" (to some extent), none of them really focuses on the facts (or fictions) covered in the actual news reports and how that impacted the person or party being covered.

So, combining Problem I and Problem II, this study and the interpretation of its results totally break down even before we get to the actual data. Needless to say, this study's findings are worthless as a result. 

[AN ASIDE: To see how ludicrous the "partisanship" assumption is, one only needs to look at George Bush Jr. and social security. Although Bush II was not part of the study, let's remember that there has been significant coverage of Social Security in the New York Times in the past three months because the Republican President is pushing it NOT the Democrats. If we took Puglisi's methodology seriously this might imply "Democratic partisanship" by the Times.]

Sadly, the problems with this paper don't stop there.

MAJOR PROBLEM III: What about topics that are not assigned an "owner"?

By Puglisi's own admission (Table 3, page 49), when we look at "All stories" that appeared in the New York Times in the period 1946-1994, the so-called Republican topics and so-called Democratic topics were only 21.7% (8.37% + 13.36%) of the total. Thus, this study claims to show "Democratic partisanship" (or otherwise) based on a study that essentially ignores over 78% of all stories published in the New York Times. Stunning.

For example, "Banking, Finance and Dom. Commerce" (14.66% of all stories) and "International Affairs" (13.22% of all stories) are not part of Puglisi's model because they are not "owned" by Republicans or Democrats. What category would "taxes" or "spending" or "budget deficits" fall under? This is one of the most important topics in all Presidential campaigns - which often make or break campaigns - and there's no mention of it in the analysis. Also, what category would draft-avoidance or alleged extra-marital affairs fall in? Other? Or is it "Law and Crime?" There's a whole slew of topics relating to the individuals or their policies, that fall into the supposed "non-owned" issue category, which have a habit of coming up frequently during campaigns. It is extraordinarily odd that one would simply ignore all that and still feel comfortable reaching sweeping conclusions of the kind the author has.

MAJOR PROBLEM IV: Topics may be entirely event driven and have nothing to do with Executive Orders

Puglisi's paper does not consider seriously the fact that major events happen which have nothing to do with the "strength" of Democrats or Republicans. For example, George Bush Sr. started significant cuts to defense spending at the end of the Cold War and Bill Clinton continued this effort. When there are no major wars and when there is no overarching concern about national defense, there is no reason for papers to simply keep writing more articles about "defense" just because a Democrat is in power. This same argument applies to every topic under the sun.

It is also obvious that many topics are raised, especially in electoral campaigns, by the politicians who are campaigning. The New York Times, or any other paper, may simply be reporting the issues raised by the candidates themselves. Candidates who run for President have to talk about most topics (not just "Republican topics" or "Democratic topics"). It would have made no sense for George Bush Sr. to completely avoid talking about jobs, since jobs was one of the most important issues of his re-election campaign. Would it be "Democratic partisanship" for the NYT to have written reports about jobs then, even though it was one of the most important concerns for Americans? 

Not to mention, one of Puglisi's "findings" is that the coverage of "Republican topics" actually goes up significantly in the campaign coverage when the challenger is a Republican. As he says (bold text is my emphasis):

When considering the 1961-1994 subperiod such effect of less Republican stories under a Democratic incumbent is larger in magnitude and more precisely estimated. 

Moreover, considering the same subperiod, under a Democratic incumbent there are more stories about Republican topics when the presidential campaign kicks in. This effect is quite strong in magnitude and very precisely estimated when excluding the 1964 presidential campaign, namely the only year during which the defense issue (with reference to the handling of the Vietnam War by Lyndon Johnson) was clearly owned by the Democratic party. 

This takes us right back to Problem II. Either the NYT has "Democratic partisanship" or it doesn't. It makes no sense to claim that it has "Democratic partisanship" and simultaneously say that "...under a Democratic incumbent there are more stories about Republican topics when the presidential campaign kicks in. This effect is quite strong in magnitude...". Why is the latter considered a "watchdog" behavior rather than "Republican partisanship"? After all, if part of the "results" point one way, it is sufficient for Puglisi to label it "partisanship" of one kind; yet, when another part of the "results" points in another direction, it is not partisanship in the other direction - it is "watchdog"ism! If that isn't bias convenient, I don't know what is.

To make things even more confusing, let me also note this discussion in the paper, which appears earlier in the paper (p 23):

Suppose that, after controlling for the presidential activity across issues, out of the presidential campaign there are systematically more stories about Republican topics when the incumbent President is a Republican. There are two plausible explanations for this kind of behaviour. 

First of all, this bias could be due to the fact that the newspaper is acting as a pressure group with respect to these Republican issues, and is taking into account the fact that a Republican incumbent could be more responsive to pressures that are related to owned issues. In other words, the idea is that the newspaper, over and above the incumbent’s behaviour, is devoting more room to stories about Republican issues when the incumbent President is a Republican, exactly because this incumbent could be more responsive to implicit and explicit appeals that deal with her “natural” issues, i.e. with the issues that are owned by her political party. 

Alternatively, this bias introduced by the newspaper could be demand-led. Suppose that the issue ownership hypothesis holds, and that citizens have elected a Republican President, just because they think that the most relevant problems would arise in the policy fields that are owned by the Republican Party. Therefore, within a political agency framework, they want to obtain pieces of information about what the Republican incumbent is delivering with respect to those problems. The newspaper responds to this demand for specific information by publishing more stories concerning Republican issues.²³

Whoa! So there are other possible interpretations of topic coverage than partisanship? Who would have known!

The bottom line is that this is a sad paper that seems to have been put together without much critical thought given to the basic premise that is being explored. It is absolutely stunning to think that someone took all the trouble to do a lot of hard work and write up a 60-page paper without ensuring that the foundation and assumptions behind the paper are correct.

MAJOR PROBLEM V: Stories on "Republican topics" vs. "Democratic topics" in the NYT

As much as I was loathe to spend more time on this paper given that the earlier problems already showed that its results were meaningless, I thought I should spend a little bit of time scanning through at least a sample of the data. What caught my eye before I started reviewing Puglisi's detailed analysis was this sentence on pages 20 and 21:

Coming back to the comparison between Democratic and Republican issues, the last two rows of Table 4 indeed show that the relative advantage of stories about Republican topics over Democratic ones, being less than 5 percent on internal pages, jumps to more than 11 percent on the front page. [eRiposte emphasis]

Here he is talking about the "All stories" coverage in the New York Times. What is stunning here is Puglisi's definition of "5 percent" and "11 percent". Let's look at the numbers and you'll see what I mean. 

I have reproduced in the first three rows of the table below, a part of the data from Table 4 of the paper. In the bottom two rows I have populated some results extracted from the data in the two preceding rows, with the items in red being the numbers cited by Puglisi above:

Let me draw your attention to the bottom-most row. When Puglisi claims:

...relative advantage of stories about Republican topics over Democratic ones, being less than 5 percent on internal pages...

what he means is that the difference in absolute percentages (relative to a scale of 100) is less than 5%. This DOES NOT mean that the so-called Republican topics are found only <5% more than the so-called Democratic topics, relative to one another. As you can see in the bottom-most row, the ratio of REP topics covered to DEM topics covered (in internal pages) is 1.55, i.e., there were 55% more REP topics than DEM topics in the inside pages of the Times, per Puglisi's own data! For front-page coverage, the ratio is 2.2, which means there were 120% more REP topics than Dem topics on the New York Times' front page!  

Let me reiterate the point of the comparisons above. Puglisi's wording is very misleading, and gives the impression that the Times barely covers the so-called REP topics over the so-called DEM topics, when the reality is drastically more in favor of REP topics than DEM topics.

Clearly, that doesn't sound like a "Democratic partisan" paper by Puglisi's own definition, which is probably why he adds a "control" using Executive Orders (as a proxy for Presidential activity), making the assumption that because there were more Executive Orders on REP topics than DEM ones, the coverage in the Times must be normalized to reflect this (this assumption is questionable in my view, but I'm going to ignore this for now). 

So, let's look at the numbers vis-a-vis the "Executive Orders".  To do this, I've combined the relevant data from Puglisi's Tables 2 and 3 into a single table below:

Once again, the last row is the most important. What is it saying? It's saying that:

• There were 46% more Executive Orders on so-called REP topics than on so-called DEM topics across all Presidencies from 1946-1994, and, during the same time period, 

• There were 60% more stories on REP topics in the NY Times than on DEM topics

Thus, at this broad level, even if one makes the assumption that Executive Orders get proportional coverage in the NY Times, the data above suggests that even when the New York Times' topics-coverage is normalized to Executive Orders, it provided more coverage overall on the REP topics than on the DEM topics. 

Now, I could not find any data in Puglisi's paper which separates out "All stories" in the NY Times based on the Presidential administration. I don't know why that is not provided, just like the Executive Orders are partitioned, but that data is important. Why? Because the REP/DEM Executive Order ratio is lower (1.26) for Democratic administrations than the average (1.46), which in turn is lower than the ratio for Republican administrations (1.80). So, without a similar breakup of news stories between the two types of administrations, it is hard to make a conclusion about whether REP topics are always over-represented in the NY Times or only in one of the two types of administrations.  

Regardless, the fact that normalizing the data to Executive Orders doesn't reverse the extra coverage given by the Times to the so-called REP topics, argues against Puglisi's main conclusion. 
(I invite readers who are more statistics-aware to comment on whether I made any mistakes in my assumptions/calculations because I am not a statistics expert).

MAJOR PROBLEM VI: No real "control" for the study

Puglisi discusses a lot of "controls" on the data, based on accounting for the ideology of the New York City Mayor, New York State Governor, NYT Publisher etc. But the real "control" needed for the study is glaringly missing.

His comments in his conclusions hint at this, but it is not clear to me that he appreciates the seriousness of this. He says:

At this point, one can formulate a conjecture, according to which the behaviour of the New York Times within the issue space is in fact a typical strategy, that is followed by other news providers. The idea is to check whether other news providers, whose explicit political orientation can be for example disentangled from their endorsement choices, during the presidential campaign systematically publish more stories about issues that are owned by their preferred candidate. In particular, a natural question to ask is whether conservative newspapers are indeed characterised by an issue pattern in the time series which is specular to the one displayed by the New York Times. [END]

I assume Puglisi is thinking out aloud about whether conservative papers show the reverse of the results he derived for the NYT in his study (note the definition of "specular": "Of, resembling, or produced by a mirror"). This brief mention is disappointing because this is another standard problem with a lot of these "media bias" "studies" - they don't seem to want to check out such an obvious thing before they make claims of "liberal media" bias or "Democratic partisanship". What I mean is this: even if we assume that the results of this study are correct (which they are not), how can someone claim that a paper is partisan without even evaluating another paper - with an ideology known to be conservative - to see whether that paper's topic coverage was similar or the opposite?

To understand the importance of this, let's review a couple of examples from Bob Somerby at the Daily Howler:

But with Sully, the lies are just foreplay. Yesterday, he quickly "defer[red] to a young and fearless blogger," Patrick Ruffini, who had done "a quick statistical analysis of the use of the term ‘right-wing’ in a couple of major papers." Trembling over his acolyte’s brilliance, Sullivan quoted at length:

RUFFINI, AS QUOTED BY SULLIVAN: Since 1996, the Washington Post has used this loaded term ["right-wing"] more than twice as frequently as "left-wing"…This disparity was even more palpable at the New York Times, where 80.2% of the left-right mentions on the national news pages since 1996 have spotlighted the right. The research also found that the more loaded and derogatory the phrase, the more likely it was to be associated with the political right. The term "conservative" outpolled "liberal" by 66-34% in New York Times news page mentions, while the aforementioned "right-wing" clocked in at 80% in a similar measure. However, the term "right-wing extremist" was used at least six times as frequently than "left-wing extremist" (at 87.4% since ’96 in the Times). [emphasis added]

If that didn’t prove it, nothing would. At the New York Times, "right-wing extremist" was used much more often than "left-wing extremist." Case closed.

But duh. Does unequal usage of those terms show a liberal bias? We were dubious, so we did a test—we checked out the use of these terms at the Washington Times. How many times did the Wes Pruden rag use those terms in the last five years? Our finding? The Washington Times reeks of liberal bias! In fact, its liberal bias is even worse than that found in the Times of New York!

That’s right, folks. Over the past five years, NEXIS says that "left-wing extremist" has appeared in the Washington Times all of eight times total. But the term "right-wing extremist" has appeared there 72 times, exactly nine times as often. Surely this fact doesn’t mean that the paper is full of liberal bias. But that’s the conclusion that Sullivan’s method would force us to reach. There’s a word for such a dude. It’s "poseur."

Alas! Ruffini simply counted the use of certain expressions, then leaped to conclusions about liberal bias. There are so many problems with this technique that it would take a whole book to explain them. But no matter. The Brainy Brit quickly bought his method, and soon was broadcasting drek to the planet.

That’s right, gang. If you buy the Brainy Brit’s latest researched technique, the Washington Times is swimming in liberal bias. Count Ruffini will live to research again. But where in the world—where on earth—did we find his hapless promoter?

Next: Motive mavens

The Times, D.C. and Gotham: Ruffini charted the usage of his selected expressions "since 1996." According to NEXIS, if you start your search at 1/1/96, here’s how the Times Two stack up:

The Washington Times:
Right-wing extremist: 86 uses
Left-wing extremist: 9 uses

The New York Times:
Right-wing extremist: 75 uses
Left-wing extremist: 9 uses

According to Sullivan’s brilliant technique, the WashTimes has slightly more liberal bias. Question: Where in the world—where on earth—did we ever come up with this dud? 

Another example from Somerby:

In Chapter 9, Coulter complains about the press corps’ use of the terms “Christian conservative” and “religious right.” According to Coulter, “[t]he point of the phrase ‘religious right’ or ‘Christian conservative’ is not to define but to belittle.” And lefties, of course, get a pass:

COULTER (page 166): Despite the constant threat of the “religious right” in America, there is evidently no such thing as the “atheist left.” In a typical year, the New York Times refers to either “Christian conservatives” or the “religious right” almost two hundred times. But in a Lexis/Nexis search of the entire New York Times archives, the phrases “atheist liberals” or “the atheist left” do not appear once. Only deviations from the left-wing norm merit labels.

In a footnote, Coulter extends her complaint. “In a one year period (roughly corresponding to calendar year 2000), the New York Times found occasion to mention either ‘Christian conservatives’ or the ‘religious right’ 187 times. Not once did the paper refer to ‘atheist liberals’ or ‘the atheist left.’” To Coulter, of course, this is all a sign of gruesome bias. She goes on to claim that the terms “religious right” and “Christian conservative” are now used “[j]ust as some people once spat out the term ‘Jew’ as an insult.”

It certainly makes for high excitement, but does it make any sense? Do newspapers use “Christian conservative” as an emblem of hatred, and avoid “atheist left” due to liberal bias? If so, we have big news to share. If Coulter’s NEXIS search has proven these things, then the once-conservative Washington Times is spilling with lib bias, too.

In the calendar year 2000, how often did the New York Times refer to “Christian conservatives” or the “religious right?” A NEXIS search of that year presents 182 references. But the Washington Times—a much slimmer paper—had 151 such cites that same year. And how about those other terms—“atheist liberals” or “the atheist left?” Incredibly, Coulter was right in one of her claims; the New York Times never used either term. But guess what? The Washington Times never used the terms, either. If Coulter has sniffed out a vast left-wing plot, Wes Pruden is in on it too.

Why do newspapers write about “Christian conservatives?” Because they exist, and because they’re important. And why don’t we read about the “atheist left?” Because the group doesn’t exist. That’s why the New York Times doesn’t mention the group; that’s why the Washington Times doesn’t mention it, either. Everyone in America knows this is true—until they read Coulter’s cracked book.

We have no idea whether Puglisi's findings will be "mirrored" or "similar" in a rag like the Washington Times. But it was inappropriate to make the kind of sweeping conclusions he makes in his paper without doing such a basic comparison in the first place.

At this point I really have to stop because I've spent far more time on this paper than I had originally intended to. But I've shown clearly that this paper is fatally and deeply flawed and proves nothing.