Nate Silver falls off

In 2012, Nate Silver faced a conservative and media-led backlash for bringing rigor to election forecasting. His newly launched journalism project is now facing a backlash for failing to live up to its promise.

I am probably the ideal audience for Nate Silver’s new journalism project FiveThirtyEight.com. I am someone who values data in a frequently-substance-free world of reporting. Although I am an unabashed Sam Wang PEC partisan, I certainly appreciated Silver mainstreaming election forecasting based on factors other than wishful thinking and journalism biases towards “the horse race” and “momentum”. When Silver was attacked by know-nothings in the media and the conservative blogosphere, I cheered him on, and savored his election day vindication, anti-climatic as it was.

Rather than topping my “must read” list, however, the new FiveThirtyEight is something I won’t be reading. Here’s why:

I became aware of Silver’s imminent launch by his public Twitter announcement of two hires to cover science for his new venture: Emily Oster, a University of Chicago economist famous for counter-intuitive revelations (sound familiar?); and Roger Pielke Jr.

Image courtesy of Flickr user “ferdicam”, used under Creative Commons.

Now, I am not going to get into Roger’s pathological attacks on climate scientists. I am not going to get into his sweaty delusions of persecution. I am not going to get into Roger’s complete misunderstanding of elementary aspects of climate science. I am going to focus on just two things: what Nate Silver is known for, and what Roger Pielke Jr. is known for.

Nate Silver’s reputation is based on being a stats wiz. This is what his blogging was devoted to, what his best-selling book is about, and the one thing he has that his competition/peers like Ezra Klein or Matt Yglesias don’t. And one of Nate Silver’s very first, very public hires (Roger Pielke Jr.) sucks at statistics. Not “published something in need of minor correction once or twice” sucks. “Doesn’t understand how a t-test works” sucks. “Doesn’t understand basic probability” sucks. Sucks out loud. Sucks on ice.

Roger’s very first article for Silver’s new site is, unsurprisingly, about Roger’s hobbyhorse. The claim that disaster losses are not increasing due to climate change.

Let’s be clear about some things. Climate change is real. Humans are not just “contributing” to it, we are responsible for essentially all of it over the past several decades. Our perturbation of the climate through our emissions of greenhouse gases is fundamentally changing the Earth system. The biosphere and human systems are going to have to adapt to a rate of change as of now unseen anywhere else in the paleoclimatic record. In the absence of emissions stabilization, a difficult but decidedly achievable outcome, the threat to the biosphere and society is daunting. The amount of climate change we’ve already experienced, while extremely serious, is tiny compared to the impacts we will see in world of unchecked fossil fuel exploitation. In addition to changes in the average or mean state of the system, we have already begun to see changes in some types of extreme weather events, and changes to the drivers of yet other extreme events.

Ostensibly, Roger Pielke Jr. accepts all of the above. He just doesn’t want you to focus on this big picture. Instead Pielke wants you to believe and to focus on the claim that we’ve seen no increase in “normalized” damages due to climate change. The fundamental conceit of this claim is that even though disaster losses are unquestionably on the rise, once you account for changes in the value of infrastructure being built in areas affected by disaster (due to population growth, inflation, etc.), there is no “statistically significant increase”.

This claim rests on our ability to account for factors which might spuriously inflate the damages caused by disasters, but also our complete failure to account for factors that have allowed us to avoid even greater losses.

The case of 2012′s Superstorm Sandy is illustrative. While Roger spent the first few days of the disaster trying to play down the magnitude of the mounting carnage, Sandy ultimately ranked among the most costly storms on record, even using normalized losses. Preliminary estimates range from $50-65 billion USD.

And yet it could have been so much worse.

Hurricane Sandy uncharacteristically failed to recurve out to sea, and barreled back towards the East Coast of the US. Due to the amazing advances we’ve seen in our ability to model the global weather system, we knew well in advance that this unexpected turn by Sandy was a real possibility.

This possibility was taken into consideration by those trying to game out the impact of Sandy’s landfall. The impact of rising sea levels on the frequency and severity of storm surge flooding was as well. Looking to a future of warming-boosted surges, researchers identified huge vulnerabilities in the New York transportation infrastructure to previously rare events. Such considerations ultimately led the MTA to shutdown the subway system in order to avoid the corrosive impact of salt water if the tunnels were flooded. This decision, informed by modeling and meteorological sophistication unimaginable in the early 1900s, saved the subway system and prevented New York City from being paralyzed for weeks and nearly doubling the economic damages.

Image via Twitter

Roger Pielke Jr.’s “normalized” disaster loss fixation takes none of this into account. Nor does it account for the benefits of building code improvements. Or other disaster prevention measures like dikes.

Paul Krugman is among a growing list of knowledgeable folks who were hopeful about Silver’s new enterprise but are less than impressed. Krugman writes:

… data are never a substitute for hard thinking. If you think the data are speaking for themselves, what you’re really doing is implicit theorizing, which is a really bad idea (because you can’t test your assumptions if you don’t even know what you’re assuming.)

I feel bad about picking on a young staffer [Note: not Pielke Jr.], but I think this piece on corporate cash hoards — which is the site’s inaugural economic analysis — is a good example. The post tells us that the much-cited $2 trillion corporate cash hoard has been revised down by half a trillion dollars…

… what does this downward revision tell us? We’re told that the “whole narrative” is gone; which narrative? Is the notion that profits are high, but investment remains low, no longer borne out by the data? (I’m pretty sure it’s still true.) What is the model that has been refuted?

“Neener neener, people have been citing a number that was wrong” is just not helpful. Tell me something meaningful! Tell me why the data matter!

Though Krugman is referring to a different 538 article, he could easily be making the same criticism of Pielke’s. Why do Pielke’s data matter? Are disaster losses not increasing? They are. Does “normalizing” the loss data tell the whole, unbiased, story? No, it doesn’t. Are extreme events, and drivers of yet more extreme events, changing in response to GHG emissions? They are.

If Nate Silver’s mission is to bring statistical cachet to good journalism, he’s off to a terrible start. One of his first big hires is terrible at statistics. If Silver wants to tell us something meaningful instead of peddling freakonomics-lite contrarianism, he’s similarly off to a poor start. Pielke’s personal hobbyhorse obscures far more than it enlightens. It offers a cocktail party morsel of contra-conventional wisdom instead of intellectual nourishment.

There are probably a lot of people who would like to see Silver fail. I’m not one of them. I just won’t be one of his readers, either, unless he makes some big changes to his current model.

28 responses to “Nate Silver falls off

  1. Interesting that Pielke wrote about his hobby in the first post. Suggests that he does not expect to last long.

    This quote is so important: “data are never a substitute for hard thinking. If you think the data are speaking for themselves, what you’re really doing is implicit theorizing, which is a really bad idea (because you can’t test your assumptions if you don’t even know what you’re assuming.)”

    That is one of the main problems of WUWT “science”. That is why climate ostriches typically link directly to a picture. They seem to think that the data speaks for itself and that you do not need to know how it was computed.

  2. Pingback: 538 Broken Things – A Few Things Ill Considered

  3. Equating dollar value of storm damage to whether climate disasters has increased or not is just nonsense. I can give a very good example of why it is nonsense.

    Last June the city and surrounding area where I live was subjected to a massive flood because of a prolonged torrential downpour. The flood to the Bow and Elbow Rivers was probably the worst in recorded history and caused an estimated 6 billion dollars in damages. That weekend I was 100 miles SE of the city and we got no rain at all. If that storm had been centered 100 miles SE the cost would have been orders of magnitude less and according to Pielke that storm would not have been considered exceptional. It just shows how useless his methods are if just moving a storm by 100 miles would have such a drastic reduction in dollars of damage.

    Surely it is the number of climate disasters and their severity that is important for determining trends not “normalized” dollar damage.

    Here is a graph showing numbers of climate disasters over the past few years. This clearly shows an increase.
    http://www.preventionweb.net/files/20120613_ClimateDisaster1980-2011.pdf

  4. Pingback: 538 Broken Things [A Few Things Ill Considered] | Gaia Gazette

  5. This isn’t a desperately convincing rebuttal or RP’s article. Casting aside all the intro stuff, which is irrelevant, we’re left with you not liking his normalisation. But your arguments are rather handy-wavy; there’s nothing concrete, no numbers.

    At http://thinkprogress.org/climate/2014/03/19/3416369/538-climate-article/, where Mann is quoted making similar claims, RP replies in an update:

    Pielke particularly took issue with Mann’s claim that Pielke “completely ignores technological innovations (sturdier buildings, hurricane-resistant structures, better weather forecasting, etc.)” when analyzing disaster damage and its cause. Pielke said he has considered mitigation data in previous work, but has found through four of his own research papers not mentioned in the FiveThirtyEight article that there were no strong effects on the data.

    There are links to his papers there.

    So your (bold) claim “but also our complete failure to account for factors that have allowed us to avoid even greater losses” would appear to be wrong to me.

    • I am not “rebutting” Pielke’s argument. If you read what I actually wrote, I am making two points: 1) Silver is, erm, tarnishing his brand of stats wizardry by hiring someone who has picked very public fights based on his gross misunderstanding of basic statistics; and 2) Pielke’s whole shtick is typical counter conventional wisdom but not very meaningful crap that does not ultimately tell us anything substantive about the big picture.

      I assume that you’re not going to argue point 1, and you seem to be somewhat misunderstanding what I mean by point 2. I am stipulating his claim and saying it ultimately tells us little, I am not “rebutting” it.

      Pielke said he has considered mitigation data in previous work, but has found through four of his own research papers… that there were no strong effects on the data.

      I have no doubt that Pielke said this. Pielke, as with his father, is fond of claiming things based on an elaborate chain of self-citation (Pielkes all the way down) that he knows people are probably not going to bother digging all the way back through. I will say that my statement is based on many assessments by groups other than Roger, who are quite explicit that normalization does not account for such improvements. In fact, by your wording, you seem to be saying that Pielke is actually ceding the criticism that there is no effort to take such factors into account in his current analysis (justified by finding no impact in other analyses).

      The first of the four papers that purport to show no impact from building codes that I clicked on actually says this:

      Another important factor is mitigation and the implementation of stronger building codes. There is considerable evidence that strong building codes can significantly reduce losses; for example, data presented to the Florida Legislature during a debate over building codes in 2001 indicated that strong codes could reduce losses by over 40% IntraRisk 2002. As strong codes have only been implemented in recent years (and in some cases vary significantly on a county-by-county basis), their effect on overall losses is unlikely to be large, but in future years efforts to improve building practices and encourage retrofit of existing structures could have a large impact on losses.

      That’s it. No quantification. No analysis of any kind. A conjecture, made with no supporting references. In fact, in the same paper, it is explicitly warned that mitigation efforts are not included in the analysis.

      Note on Demand Surge and Loss Mitigation

      The normalization methodologies do not explicitly reflect two important factors driving losses: demand surge and loss mitigation. Adjustments for these factors are beyond the scope of this paper, but it is important for those using this study to consider their potential effect.

      Typical Roger.

  6. I don’t know why anyone bothers to read Pielke, as it’s all an irrelevance (I guess there’s some academic interest in recoding these things, but it affects nothing for the future). Take the UK, once London gets swamped, there’s going to be a big spike in storm damage, normalised or not, but, I think Miami will go under first, and larger parts of NY at some point, so we’ll start seeing some large spikes over the next 50-100 years.

    The only thing of relevance I can see is, if there is a trend in normalised storm damage protection measures? For example, in the UK one of this last winter’s storms produced a surge larger than the one in 1953. This one did far less damage and killed far fewer people. That was all down to improved protection. Those protective measures are already nearly out dated, and will need to be improved. So any protection they’ve provided between now an 1953 is already irrelevant.

  7. > I am stipulating his claim and saying it ultimately tells us little, I am not “rebutting” it.

    Then you’re being misinterpreted. For example, in comments at mine, Eli says:

    Nah. The problem with RPJ’s take is that he discounts for everything that reduces damage intensity such as GDP and population and just forgets to mention things like improved building codes, improved construction and better warning and track forecasting. Things had a pretty good take on this

    and he’s linking back here.

  8. I have discussed with Roger the issue of advances in building codes and building technology, and their impact on his “normalization” figures. My take is that he has examined those issues, but not in a comprehensive manner. So the claims on the one side – that he has ignored those issues – seem overstated – but on other hand it seems to me odd that anyone would take Roger’s claims about controlling for those factors for granted. Things’ point w/r/t a need to address events such as closing NYC’s subways in anticipation of Sandy, when “normalizing” damages, would surely seem consistent with the minimal requirements of appropriate skeptical scrutiny.

    But anyway, it seems to me that Ian (above) is focusing on what’s the most important question w/r/t Roger’s advocacy:

    >”Surely it is the number of climate disasters and their severity that is important for determining trends not “normalized” dollar damage.”

    Normalizing on the basis of GDP seems to me to be an odd way to measure what’s important: Whether climate change is causing more misery. Suppose GDP increases at fast clip, and concurrently, climate change increases damages from extreme whether at a directly proportional rate. What would that tell us? Would it tell us that increased amounts of extreme weather have not caused more misery? No.

    Given that Roger seems to accept the assertion that climate change will, eventually, cause more extreme weather (he’s a bit cagey on that when asked directly, why does he repeatedly focus on the rate of increase in damages as a ratio of growth in GDP?

    WAC? Perhaps you could explain why that is a meaningful point of focus?

  9. Sorry – that should be WC, not sure where I got WAC from….

  10. I’ve replied at http://scienceblogs.com/stoat/2014/03/20/3601/#comment-39629 re the 4-papers. We seem to read them differently.

  11. Pingback: Denier Darling’s New Job Off to Rocky Start | Climate Denial Crock of the Week

  12. William can’t have read the papers very carefully. First off, the earthquake is irrelevant to the subject at hand, Pielke is specifically talking about weather-related damage costs in his 538 diatribe. Two of the papers do not even mention how technology may have affected damage costs. The paper on tropical cyclones does make some brief mention, but the authors then dismiss the impact of changes in building codes with nary a citation or any analysis.

    William is not being honest with himself. William probably doesn’t really cares, but his handling of this reflects just as badly on him as it does on Pielke Jnr.

    • I think it’s probably better to make this point over at his place. We’ve had comments on this blog, comments on his, and emails. It’s probably best to consolidate them.

      I do think WMC has a knee jerk tendency to defend Junior for some reason, without RTFRs or conversations or what have you. I think he’s using a heuristic of “if Romm attacks it, I’ll be contrary” or something like that.

  13. Yeah, I think William is just trying to be contrary and annoy people, but that is not productive and only adds to Roger’s goal to confuse.

  14. WMC seems to like to post a defence of RPJr after a relatively quiet period. I sometimes wonder if it’s a drive to get more comments/hits. I know very probably that’s wrong, and probably wouldn’t stand close scrutiny, but that’s what it can look like. It’s also not really a criticism, either.

  15. That's MR. Ball to you.

    If I may be so bold, I think William is asking for some objectivity instead of the usual reaction to RPJ based on something other than his actual writing. I interpret this as being prompted by a sense of fairness more than anything, and tempered by a conservative approach to making “scientific” pronouncements. Why not?

    (ps. Ian Forrester – pls drop me a note: hoogreg@gmail.com)

    • WMC is expressing lopsided skepticism of Junior’s critics, because WMC does not like a group of “alarmists” and he needs someone to “attaboy” as pushback to them. If he starts looking at Roger’s claims seriously, he will see them unravel, so instead of actually checking Roger’s references and other literature, WMC is attacking the critics.

      I would have a hell of a lot more respect for what WMC is doing if he bothered to remotely investigate Roger’s claims before defending them. He didn’t. WMC is now post hoc justifying his defense of Roger by ignoring or finding a way to dismiss counterexamples people are providing him. It’s very obvious motivated reasoning.

  16. Pingback: Another Week of Global Warming News, March 23, 2014 [A Few Things Ill Considered] | Gaia Gazette

  17. Pingback: FiveThirtyEight undermines its brand by misrepresenting climate research | Dana Nuccitelli | Network and Technology

  18. Pingback: Svapo, non svapo, svapo, non… – Ocasapiens - Blog - Repubblica.it

  19. Pingback: Cherry picked and misrepresented climate science undermines FiveThirtyEight brand

  20. Pingback: Cherry picked and misrepresented climate science undermines FiveThirtyEight brand | Gaia Gazette

  21. Pingback: Cherry Picking Mars “Data Driven” Journalism | Climate Denial Crock of the Week

  22. Hmmmm it seems to me that there’s a fundamental problem with the 538 model – at least as employed by Peilke here. We skip peer review, which is the filter that catches nonsense and improves and vets models before being read by the public. Notably, this is the part of science that takes a really long time. But what happened here is that Peilke ends up making a statistically dubious claim, with dubious data, and without good vetting. There’s certainly something to be said for damages, as a % of GDP not increasing. But there’s a lot more to this story, and so many caveats, and so many other things that haven’t been controlled for. It seems like the 538 team needs a bit more internal peer review or to hire some external reviewers…

  23. Calamity Jean

    “There’s certainly something to be said for damages, as a % of GDP not increasing.” Repairing weather-related damages will also contribute to the GDP, won’t it? But the damages represent a waste of labor and capital that wouldn’t have been necessary if the weather disaster hadn’t occurred.

    This reminds me of the old economic hypothetical story about improving a village’s economy by breaking a window.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s