Suggested Readings List

An international collaboration of the best of data journalism worldwide.

Daniel Gilbert had spreadsheets with thousands of rows of data. "There was a story there, he was certain. But control-f would not find it." After he took a class on how to use Microsoft Access, he could finally ask and answer the questions he knew were in the data. Gilbert's meticulously detailed series on landowners being cheated by oil and gas companies ended up winning a Pulitzer.

How the structure of data can create new kinds of storytelling.

How exactly do you define "worst"? Here's a high-level description of the factors reporters used to rank the performance of charity, and the amount of data they had to sift and analyze to do so.

A video report on data visualization as a storytelling medium

Using the MTA turnstile data to see what kind of crowds Jay-Z drew at Jay-Z's inaugural concert with Barclays Center.

WNYC's political team wanted to know: how much money was being raised by candidates for a state legislative district from within the district itself? John Keefe walks through his use of Fusion Tables and additional database techniques for the analysis and visualization.

One day John Keefe was just browsing the official NYC data site and tucked away some code that involved the city's flood zones. When Hurricane Irene stuck, NYC.gov was flooded (with web traffic) and WNYC became an invaluable resource.

Why aren't more Congressional reporters taking advantage of the copious generated by our legislators?

A brilliant walkthrough of benign data sets and the code that could potentially launch investigative projects.

This 2013 Pulitzer Public Service winning story epitomizes the best of watchdog journalism, as its expose of a deadly public injustice sparked swift reform. But it deserves an award for its ingenious do-it-yourself way of collecting the kind of data that, in theory, was non-existent.

OK, this is not a traditional data journalism book, though it's one of the best true crime and/or journalism books ever written. Simon does keep and review a tally of the bodies and where they marticulate in the Balitmore justice system throughout the year. Also, there's a memorable passage about the "murder board", a simple visualization that has unintended consequences on the efficency of the homicide squad.

The modern computing age may have irreperably damaged the business of journalism, but it may also streghten the act of journalism.

This landmark study that studied whether financial disclosure laws were actually being followed was the result of old-fashioned data collection: travelling to Minnesota and manually copying documents, before analyzing them in Access.

"That intellectual curiosity. That bullshit detector for lack of a better term, where you see a data set and you have at least a first approach on how much signal there is there...That stuff is kind of hard to teach through book learning. So it’s by experience. I would be an advocate if you’re going to have an education, then have it be a pretty diverse education so you’re flexing lots of different muscles...You can learn the technical skills later on"

Is it possible to run a mugshot-aggregating website and still follow journalistic ethics?

When there is no data, build your own. After discovering that no one was keeping track of Washington D.C.'s numerous homicide victims, Laura and Chris Amico simply started counting. The hand-built data source became a community resource as well as a source for data analysis.

"But when you open up the Excel spreadsheets from the VBA, you’ll find lots of figures with little explanation. So we’ve put together this guide to get you started in your pursuit of good local, state, regional and national stories that are to be had in the data. A grounding in Excel basics is needed to follow along."

A great example of making use of limited data through plain deductive reasoning, cleverness, and good reporting. Comes with pretty pictures, too.

Computers have made mapping almost too easy, to the point where we overlook clearer, more efficient ways to display data that isn't particularly geospatially interesting.

A well-lillustrated list of design concepts to bring to visualizations and news interactives.

Are modern web interactives becoming too complicate for users to interpret? Probably.

How ugly formatting, including improperly bolded text, can obscure the important information in technical presentations.

A great discussion between Edward Tufte and his readers on why pie charts aren't ideal.

Chart types, from simple to exotic, in tabular form.

Applying Tufte's principle of small multiples to online video thumbnails.

Charts can be too beautiful for their own good.

An awe-inspiring visual essay from the creator of D3, its purpose "to share my love of examples with you."

Drawn sketches can hint at what's visually compelling to the designer.

This is not really about data, but about the details of the creation process, and the importance of streamlining your workflow.

Another popular web visualization that is more junk than data.

When you want something done beautifully, sometimes you have to do it yourself.

A critique of a well-liked New York Times interactive.

A critique of map design and thresholds, in the contest of the NYPD's stop-and-frisk data

A case for where hand-editing has advantages over algorithmic rendering.

The beach ball chart is about as dumb as it sounds.

Millions and millions of beautiful dots, without crashing your computer

An behind-the-scenes explanation of how one of the best web visualizations ever was built.

How to use Google-provided election data to build a Fusion Table-powered map.

With Fusion Tables and a little Ruby magic, John Keefe describes how he mapped contributions to presumptive NYC mayoral candidates by zip code by lunchtime.

“When we understand that slide, we’ll have won the war.”

A visualization of what makes great visualizations at the New York Times

History's most memorable 3D-column chart.

A modern critique of Florence Nightingale's, "Diagram of the Causes of Mortality"

A (personal) blog of data sketches from the New York Times Graphics Department. Maintained by @KevinQ.

"Most of what we can do to make the world a better place involves, not doing the unprecedented, but doing what matters and what works, whether unprecedented or not. This might not be as exciting as the unprecedented, but it’s desperately needed."

An elegant and compact way to encode a metric with comparative data.

This book was revolutionary when Tufte published it in 1982 and it remains just as relevant in our current state of PowerPoint and tacky CSS transitions.

After you laugh, remember: he's a Nobel Prize winner.

There's no shortage of ways to make data just a little bit easier to work with, and usually the effort is worth it.

The importance of limiting colors in a visualization is one of the most un-obvious principles to grasp. And yet, once it's explained to you, you'll immediately look down on everyone else who failing to see the tackiness.

How to turn a PDF into a Leaflet-powered crime map.

WTF indeed

Is the Tufte-school of aesthetic too boring for our Jerry-Bruckheimer-corrupted brains?

A rebuttal to the University of Saskatchewan paper on whether "chartjunk" is more effective in conveying data.

"Mapping is not value-free, dangers occur when professional cartographers and geographers attempt to map data from fields in which they are ignorant. "

A nice guide on to turn stock Excel charts into something Tufte-approved.

A nice list of visualization blunders, though the main reason I included this was for its funny title.

Even scientists have trouble with data visualizations

Edward Tufte's influence arrived well before the Web (and yet still persists). Here's one of his relatively few reflections on web-based visualizations.

A hallmark essay on how data can be used in the very core of journalsitic operations, by the journalist who created the popular Django framework. Too bad his advice hasn't been much heeded.

An influential economic analysis by two Harvard economists, based on a spreadsheet snafu? This paper and the articles that followed are a bag of laughs.

Google Refine is now Open Refine and its interface and feature sets have improved quite a bit since I wrote this essay. But it's still a good overview of how a single digital tool can make a data-intensive investigation possible.

The horror of scraping government data.

A brilliant example of a (legal) hack to get public data.

A three-part series to just how complicated just doing data updates can be.

Etsy found interesting and practical ways to mash up the (non-HR-private) data of its staff.

If you're new to SQL, or even if you're skilled at it, this is one of the best collections of virtually every half-way complicated query you can dream of.

A clever example of how to use code and available technology to track variables in the "meatspace"

A high-level description of the engineering used to mine and index 3.4 billion web pages in about 14 hours.

A guide to practical data mining, collective intelligence, and building recommendation systems

If there's one data-related skill I wish I had learned years ago, it would be regular expressions.

Easily one of the most esoteric and brilliant crowdsourcing data projects yet conceived.

This says everything you need to know about the complexity and annoyance of real-life data: "For example, in the Senate it is possible for the Majority Leader and Minority Leader to alter the rules of math when it comes to how many senators constitute a three-fifths majority."

Citizen data journalism at its best, from John Krauss: "I already knew that they were finally releasing -- after the Council forced them to -- crash data as idiotically obfuscated PDFs, but reading that they justified this out of concern for "the integrity of the data," was so galling that it goaded me into action. I would make the data accessible as friendly, parseable CSVs."

When it comes to large data-projects involving messy data, don't always get hung up on the technology to use. Coming up with an efficient way to divide and distribute the data collection and cleaning, even within your own newsroom, can often do the job

If you've ever tried to scrape a non-trivial website, you'll understand why this book is 80,000+ words long.

Quick, light-weight ways to reshape and visualize data from the command-line.

SQL joins illustrated with Venn diagrams

Mmmm...munging

A short readable guide to one of the most vital and fundamental data journalism tools.

A wonderfully clever way to use computer-vision and a webcam for in-the-wild text recognition

A comprehensive guide to properly using indices in SQL databases: a must for any developer dealing with more than a few million rows of data.

A free book written by one of the best R programmers around, aimed at experienced programmers wanting to use R as a programming language.

A long list of slides about statistical data mining and foundational principles of statistics.

This document is aimed towards game developers, but it is the most thorough explanation of how to best incorporate creative writing into a dynamic code-generated experience, while giving writers the most degree of freedom possible. It's a great examination of the core facets of good writing and goes deep into the technical implementation.

Besides data-munging, taxonomy is probably the most underappreciated data skill.

Even nuclear historians write web scrapers.

Predictably, the answer is very complicated and involves calculus.

A short list of statistical snafus as applied to data journalism.

A nice explanation of selection bias, in the context of Twitter and other social media.

An introduction to Probability and Statistics for Python programmers.

An exploration of Bayesian statistics, using the Python programming language

A great explainer of how to implement real metrics to the field of news applications.

The master of presidential predictions explains why he's not so great at predicting Oscar winners.

Simulations don't yet have much place in journalism, but I like this analysis of laundry room behavior because it acknowledges the limitation of analysis and the impact of assumptions, as well as intuitively walking through the design of the simulation (with Julia code snippets). Also, it tackles a question near to all of us.

"Merely knowing who people are, paradoxically, isn’t very predictive of their movie tastes."

The final product of this fashion interactive is of course a visualization. But Hinton's writeup is more about the analysis used to "fingerprint" each fashion pohoto.

Saunders's blog is fascinating to me, as he describes his struggles (and triumphs) as a hobbyist programmer in a field - bioinformatics - that you'd think would embrace his type of coding skills. Here, he shows the code needed to analyze the writing tics found in 47 gigabytes of PubMed abstracts.

This is both an interesting read about sex and relationships as well as something that should pique your interest about how online services measure your behavior.

Because even his elite engineering and mathematics friends didn't understand how easy it is to make a spell-checker, Norvig explains the theory and shows the less-than-a-page-worth of code needed to achieve 80 to 90% accuracy.

Here's a way to look at data science as five distinct steps. Also, it's the first I've heard of the awesome acronym, OSEMN.

After a triumphant night of presidential election predictions, Nate Silver rubs it in by examining how the vairous pollsters did in their predictions.

"This text­book is intended to pro­vide a com­pre­hen­sive intro­duc­tion to fore­cast­ing meth­ods and present enough infor­ma­tion about each method for read­ers to use them sen­si­bly. We don’t attempt to give a thor­ough dis­cus­sion of the the­o­ret­i­cal details behind each method, although the ref­er­ences at the end of each chap­ter will fill in many of those details."

A brief overview geared toward engineers with no prior knowledge of machine learning.

A health-care themed example of how data analysis can discover counterintuitive solutions.

One of the classics of scientific papers, influential and controversial to this day.

"What’s actually going on is that the division between red and blue America is mostly about a split among richer voters."

"Rap’s history has been traced many ways -- through books, documentaries, official compilations, DJ mixes, university archives, even parties. But until now you haven’t been able to look at the development of the genre through its building blocks: the actual words used by emcees."

This essay's theme encompasses the frustration of engineers, the wild imaginations of product managers, and the proper use of statistics.

A free online guide to the most popular statistical errors committed by working scientists and analysts.

Another generously-made-free book on data science.

Another detailed write-up about web analytics and modern news organizations by OpenNews Fellow, Brian Abelson

"The states where a single vote was most likely to matter are New Mexico, Virginia, New Hampshire, and Colorado, where your vote had an approximate 1 in 10 million chance of determining the national election outcome. On average, a voter in America had a 1 in 60 million chance of being decisive in the presidential election."

A primer to the latest buzzword in big data and artificial intelligence.

One of a series of posts tackling the eternal, "If correlation isn't causation, than what is?"

It's not what you say but who you like.

"As far as I'm concerned, Gould's The Median Isn't the Message is the wisest, most humane thing ever written about cancer and statistics. It is the antidote both to those who say that, "the statistics don't matter," and to those who have the unfortunate habit of pronouncing death sentences on patients who face a difficult prognosis"

Correlation doesn't equal causation, but the strategy behind this analysis does a good job to isolate the hidden factors here.

In a tournament of freestyle chess, the winners are not a human grandmaster, nor the best supercomputer, or even a grandmaster with a supercomputer, but amateurs who best know how to use their mediocre computers.

This paper discusses how the current standards in psychology research make it more likely for false positives to be found.

I don't know what's more astounding: that 30 years ago, before computers were in our homes, Arthur Luerhmann could so clearly describe the difference between being taught by a computer and learning to teach the computer, or that 30 years later, so few people even realize there's a difference.

It turns out that the work of data journalism and startup founders wanting to appease Google are very much alike. Excellent write up and source code to check out here.

I think the metaphor of "dry lab" and "wet lab" work applies to journalism too.

"An astonishing act of statistical chutzpah in the Indiana schools’ grade-changing scandal."

"P values, the 'gold standard' of statistical validity, are not as reliable as many scientists assume."