Why History Students Should Love Big Data

Eric Schultz

Spring 1976. Wilson Hall, Brown University. The late, great Professor William McLoughlin has just informed his 85 students in “American Social and Intellectual History” that they are to write their first paper. All he has given us is the title: “The Age of Jefferson and Adams.” We groan. Then he adds: “Keep it to three pages or less. Double-spaced.” We smile. Three pages? How hard can that be?

“If you make the margins too wide,” McLoughlin adds, “I’ll mark you down a grade.”

Needless to say, nobody got an A on that paper, or so the good professor informed us. There may have been a B or two. Not me. It was all I could do to contain my flowery opening paragraph to a single page. Some of us recovered slightly in round two, wherein we committed “The Age of Lincoln and Calhoun” to three, double-spaced pages. Some retreated to organic chemistry and other more reasonable challenges.

Little did I know, but I had just been introduced to Big Data—though it would take 35 years to earn that name. Take an endless, insurmountable, seemingly disconnected pile of information, separate the grain from the chaff (or, as my engineering buddies would say, the signal from the noise), and tell a concise, compelling story about what it all means.

For the last year, you may have noticed, it’s been hard to escape stories about “Big Data.” In a world where everything can be measured—from your location to how well you sleep to how long you brush your teeth to all of your “Likes” on Facebook, Big Data is upon us with avengeance. Or, as it were, like a Cloud.

Some of you will be lifelong historians and wrestle with Big Historical Data for your careers. I happened to take a left turn into business school and ended up working at and running companies. In the process, I spent an awful lot of time pondering questions about marketing and strategy.

This is how strategy works: Take an endless, insurmountable, seemingly disconnected pile of information, separate the grain from the chaff, and tell a concise, compelling story about what it all means. Sound familiar? I’ve had to do that kind of work in everything from baby products to pet food to Red Sox baseball to the global perishable supply chain. I thank my lucky stars every day for Professor McLoughlin.

Now, we’re being told, in the emerging world of Big Data there will be more and more piles of the stuff lying around. Is there any group in the world better trained to make sense of it all—to wade confidentially into the sea of Big Data—than historians? It’s not just about monitoring, data gathering, and quantitative analysis. (Though a course or two in statistics is a good replacement for the Greek most of us got to skip, and get yourself over to the Computer Science building and spend a semester or two doing some simple coding, just so you can see how the other half will live their lives.)

But in the end, conventional data analysis falls short. When Chris Anderson (of “long tail” and “information wants to be free” fame) wrote that “[w]ith enough data, the numbers speak for themselves,” he was dead wrong. Causal analysis is an extraordinarily deceptive and nuanced thing. Data sets on their own are neutral and largely useless. Cobbled into relational information they begin to sing. But only when a storyteller comes along and provides context and human insight does Big Data really give up the goods.

Tom Friedman, author of The World Is Flat, believes that integration is the new specialty—that someone with a renaissance view of the world is more likely to spark an innovation than a pure engineer. If you are learning the craft of history, that could very well be you.

I do not know exactly what Big Data jobs will look like over the next generation, but I couldn’t predict a decade ago that there would be thousands of “app developers” or positions called “Chief Evangelist” or professional bloggers. I certainly didn’t know to put “Pope” and “tweet ‘in the same sentence. But I stand firm in the belief that God blesses the storyteller; it is he or she who makes data human, and our only real chance to use it like a tool instead of a club.