Wednesday, April 10, 2013

Why History Students Should Love Big Data

Eric Schultz

Spring 1976. Wilson Hall, Brown University. The late, great Professor William McLoughlin has just informed his 85 students in “American Social and Intellectual History” that they are to write their first paper. All he has given us is the title: “The Age of Jefferson and Adams.” We groan. Then he adds: “Keep it to three pages or less. Double-spaced.” We smile. Three pages? How hard can that be?

“If you make the margins too wide,” McLoughlin adds, “I’ll mark you down a grade.”

Needless to say, nobody got an A on that paper, or so the good professor informed us. There may have been a B or two. Not me. It was all I could do to contain my flowery opening paragraph to a single page. Some of us recovered slightly in round two, wherein we committed “The Age of Lincoln and Calhoun” to three, double-spaced pages. Some retreated to organic chemistry and other more reasonable challenges.

Little did I know, but I had just been introduced to Big Data—though it would take 35 years to earn that name. Take an endless, insurmountable, seemingly disconnected pile of information, separate the grain from the chaff (or, as my engineering buddies would say, the signal from the noise), and tell a concise, compelling story about what it all means.

For the last year, you may have noticed, it’s been hard to escape stories about “Big Data.” In a world where everything can be measured—from your location to how well you sleep to how long you brush your teeth to all of your “Likes” on Facebook, Big Data is upon us with avengeance. Or, as it were, like a Cloud.

Some of you will be lifelong historians and wrestle with Big Historical Data for your careers. I happened to take a left turn into business school and ended up working at and running companies. In the process, I spent an awful lot of time pondering questions about marketing and strategy.

This is how strategy works: Take an endless, insurmountable, seemingly disconnected pile of information, separate the grain from the chaff, and tell a concise, compelling story about what it all means. Sound familiar? I’ve had to do that kind of work in everything from baby products to pet food to Red Sox baseball to the global perishable supply chain. I thank my lucky stars every day for Professor McLoughlin.

Now, we’re being told, in the emerging world of Big Data there will be more and more piles of the stuff lying around. Is there any group in the world better trained to make sense of it all—to wade confidentially into the sea of Big Data—than historians? It’s not just about monitoring, data gathering, and quantitative analysis. (Though a course or two in statistics is a good replacement for the Greek most of us got to skip, and get yourself over to the Computer Science building and spend a semester or two doing some simple coding, just so you can see how the other half will live their lives.)

But in the end, conventional data analysis falls short. When Chris Anderson (of “long tail” and “information wants to be free” fame) wrote that “[w]ith enough data, the numbers speak for themselves,” he was dead wrong. Causal analysis is an extraordinarily deceptive and nuanced thing. Data sets on their own are neutral and largely useless. Cobbled into relational information they begin to sing. But only when a storyteller comes along and provides context and human insight does Big Data really give up the goods.

Tom Friedman, author of The World Is Flat, believes that integration is the new specialty—that someone with a renaissance view of the world is more likely to spark an innovation than a pure engineer. If you are learning the craft of history, that could very well be you.

I do not know exactly what Big Data jobs will look like over the next generation, but I couldn’t predict a decade ago that there would be thousands of “app developers” or positions called “Chief Evangelist” or professional bloggers. I certainly didn’t know to put “Pope” and “tweet ‘in the same sentence. But I stand firm in the belief that God blesses the storyteller; it is he or she who makes data human, and our only real chance to use it like a tool instead of a club.


Randall said...

Eric: This is really interesting. I hadn't thought of history and the telling of a story as linked to Big Data as you suggest. I like how you put it: "If you are learning the craft of history, that could very well be you."

Case in point: NPR's recent story on data mining, history, and emotions.

"Were people happier in the 1950s than they are today? Or were they more frustrated, repressed and sad? To find out, you'd have to compare the emotions of one generation to another. British anthropologists think they may have found the answer — embedded in literature."

Gabriel Loiacono said...

This is a really helpful article from the "careers for history majors" point of view. Good work!

dianee said...

What a marvelous perspective! I recently changed the focus of my life from a career in IT to genealogical research and have learned that contrary to my miserable experience in high school, history is fascinating and really about life. Thinking about it in a Big Data context makes so much sense. Thanks for an enlightening story!

Lisa Clark Diller said...

I will be cribbing from this for years. Well said, with the legitimacy of the "real world" experience to back it up. (My students are very skeptical about any claims I make that impinge on the world of work outside academia.)

And I hadn't heard the NPR story, Randall, so thanks for pointing it out.

Eric Schultz said...

Thanks, everyone! One of the things I didn't mention about Prof. McLoughlin (which you may know): As soon as class ended, he'd head down to the bottom of College Hill, or into town, or join a bunch of students nearby and protest unfair wages or work conditions, or American foreign policy, or (before my time) the Vietnam War. He was fearless, a person of principle, and an inspiration in the classroom and out. But he was one nasty grader! (P.S.--off to find Randall's NPR story; thanks!)