The limits of data

Having limited data can't be an excuse to do nothing

Welcome to another August edition of The Rebooting, written this week back in New York. I haven’t returned in a few months. It’s nice to be back and have a break from Miami’s August heat, which I compare to the feeling of opening a dishwasher right after the cycle ends. I’m continuing this short essay series with a piece thinking through how data is often a challenge for niche media businesses. Also, if you aren’t subscribed, please consider doing so.

The collection and deployment of data is, in theory, at the heart of just about every business these days. This is a natural outgrowth of software eating the world, because once it digests the world, it burps data. And as seen by the tech giants, data, particularly when married with networks, can be a nearly impregnable moat. Alas, the dynamics are different in most businesses.

The collection and manipulation of data is key to building sustainable media businesses. Knowing more about your audience and customers is critical to any business. Data about an audience allows for higher value ads, whether sold directly or programmatically, and is the cornerstone of the biggest growth areas for many publishers: subscriptions and commerce. And yet, data is a challenge for many media businesses, particularly niche media.

Scale still has its advantages. Data is one of the most obvious. The more data you collect, the more signals you can derive from it. More data isn’t necessarily more accurate, but more data is more likely to lead to better insights. Often executives will breezily affirm themselves “data driven.” In reality that means data is used to justify already made decisions and ignored if it points to a different course of action than the preferred one.

This is a normal state of affairs. In The Plague Year, Lawrence Wright writes about how the U.S. government’s inept response to the pandemic was hobbled by health officials paralyzed by incomplete data (thanks in large part because of the Chinese government’s secrecy). Decisive action was needed, but the federal health bureaucracy dithered. “Let the data guide us” became an excuse for inaction at a time when action was what was most needed.

For the last 9 months, I’ve done consulting with several niche publishers. One thing that comes up time and again is the challenge of using data in smart ways. The problem tends to come down to simply not having enough data to have confidence in the direction it is leading. If you have a membership with a few thousand people, you don’t have much information to go off. Your numbers can get skewed easily and lead to hasty (and incorrect) decisions. Many tools and platforms are built for having far more data in order to optimize. For niche publishers, that’s not possible.

There’s an added risk I’ve noticed over the years. That’s the absence of enough data being an excuse to put off decisions. I get it. Making decisions is risky. Sometimes the results are bad, after all. And without a ton of data, you are operating in a very ambiguous environment. But ambiguity isn’t an excuse for inaction. Make enough decisions, you’ll be wrong a lot. There’s a reason the list of the quarterbacks with the most interceptions is filled with Hall of Famers: They threw a lot of passes, including risky ones. An NFL quarterback has to fit throws into tight windows and “throw a receiver” open. Otherwise, he’ll just hold the ball too long and get sacked.

I would urge managers on our teams to get past the idea that there isn’t enough data. Ultimately, not making a decision is a decision, and it tends to be a bad habit. The tendency is to use the absence of perfect data as an excuse. It’s easier to wait until there is more data, more signal and less risk. In most businesses, it’s better to move quickly than to wait and have an incrementally better odds at a positive outcome. Some data is better than no data, and you’ll never have all the data you need to be 100% sure of a positive outcome. Better is to focus on what data you have and find ways to run specific experiments in order to produce useful data, even at a small scale. 

The flip side for managing in a data-scarce environment is the need for a higher tolerance for bad outcomes -- so long as the process for making the decisions was sound and made use of whatever information was available. In the pandemic response, Trump didn’t want to be brought data that didn’t conform to the fantasy narrative he constructed. The people working for him gave him a far rosier picture than they did among themselves. That’s a leadership failure. Trump’s not unique in building an organizational culture where inconvenient facts and data are ignored.