The Uncomfortable Truth: Data Lies, Making Anything "Real"

How numbers are shielding flawed research from criticism, distorting our reality

A data point doesn’t make it true.

In our strategy work, we often leverage data as a form of assurance, but like any drunken anecdote, data is also prone to blurry fabrication and biased self-interest.

Data is a crutch. And it can deceive effectively.

As one trend report this year confidently proclaimed:

“Searches for Moo Deng increased +98,588%!”

Cool. But who the hell was searching for Moo Deng, the viral pygmy hippo, a year before Moo Deng even existed?

Remember: an increase from 1 search, to 2 searches is a whopping +100% growth.

Don’t always trust the stat.

I was recently interviewed by Katie McQuater from Research Live & The Market Research Society (MRS), the U.K. professional body for research, insight and analytics.

When asked about my outlook on the research and insights industry — “Are we at a turning point?” — my fears of AI didn’t seem to match my peers’.

Instead, I answered that my most pressing concerns were more human — specifically, our fallible approach to the proliferation of available data. Or in other words: too much info, not enough sensitivity and sense-making.

I shared,

“We continue to gain access to truly invaluable data, and in ways that we haven’t had access to before. I think we know that the world runs on data, and as insights people, we are grouping as much of that as we can to try to make sense of it.”

“[But] I don't know how much of that data we actually need in order to find insight... as much as we just want to say we have it. [And secondly], in light of all that access to data, the more we collect it, the more of a responsibility we have to analyze it rigorously.”

I don’t see a whole lot of collective interest in this responsibility.

“We think a data point is unwavering support for whatever the claim is, and the truth of the matter is: data lies. Data can tell any story you want it to tell. And as we've gained this unprecedented access to data, we think including the stat or the percentage point is the trump card – saying ‘data said it’ – forgetting that the ways in which that data is captured, who it represents, the ways it’s analyzed, the ways it’s phrased, the ways it’s visualized, this all tells stories on its own.”

“My fear is data is used in a way that does not necessarily represent cold hard objective truth and is being used as entertainment and sales. It’s being used for narrative construction.”

“And the reason this concerns me is because if you look at the trickle-down effects of this, do this enough, and data eventually loses its power, its truth, its weight. And if we lose that, I don't know what else we have.”

Integrity is all we’ve got.

One of the most beloved annual trend reports is quite lax in its use of statistics, yet the inclusion of numbers creates an effective halo. A mirage of accuracy. A forcefield from criticism.

However, this warm glowing presence of percentage points should not automatically be respected.

As they proclaim,

“[Our] trends last nearly 2X longer than trends originating elsewhere, growing bigger over time.”

Let’s unpack this. If you’re using search data to represent social change, claiming search activity as “ours” is awfully presumptuous. No one platform can own interest. Yes, one platform can be where a search is occurring, but there are days, weeks, months or sometimes even years of influence compounding before that search even transpires.

Further, as for “...than trends originating elsewhere” — what’s the fair, statistical comparison for “elsewhere?” Whose trends and from where, when or how? How do you even get “their” data to make a fair comparison to then make this claim? Regardless...

This “vs.” doesn’t convey a sound understanding of how culture fundamentally works:

Trends do not emanate from singular locations like a sterile test tube.

Social change emerges from collisions.

Culture is a vast, historically rich ecosystem of influences — actions and reactions.

Sticking a flag in “our trends” is egotistical, corporate colonization, revealing a dark preference for ownership over respecting the cultures themselves. I assure you, this thinking never ends well.

Meanwhile, the organization continues on with a case study: After coining an observation gleaned from their data, they found that mentions of “their trend” then grew +127%. With that as an example, they claim,

“80% of the predictions we’ve made have come true.”

But of course mentions will grow after seeding a made up “trend” name across the internet. Press connections, pitched stories and marketing budgets are doing a lot of heavy lifting for “trend prediction accuracy.”

This is the self-fulfilling prophecy taken to an audacious extreme.

This is strategic astro-turfing.

This is trend laundering, hyping something into fruition.

An org does not care about trend accuracy as much as landing advertising budget. Unfortunately, “trend accuracy” just becomes a selling point at the expense of true cultural intelligence and insight.

But in all fairness, if an org has the resources to prop up and puppeteer “the success” of their identified trend, why wouldn’t they? It’s “market making”... but for a micro-trend with no real value or connection to human needs. It’s just to sell more ads. And these paid ads then fund the budget and team to create more reports to repeat the cycle.

Futures literacy isn’t just about understanding the future and one’s role in its creation, but becoming intimately aware of how the trend sausage is processed.

...And it’s not always sanitary.

Buyer beware.

I concluded my interview with McQuater of MRS with:

“As insights people, we are afraid of contradictions. If data point says A and data point says B, and those two things contradict one another, we're going to throw one of those out to make sure we tell a consistent, standardized, believable story.

But we often forget that where there is contradiction, when things don’t add up, [that’s] where there’s true insight.”

“I think what’s happening is because there is so much data, and because there’s so much opportunity, we get to cherry pick, we get to pick and choose what stories we want to tell. We’ve optimized for convenience.

So [for our industry], I think that the opportunity is to reclaim inconsistencies and reclaim things that don't add up. And by doing so, you can honor the integrity of the data and honor what the true insight is.”

As for using data to support our cultural theories — it’s not always necessary. After all, the best insights are the ones you can feel, and no percentage point can compete with that.

This ultimately begs the question: Are we hiding behind the data, and if so why?