Fast Food Data - When “Test and Learn” Forgets the Learning
We have an odd relationship with time.
When something takes months, we assume it must be thorough. When it takes hours, we suspect corners were cut.
But time is a terrible proxy for confidence.
Think about getting to know someone. You can spend months together and barely scratch the surface — or have one dinner where the right questions reveal more than twenty conversations before it.
Discovery — no matter where it happens in the human experience — isn’t reliably measured in “time invested.” It’s measured by how reliably we identify meaningful levers for change, relevant to an audience significant enough to justify action..
And yet, somewhere between move fast, test everything and data-driven, we turned Product Discovery into fast food.
Prototype. Test. Measure. Next. Quick to consume. Comforting as a signal that problem-solving is progressing. Not necessarily nutritious — or particularly good for your long-term health.
The problem isn’t speed. And it’s certainly not for lack of frameworks. PISC, SCQA, PSB, PIR, STAR, SPIN, 3Cs, If–Then … pick your acronym. We have plenty of structures for turning assumptions into something testable.
Good Discovery can take only hours, even without AI. But only if the fundamentals are in place: The framework isn’t the skill. Knowing what to put into it is.
Can you formulate a hypothesis that can actually be proven wrong? Do you know where to look for evidence you can trust? Do you know which method can answer this specific question rather than simply produce some data?
And, crucially, can you distinguish output, outcome and business impact?
Shipping a feature is an output. A change in user behaviour is an outcome. The commercial consequence of that behaviour is business impact. One may cause the next — but writing the same idea into every box of a framework doesn’t establish that causal chain.
Yet I regularly see hypotheses that effectively read:
“We observed that strawberries are green [problem]. Therefore, there aren’t enough red strawberries [impact]. So we’ll grow more red strawberries [solution], which means more people will have red strawberries [consequence].”
A framework can force you to fill four boxes. It cannot force the four boxes to make sense. If problem, intervention, user outcome and business impact are simply different phrasings of the same assumption, you don’t have a hypothesis. You have a solution looking for justification.
And with a hypothesis like that, your project probably shouldn’t start yet. Unless, of course, discovering how quickly you can burn the budget is part of the experiment.
Get those things right and you can learn remarkably fast. Get them wrong and teams can spend months “doing Discovery” without discovering much of anything.
Today AI can accelerate research, prototyping, synthesis and analysis considerably. But ask AI to design a study or prototype without understanding what a good problem statement looks like, and it will happily produce a beautifully formatted way to learn — or build — nothing of value.
“The data says…” is one of those phrases that sounds reassuringly objective. Except data doesn’t say anything, people interpret it.
A drop in completion rate can mean the journey broke — or that you finally filtered out people who were never going to convert. Shorter time on task can mean efficiency — or frustration that ended the session early. Higher engagement can signal value — or confusion. The number can be perfectly correct. Our conclusion can still be perfectly wrong.
This becomes particularly dangerous when measurement and product scope stop matching.
Imagine launching v0.3 of a larger experience and measuring it against the business impact modelled for v1.0. Nothing moves. So the idea “failed.”
Maybe.
It’s like attempting a 10K when 300 metres still require a recovery plan — and concluding that running is inherently slow. The 10K isn’t the problem. The missing foundation is.
Evidence only becomes useful when we understand what the thing we built could reasonably have caused in context of human adaptation cycles. That means connecting: What we built to → expected user behaviour → expected business value. If the chain doesn’t make sense, the metric at the end won’t rescue it.
This is where patience and foresight matter, but perhaps not in the way we usually think - Good Discovery does not equal long Discovery.
Some questions can be answered in three hours. Others genuinely need three months. Problem Discovery is not Solution Discovery either — and they come with different questions, methods and timelines.
Behaviour — especially habit formation — may need repeated use to emerge. Trust may develop across interactions. A business effect may sit several steps downstream from the behaviour we changed — and sometimes that effect gets overlooked simply because the relevant touchpoint sits outside someone’s immediate scope or job description.
Knowing what should move, why it should move, when it should move — and what you can legitimately conclude if it doesn’t is the discipline.
That isn’t waiting. That’s strategic foresight.
Back to that dinner…
One dinner can tell you something profound about someone. Ten dates can tell you remarkably little — and sometimes vice versa.
The difference isn’t time. It’s whether you asked the right questions, paid attention to the answers, and knew how to interpret what you saw.
The same is true for everything else we’ve talked about.
Fast food isn’t bad because it’s fast. It’s bad when convenience replaces nutrition.
A 10K isn’t difficult because running is inherently slow. It’s difficult when the fundamentals aren’t there yet.
And Product Discovery isn’t rigorous because it takes a long time. It’s rigorous when the question is sound, the method fits, the evidence is trustworthy, and we know what the result can — and cannot — tell us.
Master the fundamentals, and you don’t have to choose between speed and rigour.
Speed becomes the result of rigour.