frog实验揭示AI合成研究局限：速度不等于理解，需建立人机混合边界

For years, those of us working in design research have argued a simple idea: to design better products, services and experiences, we must first understand the people we are designing for. Yet this has rarely been an easy case to make. Pressure to launch faster, limited budgets and the assumption that “we already know our users” have too often pushed research down the priority list.

Paradoxically, these decisions were meant to accelerate innovation but often achieved the opposite. People constantly evolve. Their expectations, behaviors, technologies and contexts change. Research does not exist to confirm what we think we know. It exists to uncover what has changed while we assumed everything had stayed the same.

Then AI arrived. What once seemed slow, costly and difficult suddenly became possible in minutes. Synthetic research—using AI-generated users, interviews and insights to simulate research activities—promises to democratize a capability long constrained by time and budget.

This is an extraordinary opportunity, but also a new responsibility. We must avoid confusing speed with understanding, coherence with evidence and probability with knowledge. AI is transforming not only how we conduct research, but also how organizations learn. The question is no longer whether AI can do research for us, but how to harness its potential without sacrificing the rigor that turns information into evidence—and evidence into better decisions.

The wrong question

When we first began experimenting with AI-enabled research platforms at frog, we expected answers, but what we found were much better questions.

Our experiments align with an emerging trend in the research community. As Nielsen Norman Group has also highlighted in its reflections on AI and UX Research, the real challenge is not automating research but preserving human judgment where it creates the greatest value.

At the same time, we are seeing more organizations adopt AI-augmented research platforms before defining when to use them or how to govern them. Adoption is moving faster than our ability to establish the principles needed to use these tools rigorously.

Great research is not just about finding answers. It is about uncovering the questions we have not yet thought of asking, revealing contradictions and allowing reality to challenge our assumptions. The studies that truly change decisions are rarely comfortable: they question certainty, challenge assumptions and force us to rethink our direction. That capability remains human.

The question is no longer whether AI will replace parts of research. It is how we combine the strengths of artificial intelligence with what remains uniquely human.

Because the future of research will be neither fully human nor fully artificial. It will, necessarily, be hybrid.

When research becomes synthetic

Over the past few months at frog, we moved from theory to practice. Rather than debating what synthetic research could do, we wanted to understand what happens when it becomes part of our everyday design work.

To explore this, we ran a series of experiments across different contexts: comparing synthetic and real users, testing interfaces, conducting sensitive interviews, evaluating user experience on prototypes and internal tools, and generating research materials. The goal was not to validate the tools themselves, but to observe how they behave, where they add value and where they begin to break down.

Looking at these experiments, a consistent pattern started to emerge. Synthetic outputs are clear, structured and easy to use. They reduce noise, compress complexity and make it easier to move forward. But that clarity comes with a trade-off: friction disappears. Contradictions become rare and uncertainty is resolved too early.

In comparison with real users, this became particularly visible. In our scenario, younger participants in real interviews expressed hesitation, changed their minds and struggled to articulate their future choices. Synthetic users did not. Their responses were coherent, confident and aligned. What in reality was a process of doubt became, in simulation, a resolved narrative.

The same happened in other contexts, including sensitive interviews with survivors of natural disasters. While synthetic interviews can recreate respectful language and well-structured narratives, they remain fundamentally detached from how these conversations unfold. Having worked on this type of research in real settings, one detail becomes particularly clear: these interviews are not just about what is said, but about how and when it is said. They require pacing, pauses and the ability to adapt in the moment. Synthetic interviewers, even when configured with follow-up questions, tend to run from beginning to end without questioning the interaction itself. They do not stop; they do not adjust or recognize when a participant might need space or when a conversation should naturally shift direction.

In usability testing, while evaluating user experience on prototypes and internal tools, interactions felt smooth, almost too smooth, with no hesitation or drop-off. In designing research studies, outputs were structured but lacked nuance and required significant reworking to become usable.

What we were seeing was not incorrect, but something fundamentally different. Synthetic research does not observe behavior. It constructs a plausible version instead.

And that distinction matters because it changes the nature of what we call insight.

What should be automated

Once AI-augmented research becomes part of the process, the most practical question is not whether to use it, but where it can help without distorting the outcome.

What we have seen is that synthetic systems perform best when dealing with scale, repetition and structure. Tasks such as transcription, clustering, desk research or early synthesis can be accelerated without significantly changing the output. In these cases, automation reduces effort, not meaning.

This is where synthetic research creates its clearest value. It allows teams to explore more directions, structure information faster and move earlier in the process. It expands capacity and reduces operational friction, particularly in the early stages where the goal is to understand the problem space rather than validate decisions.

However, this advantage is held only while automation stays within these boundaries. The moment it begins to influence interpretation or decision-making, the nature of the output shifts. Synthetic systems do not simply process information. They shape what we see.

Because outputs arrive already articulated, already structured and often convincing enough to act on, the line between preparation and decision becomes blurred. What seems like efficiency can quickly turn into unexamined assumptions.

This is why automation in research is not only about speed. It is about understanding what kind of work is being accelerated and what kind of thinking might be replaced along the way.

What must remain human

If synthetic systems expand what we can automate, they also clarify what cannot be delegated.

Across all our experiments, the same limitation appeared in different ways. Synthetic outputs struggle not with complexity itself, but with the kind of complexity that depends on context, contradiction and lived experience.

Judgment is central here. Deciding what matters, what is relevant and what should be questioned is not a procedural step. It requires interpretation, context and often an awareness of things that are not explicitly visible. Synthetic systems can generate patterns, but they do not understand why those patterns matter.

The same applies to meaning. Insight is not just a structured summary. It is a shift in understanding that connects signals to decisions. It often emerges from ambiguity, from things that do not fit, or from moments that require interpretation rather than resolution.

This becomes more evident in sensitive or high-impact contexts. As we observed, synthetic systems can reproduce how people describe experiences, but not how those experiences unfold.

Synthetic empathy is linguistic, not experiential.

And this is where the limits become critical to acknowledge. Plausibility is not evidence. A coherent narrative is not the same as understanding. And a well-structured output does not guarantee validity.

This is also where positioning AI-augmented research correctly becomes essential. It works well as an upstream layer, supporting exploration, hypothesis generation and preparation. But it should not be used as the sole source of validation, particularly in contexts where decisions depend on real behavior.

This distinction is summarized in the framework below, which shows where AI-enabled research creates value, where it requires human judgment and where real human evidence remains essential.

The question is not whether AI can replace research, but where each approach creates the greatest value. The boundary is not fixed, but the pattern is clear: when AI is used to explore, it expands thinking. When it is used as evidence, it can narrow it.

And that shift leads directly to the next question: not only what we automate or what remains human, but how both should work together.

Human-led, AI-augmented model

From everything we observed, a clear pattern emerged. Synthetic research works best not as a replacement, but as part of a hybrid sequence that combines human judgment and artificial intelligence with clear boundaries.

Rather than replacing one another, human expertise and AI contribute different capabilities at different moments in the research process. What emerged from our experiments was not a linear handoff, but an iterative model in which human expertise and AI each play a distinct role.

Not everything in research should be automated. Not everything needs to remain human. The challenge is knowing where each creates the most value. Research is no longer a choice between humans and AI. The question is which role each should play in the process.

At its best, this model does not reduce the role of research. It reframes it. The role of the researcher shifts from collecting information to designing the conditions where insight is generated, challenged and turned into evidence.

The real challenge is not how much we automate, but how deliberately we combine both.

Because the true risk is not that AI-augmented research is wrong, but that it becomes convincing enough to stop us questioning it.

So, what becomes important is not the sequence itself, but the clarity of the boundaries between these roles. When these boundaries blur, problems appear. If AI defines the problem, assumptions become invisible. If it drives interpretation, plausibility can be mistaken for insight.

Designing the next generation of research capabilities

If there is one thing we have learned throughout this exploration, it is that the future of research will not depend on who adopts AI first, but on who learns to integrate it with the greatest judgment. AI is forcing us to move beyond a project-based view of research toward organizations that learn continuously. This requires far more than adopting new tools. It means deciding which tasks should be automated, which must remain human-led and how to combine evidence from real users, artificial intelligence and behavioral data without compromising rigor.

It also demands new capabilities: evaluating AI-generated outputs, identifying bias, validating evidence and establishing governance mechanisms that determine when a response is reliable enough to support a decision. Paradoxically, the better machines become at generating answers, the more valuable uniquely human capabilities will become: asking better questions, interpreting nuance, connecting seemingly unrelated information and challenging assumptions.

The future of research is not about doing less research. It is about doing it differently: more continuously, more strategically and, necessarily, more hybrid.

The future belongs to those who learn best

AI will continue to evolve. Models will improve, new platforms will emerge and many of today’s limitations will disappear. But the greatest challenge will not be technological—it will be human judgment. As Harvard Business Review argues, competitive advantage will no longer come from producing information faster, but from interpreting it critically and applying it wisely.

For years, the biggest obstacle to research was obtaining enough evidence to make confident decisions. Today, we face the opposite challenge: answers arrive within seconds, but not every convincing answer is evidence. That responsibility no longer belongs only to research teams. It belongs to every leader making decisions about products, services, customer experiences, and innovation.

At frog, we have been exploring this transformation alongside organizations already adopting these technologies. We do not believe there is a universal answer. But we are convinced that competitive advantage will belong not to those who adopt AI first, but to those who learn when to trust it, when to challenge it and when to rely on people instead.

Because the true value of research has never been about producing answers. It has always been helping organizations make better decisions. AI will democratize answers. Evidence will become the true competitive advantage. And knowing when evidence is strong enough to shape decisions will remain profoundly human.