AI in UXR, Part 1 – It’s Not All Awful

Back in 2023, I was asked to offer some perspectives on how AI was beginning to affect the practice of UX and UX Research, and where I thought the impact might be largest, most positive, and most detrimental. At the time the AI-hype machine was just beginning to cycle up (though I am surprised by how frenetic that cycle continues to be), and as the UX Research field at large was expressing decidedly mixed (and mostly negative) opinions about the technology, I did my best to approach the task with objectivity, and even-handedness. Two years ago, I made three predictions:

1. As we didn’t yet know what AI might be capable of, UX Researchers could best direct their efforts towards understanding its impact, and studying how people interact with AI.
2. LLMs were most likely to initially prove useful in summarizing transcripts, and might get good enough at reviewing multimodal input (text, voice tone, facial gestures, etc.) to enable quicker research insights.
3. Artificial users were likely to only be useful in very limited cases where you were performing usability testing on possible design solutions to well-defined problems.

As I look back on those initial impressions, I find that in most areas I haven’t changed my mind much: the promise of AI is potentially tectonic, persistently overpromised, and so far largely undelivered. The UX Research field has undergone a massive shift in terms of staffing, but I am dubious that has much to do with AI and LLMs – the downturn began before ChatGPT 3.5 was released, and hiring now seems to be picking back up.

In the meantime, I can say this much: the benefits of AI to the UX Research field are real, if (so far) quite modest.

Despite the promises of “instant feedback” from “artificial users” and real-time analysis of qualitative interviews, the main benefits I’ve experienced have been at the end-caps to the research process:

Qualitative Research Prep

In preparing for research, at least some LLMs (like Gemini and Claude; Copilot to a lesser extent) provide a more natural interface with which to discover background industry information that helps you get a “lay of the land.” For UX Researchers working across many different industries (such as in an agency environment), this can be quite helpful. Our expertise in research methods, synthesis and meaning-making translate well across industries and verticals, but faced with a new client and their audience’s bespoke nomenclature and habits, building up expertise in the area of inquiry can be a challenge.

As an outside consultant to many organizations, I can bring a useful perspective and willingness to ask questions. And as a UX Researcher, my primary job is to listen to and understand the human behaviors and thinking styles being exhibited by the people I’m interviewing.

But I find it’s immensely helpful if the questions I ask are phrased in language the participants find familiar, and to the extent that LLMs can help with this English-to-English translation, I find the process accelerates my own preparation. It also helps move an interview along into the core of what’s most interesting (if you’ve only got an hour with someone, you want to help them get into the deepest part of the conversation as quickly as possible.)

I believe that to truly maximize the impact of research, UX Researchers need to be well-versed in the language of the organizational environment in which they’re working. Developing a deep understanding of how the business works, and the trade-offs other people need to make in order to further the strategy of their organizations, is invaluable. Human behavior is devilishly complex, and the more you understand about the hidden machinery behind an organization’s dynamics, the better equipped you’ll be to untangle individuals’ responses to that machinery.

Research Interview Transcription

Both during and at the end of research, near instantaneous transcription (and in some limited circumstances, summarization) of interviews can be an immensely powerful tool. If your data protection environment permits it, Copilot’s or Zoom’s ability to provide transcripts and topic summaries of lengthy interviews speeds along the process of reviewing and extracting useful information from them.

(Is this a case of AI taking away a job? Perhaps; in more instances however, I’ll argue that it’s a case of AI adding capabilities that many research teams wouldn’t have been able to afford in the first place. Five years ago, for example, transcription was an absolute luxury for me and most of my research partners.)

That automated transcription can also be immensely helpful if, in between days of interviews, you can quickly summarize what participants have said about a given topic, and determine if subsequent inquiries need additional questions, or rewrites of existing ones.

Research Data Analysis

I’ve yet to see a truly useful summary or insight extraction from any LLM-based tool which stands on its own, but most can provide waypoints to help you locate key moments without having to tag them in real time; again, this is especially useful if you can’t have a second person taking notes while you conduct the interview. Care is warranted: Claude, Copilot and ChatGPT will all earnestly deliver made-up quotes, so treat them like well-resourced, eager-to-please, but inexperienced college freshman: tell them to always cite their sources, and check them.

At best, these summaries are usefully incorrect. A mid-level or senior UXR can use LLM output to help them respond to the “blank canvas” problem during their synthesis period. But take nothing at face value.

Overall, I don’t see LLMs (or their putatively termed “Agents”) replacing high quality UX Research work; rather, properly deployed, they can augment and accelerate it, and perhaps let a single Research do their work a bit more accurately or precisely. Inexperienced or mid-level UXRs will still need close supervision, however. This is an exciting – and stressful – time in the field. That warrants cautious optimism and humility, and a willingness to try to techniques.