Multimodal Search
I led ethnographic and strategic research for multimodal search experiences, uncovering how users leverage the camera to ask questions about their physical environment— enabling people to “search what they see” through live video, images, and suggestions.
Featured press: The Keyword Search, Wired
The goal
Identify opportunities to make camera-based searching more natural and powerful by understanding nascent mental models and product expectations. Enable users to ask more complex questions and get richer, more actionable answers.
The approach
Understand mental models:
Led multiple focus groups to unpack how users think and talk about multimodal search. Facilitated live demos and stimulus-based discussion to explore perceived value and concerns (e.g. AI misuse, privacy, cognitive decline), shaping early product framing and guardrails.
Understanding real-world behaviors
Diary study to capture use of multimodal search features in-the-wild, surfacing everyday moments where users wanted richer answers around hard-to-describe, visually rich contexts. These efforts helped identify organic, high-value use cases and patterns of unmet need.
Contextual Inquiry
Visited participants’ homes to observe visual search in action. Explored the role of camera and AI across complex, real-life workflows, revealing integration barriers, workarounds, and design opportunities rooted in daily life.
Iterative Usability Testing
Ran multiple lab studies with hands-on, task-based scenarios to test usability, comprehension, and ergonomic flow, exposing gaps in AI feedback quality and highlighting the need for greater contextual understanding in visual input.
IMMERSIVE RESEARCH
Exploratory research in simulated environments enabled us to hone in on the capabilities and quality our multimodal search feature needed in order to demonstrate real value to users, while home visits and diary studies provided real contextual insights that enabled our team to address experience gaps and launch the product with confidence.
Hands-on demos, observation of natural behaviors, and deep-dive interviews in participants’ home were key to understanding their daily lives and physical environment. With visual and vocal input, I explored what kinds of searches are important for users and where our quality bar needs to be to best serve them:
What’s this bug infesting my garden and how do I get rid of it? 🐛 🌿
🪵 🪑 What can I craft with this block of wood and where do I start?
My puppy is acting weird suddenly, what’s wrong with her? 🐶 🐾