In every good action movie, there comes a scene in which the hero gets cornered in a seemingly impossible situation — but they escape by being resourceful with items they have at hand. (I’m currently envisioning Jason Bourne defending himself with a rolled up magazine. Or opening a locked door with a fire extinguisher. Or, like, anything Jason Bourne does.) Recently, the Moonshot team found ourselves in a locked-door situation with the opportunity to improvise with tools we had nearby when we quickly adapted in-person design sprint research for our new era of social distancing.
The Original Plan
Our team was undertaking a four-day design sprint with our client to concept, create and test a prototype for a voice-based tool when the spread of COVID-19 forced us to close our office. Normally we would have done our user testing in person, but we had suddenly had a very different landscape to work with.
We still needed to share our voice prototype, facilitate conversation and gain rich insights about users’ priorities and preferences for voice assistant characteristics – but now remotely and in a timely way with the appropriate tools and no additional budget. Fortunately, we have been using tools that facilitate remote collaboration for years, which gave us strong working knowledge and experience. But suddenly mobilizing to use those tools during a design sprint was still going to be a challenge.
First and foremost, we felt we needed to be empathetic to the emotional and mental state of our users. We were looking to carry out research in the first days that Chicago had mandated that its citizens shelter in place and didn’t want to insensitively move forward with business-as-usual. We collaborated with our client, touching base regularly, sharing what we were learning about public opinion and having their team weigh in with other reliable sources. In a couple weeks, we felt that the public was settling into our new reality and that participants would not only able to incorporate some of their current experience, but also compartmentalize appropriately. Once we felt the time was right, we dove in.
Research Preparation
Need: Pre-session Communication
Tool: Email and Visual Directions
Typically, we provide clear and helpful instruction in advance of an interview to set proper expectations, but with additional set-up requirements on our participants’ end and requiring them to use possibly unfamiliar technology, we wanted to be sure we were equipping them for success. We created a detailed Zoom how-to, modifying steps with directions specific to our needs, and offered a test technical session in advance if anyone was feeling particularly nervous (as one participant was.)
Pro Tip: lovable directions with succinct, visual instructions make new learning less intimidating.
Need: Confidentiality Agreement Signatures
Tool: Docusign
Docusign allowed us to send privacy agreements right from Box and handle all paperwork in advance of our session. We have previously done this for remote interviews and diary studies, and it’s kind of nice not to spend the first few minutes of our time together on paperwork. When possible, I think we will explore continuing this practice in the future — even for in-person research.
Pro Tip: include a checklist of requirements/tasks so participant can see and meet clear expectations.
Need: Test Run
Tool: Enlisted a colleague
It’s a best practice to do a trial run of any research plan before you take it into the field so that you can pressure test, troubleshoot, and get the bugs out. Going remote means the possibility of technology issues arising, user error occurring, and a million new ways that research might go sideways. We were able to run a full session and retro — ironing out the kinks, reformatting some materials, making note of sticky spots, and clarifying the team roles needed for the day of the show.
Pro Tip: grab a team member not connected to the project — maybe even who is also a good fit for the study — to uncover even more.
Research Day
Need: Organic conversation and behavior observation
Tool: Zoom Video
Zoom provided an accessible platform for candid video conversation. It was easy for users to access and for me to engage as the facilitator. This approach also allowed our observing team to read facial expressions and gauge other non-verbal responses.
Our research and client teams joined the call 15 minutes early. We quickly reviewed our plan, and everyone changed their names to “Observer,” muting and going off video. As the facilitator, I was on video to welcome participants when they joined.
Pro Tip: create a waiting room so you can control participant entry.
Need: Collaborative Card Sorting Activity
Tool: Mural
Moonshot has long touted the impact good research stimuli can bring to a session and the depth and color they can offer our findings. We had done a card sorting activity in our design sprint and were curious if participants would prioritize experience elements the same or differently. We also wanted to use the activity to drive deeper conversation about expectations, preferences, and priorities. Mural allowed us to bring the same visual experience and conversation to life digitally. I asked participants to make selections and talk through their choices and placements as I manipulated the “cards.”
Pro Tip: In the future, we might share the mural link and have participants manipulate the board themselves (it’s always preferable when they can interact directly), but it felt a little too overwhelming for them this time. One step at a time.
Need: Voice prototype of multiple voice assistant (VA) experiences
Tools: Adobe XD, QuickTime, iTunes and manual operation
This was the big one. As we worked through the logistics of manipulating multiple voice assistant experiences remotely, we considered sound challenges, device accessibility, timing challenges, and the need for control throughout the session.
Our final solution was to use Adobe XD to create the actual prototype and then record only the VA responses with QuickTime. As playback there proved glitchy, we uploaded the recordings into iTunes, which proved much friendlier. I screenshared the script of the user’s side of each interaction on Zoom, and our UX Lead operated the recordings manually to administer and mimic a real-time voice assistant-to-user interaction. (Fun fact: a built in fallback error — “I’m sorry, I didn’t catch that” — felt so real to our participants, they were truly confused what it was that the VA didn’t understand.)
Pro Tip: I referred to our VA operator as “A/V “throughout experience to keep some level of an “automated” feeling without introducing a new person to the interview.
Capture and Synthesis
Need: Session Documentation
Tool: Zoom
We video record our interviews whenever possible to ensure we can revisit conversations, capture exactly what was said, and go back to the “tapes” whenever we need clarity as to not misrepresent. The record feature on Zoom allowed us to retain the capability — although maybe a bit more lo-fi than our usual video set-up.
Pro Tip: set the preferred screen view to participant with the facilitator small in the corner - keeping main view on the participant to capture all of their comments, plus unspoken moments and reactions we don’t want to miss.
Need: Notetaking and Debrief Space
Tool: Mural and Zoom
We typically set up a war room where we all gather to observe the session on simulcast and capture notes on Post-Its to fill our synthesis wall. For the remote version, we created a digital version of our typical Post-It note wall in Mural. Each member of the design and client team claimed a work area to capture their notes on digital Post-Its. At the completion of each session, we would debrief. We talked over Zoom and manipulated our board together in Mural, moving Post-Its, and doing a rough sort as we talked. We also added notes and observations that came up through our discussion. This board then became our synthesis space as we added the final card sorts, screen shots, and other content through our deeper synthesis.
Pro Tip: stay organized with color coding.
Need: Incentive
Tool: Digital gift cards
For this type of project and research, we determined gift cards would be perfectly appropriate. With gift cards, we could send an email with the contents to each participant with a thank you on the same day.
Benefits of Remote Moving Forward
Social distancing is going to be part of our lives for the foreseeable future. As we move forward with remote research, it will be important to keep key considerations in mind, including:
- How to include participants with limited technology access or experience
- How to test product usability within the context of participants’ lives
- How to observe what they actually do (and not just what they say) if we can’t see beyond their screen frame
- How to gain understanding of unexpected influences that we don’t know to ask about because we’re not there to witness
We also need to be mindful of considerations for the sessions themselves, such as:
- Needing to take a little extra time to warm up participants for a video conversation
- Realizing that remote meetings may mean more participant cancellations as easy-to-book can mean easy-to-bail
- Accommodating elements of your own remote location (my dog got really fired up at the neighbors’ dog during one session — a little distracting to say the least)
Remote research does bring some strong positives including alleviating limitations on user location and granting us the ability to go well beyond our backyard to broadly test people nationwide or globally. In addition, remote research lends itself to more flexible scheduling and a more budget-friendly approach and I’m sure there will be even more positive elements that emerge in time.
For now, we know our toolkit will only continue to grow as we adapt and adjust to broader needs. What tricks and tools are you using as you maneuver remote research? We would love to hear from you.
