How to moderate usability testing with simultaneous translation

In the decades that I’ve done user research, nearly all of my research has been conducted in English. While I’ve done some research in other countries and with products in other languages, either the product itself was still an English-language product or, when I didn’t think it would harm the research effort, I got bilingual participants who could review, for example, a website in Spanish and then interact with me in English.

So when my US-based client approached me and asked me to conduct a research study in Brazil, fully in Portuguese, I was excited – but admittedly a bit nervous – to have the opportunity to conduct user research with an interpreter. In the end, everything went really well and the study was a success!

Here are some of my lessons learned.

Deciding if you want simultaneous translation

My client had first developed their app in English and I had already done usability testing of their app in two cities in 2017. Now that they had it translated and localized, they wanted to be sure that it was as usable for a Brazilian audience in Portuguese as it was for a US-based audience in English.

Before deciding to definitely use an interpreter, we talked about other options available to them.

In-Person testing without leaving the country

When talking with a participant recruiter about the options, the recruiter felt comfortable that there were certain cities in the United States that had a large enough Brazilian Portuguese population. On the one hand, this would save significant money without the need for international travel. While I felt like it was important to let the client know that this was a cost saving option, I was also happy when they agreed that there could be something inherently different about participants who live in Brazil vs. those who are now in the United States. For example, those in the United States might understand American cultural references that someone who lived in Brazil might be confused by.

International testing with bilingual participants

I also asked the client if they wanted a recruiter to try to find bilingual participants for ease of understanding the sessions – both for me as a moderator and for them as observers. This was also a comfortable type of research for me, as I’d done it before. While they considered this possibility, we eventually decided that those who are bilingual may also not be representative of the actual users of the app. For example, they may tip towards higher education or again have more cultural knowledge from outside of Brazil.

Finding a service

If you’re going to do simultaneous translation, it’s important to find an interpreter who has experience doing simultaneous translation of qualitative research studies, as this kind of translation is substantively different than, for example, one-way translation of a foreign speaker addressing an audience at a conference.  And it’s certainly different than someone translating a conversation and writing a transcript after the fact.

Check with participant recruiters

My first stop was to check with a participant recruiter that I had used previously. This recruiter offers international recruiting for qualitative research by finding partners in cities around the world. Within a few days, they had located a recruitment service in Brazil that also offered the option of an interpreter.

Check with the client

While my client was based in the United States, I learned that simultaneous to my efforts, they had spoken with their counterparts in a Brazilian office and had gotten the name of a trusted interpreter who had translated some research studies for them previously. So we decided to go ahead and use this company to translate the sessions.

Preparing for the research

Send materials in advance

Once the contract was signed, I sent the interpretation company a copy of the test script and a link to the app that we were going to test. While the test script wasn’t followed verbatim, it still helped to give the interpreters a head start in understanding the flow of the sessions.

There were slated to be two interpreters over the three days of research, so we planned for each of the interpreters to come 30 minutes prior to the start of the first session that they’d be translating in order for them to discuss any questions they might have.

Plan for more time

While the prior sessions in the United States had been allotted 60 minutes each – timing that worked perfectly for what we wanted to cover – these sessions with a similar script were allotted 90 minutes each to allow for extra time for the translation process.

Over the course of three days, we had the recruiter schedule four sessions a day, with a fifth backup session on the second and third days.

I made sure to set the interpreters’ expectations appropriately by providing timing for the twelve original sessions but also with a caveat that if there was a no-show or two, we’d be adding one or two backup session slots.

During the Research

Experiment and modulate speed and pauses

A good translator has a memory cache and can translate a sentence while the next sentence is still being spoken. On the one hand, this keeps sessions running quickly and avoids unnecessary pauses. On the other hand, as a moderator there is a greater need to focus on the English, and both the English and Portuguese were being spoken simultaneously. The first day involved figuring out how fast to speak and when to pause in a way that worked most naturally.

While some of this achievement of balance was really through unspoken experimentation with the interpreter, there were also discussion points, like how exact the translation should be. I found that my preference was not an exact translation. Participants’ stutters and stumbles didn’t need to be translated exactly and when I spoke to the participants, if what I said would have a different meaning when translated exactly, it would be better for the interpreter to adjust the translation as best she could to have the most similar meaning.

Positioning matters

Testing was conducted in a conference room at the Brazilian office of the client. In this room, I positioned everything at a rectangular table so the interpreter sat caddy-corner to me and the participant sat on my other side. That meant that I could look at the participant and the participant could look in one direction and see both me and the interpreter.

I did my best to look at the participant as much as possible and for the most part, avoided looking back at the interpreter. For participants, I didn’t give them any instructions on where to look as I preferred to do whatever was most comfortable for the participants. Some looked at me, some looked at both me and the interpreter and occasional participants seemed to find it more comfortable to look only at the interpreter. To some extent, I believe that this depended on how difficult it was for participants to deal with simultaneous translation and whether it was most helpful for them to see the interpreter actually speaking the words.

Some bilingual knowledge could make things more difficult

English fluency is not very common in Brazil, at least among the core userbase of this app. However, participants came in with varying levels of English language knowledge. Brazilians with more knowledge of English seemed to have a little bit harder time with the interpreter. This was likely due to the fact that they could understand some of the English words and were thus processing some of what I said and then the translation a few moments later. In one case, we found that the participant who had the strongest command of English had an easier time listening to me without any translation, although she then responded to me in Portuguese, which was then translated back into English.

It’s amazing how much communication happens through body language

One thing that makes me a good moderator, and why I very much prefer in-person sessions over remote sessions, is that I interact with participants not only on a verbal level but on a non-verbal level as well. But until this experience, I never quite recognized just how rich a data stream there is from body language.

With only a slight understanding of Portuguese, mainly from similarities in some words to English or Spanish (which I took in high school, 26 years ago!), I realized that in many cases, I asked a probing question, and just by watching the participant and the way they responded, I knew the answer before the translator had an opportunity to translate. Even though I was confident that I knew how they answered, I was wary of messing up the communication flow so I tried not to then ask follow-up questions or move on before the translator had the opportunity to finish translating for me. Overall, I found that the biggest variable in terms of how much this happened was how much a participant naturally communicated on a non-verbal level.

It’s harder to mirror body language when communication is slightly unsynchronized

Speaking of body language, I know that my natural instinct is to mirror body language in whatever way makes sense. While it’s not fully conscious, I know, for example, that if I’m talking with someone and they smile at something that they are saying, I’ll reflexively smile back. However, things get more complicated when a participant would respond to something I said and smile. If I didn’t understand what their answer was nonverbally, I’d have to wait for the interpreter to catch up and finish translating before I knew what they said. What is usually unconscious thus became conscious and I found myself pausing when a participant would smile and then wondering whether a smile would make sense before I knew what was said.

Consider your microphone position

In order to manage the project budget, Edie was logging the sessions remotely from Maryland. Simultaneously, a number of client observers were observing from their US-based office. And in addition, several bilingual staff were observing from another nearby conference room. Streaming the session via a Hangout was easy, but microphone positioning was a bit more difficult. We ended up positioning the microphone closest to the interpreter.

This was best for the observers who did not speak any Portuguese, although I think that it was a bit of a negative for those in-person observers in the adjacent conference room who wanted to be able to hear the spoken Portuguese more loudly.

Determine how to handle text on the screen

While the conversation was translated, the text within the app was not. I was pretty familiar with the original English-language version of the app. That, combined with some identification of words from my background English and sometimes Spanish knowledge allowed me to feel reasonably comfortable following along with each participants’ chosen pathway. However, there were some open-ended exploratory tasks that allowed participants to make a choice and go down pathways that I may not have seen before.

When I truly couldn’t follow, I found that I had one of two choices. The first was to say to the participant something like “Explain to me where you are now.” As long as I did that judiciously, it helped me to understand both where the participant actually was in the app and, simultaneously combined with what I could understand, where they expected to be. For the most part, I found that I was able to feel pretty comfortable that my continued understanding was accurate.

My second option was to actually turn around and say to the translator “What does this page say?” I opted to do this less frequently as I felt like it disrupted the flow of communication between myself and the participant.

Avoid lengthy days

Even when everything goes very smoothly and communication is exclusively in English, I find that my preference is to limit my time to about 6 hours of actual moderation a day. Sure there are times when I’ve had unavoidable marathon moderating days that last longer, but keeping to a limit of about 6 hours of moderation time keeps me sharp and makes me a better moderator.

While this is true with English-only, all the more so it’s true with sessions that involve simultaneous translation. I felt like my laptop when I give it heavy-duty tasks, like video rendering, and very quickly I hear the fan come on as it chugs through what it needs to do. Sessions with simultaneous translation take more effort, energy and brain power, and I’m grateful that we didn’t utilize the fifth-session floater participant on either of the final two days.

One positive I found is that the first day was the most difficult as I learned to became comfortable with the simultaneous translation. The second day was a little easier, and the third day was the easiest, by which time I had the clearest sense of where to pause and how to best communicate with simultaneous translation.

Ready to do it again!

My next three usability studies that will take place in the next few weeks are all in English, and I look forward to a few weeks in what is comfortably familiar. However, I’m also looking forward to a future opportunity to work with simultaneous translation in another language. While this blog post represents an n of 1, it will be intriguing to compare and contrast the experience with that of translation in another language.

Pencil Drawing of usability test session: Eliana Lebson; Background flags of Brazil and US: Kagenmi/Bigstockphoto.com

2018-01-20T20:24:46+00:00Jan 20, 2018|Tags: , |