Due diligence in defining and measuring a device’s usability ideally requires user research methodologies that are quantitative and qualitative. Historically, the beginning of any medical device’s development relies primarily on quantitative requirements and specifications. The results are represented in marketing and engineering analytics found in spreadsheets of numerically confirmable data.

 

The typical source of user inputs is referenced in a literature review or heuristic analysis, and generally documented in a marketing requirements document or an equivalent document. Requirements such as user demographics, or making the device cheaper, faster, and lighter, are critical and enable a device’s incremental evolution. However, they alone do not provide the insights that drive transformative innovation.

 

In user research, not everything studied yields a numerical value. Therefore, one cannot rely on quantitative research methods alone. One should also include qualitative research methodologies as integrated components of design controls and inputs. This article discusses the differences between both research methodologies and focuses on three qualitative methods that round out research integrity—ethnography, projective mapping, and participatory design.

Qualitative research not only helps manufacturers design according to regulatory requirements (i.e., FDA: 21 CFR 820.30, IEC/ISO 62366:2007, ANSI/AAMI HE75:2009, etc.), it also introduces opportunities to optimize ease of use through innovative design. In other words, it transforms the user’s ability level by developing devices that will “deskill” a procedure so that a less-than-expert practitioner can function with the confidence and outcome of an expert. To appreciate how this level of transformative ease-of-use outcome occurs, it is important to clarify the differences between quantitative and qualitative research.

 

Quantitative methods reveal what the user wants but, not always, why the user wants it. Indifference to this method’s objectivity, qualitative user research typically yields subjective data. Moreover, when numerical metrics are available, in a qualitative study, they're considered not statistically significant since what is evaluated is often a limited user sampling.

 

Nevertheless, anthropologists and sociologists agree that even a limited sample size will reveal significant patterns of behavior. People’s aspirations, subconscious behaviors, latent needs, and the physical challenges associated with using a device are not easily measurable with an objective metric.

 

People find creative ways to work around device-usability issues. Characterizing how these individual workarounds mitigate those challenges offeropportunities for innovation. Instead of asking for people’s opinions on how they use a product, qualitative research observes how they use a device in its procedural context—essentially, their behavior while using it and their unique reactions to usage challenges and design deficiencies.

 

Strict quantitative methodology advocates, on the other hand, rely on traditional market research. They solicit key opinion leaders, in-house industry veterans, and practitioners for their design inputs. They possess skill sets honed beyond the average practitioner, with very strong opinions on how a device should work for them (details, features, etc.). However, to rely on experts alone to design a device that will suit only the needs of a very small population of users is considered unacceptable by FDA.

 

FDA states that due diligence, in the process of optimizing ease of use, must relate to all appropriate skill levels when using a device to execute a particular procedure. This skill level includes practitioners considered very good at a procedure, those that are considered average, and those deemed less so.

 

The marketing department’s goal is to provide a product that will satisfy as many different practitioner levels as possible in order to maximize market share. However, as with any research instrument, there are issues with many of the quantitative research methodologies regarding the level and nature of testing bias. A case in point arises in surveys and questionnaires.

 

If clinicians are interviewed about how they would execute a particular procedure or task, there are several reasons why they might exhibit potential biases. Such clinician biases include the following:

 

 

No single research methodology, quantitative or qualitative, consistently delivers absolutes. There is a gap. Nevertheless, by adding a qualitative approach to a quantitative data mix, one will ensure more reliable data, and the biases discussed would be offset. Ethnographic contextual observation and inquiry is the first of three significant qualitative research methodologies to bridge that gap.

Ethnography is widely equated with qualitative user research. It is contextually immersive in analyzing a culture (healthcare, in this case), which has made it synonymous with contextual observation and inquiry. Although applicable to any point of care, the following is an example using the operating room (OR).

 

An ethnographer observing in the OR may see an arthroscopic surgeon take his eyes off the endoscope display and visualize the grip of an instrument’s actuator. This pause interferes with the surgical team’s operatic movements, slowing down the procedure, risking potential missteps by taking focus off the surgical site, and potentially jeopardizing a positive patient outcome. Notably, this usability issue would not have been mentioned in any marketing survey or focus group because the surgeon would’ve compensated for it over many procedures and would no longer be concerned with the design deficiency.

 

This deficiency, however, would not have escaped the trained eyes of the ethnographer whose skills are highly specialized. Ethnographers are adept at knowing how and when to ask the right questions (those that do not lead the respondent), when to be passive and incognito, and when to be interactive and promote conversation.

 

These passive and interactive methods elicit real-time accounting, identify ergonomic challenges, discover latent needs, document usability efficiencies, characterize workflow, and collect design criteria inputs. In most cases, quantifiable data is simultaneously collected regarding time and motion metrics, number of attempts and close calls, number of devices used, and environmental conditions.

 

Ethnographers use a process that is comprehensive and based on a repeatable protocol. They  analyze notes and audiovisual transcripts to identify patterns, workflow bottlenecks, potential use-error modes, and procedural inefficiencies. Their analysis considers the practitioner’s planning process; equipment setup requirements; storage, packaging, and waste issues; instrument usage; environmental circulation issues; facility protocols; and potential enhancements to the overall continuum of care.

 

They develop a procedure map, an important means of documenting user research data. It is developed and continuously updated throughout the project. In addition, it helps visualize, based on important steps defined by the users, the usage scenario—time metrics, instruments and products used, personnel interaction, and emotional state (e.g. anxiety, frustration, pleasure, contentment, etc.).

 

Ethnographers translate all input into documents that will drive a design that can be readily tested, verified, and validated. Ideally, a true multidisciplined team, led by the ethnographer, industrial designer, and human factors engineer, will develop the most effective translation and synthesis of insights into usable device requirements. The more one can qualify and characterize insights into something that a product development team can deploy with confidence, the more successful the device.

 

Currently, not many user research-specific companies can effectively translate and synthesize observational analysis into actionable design criteria. Because they don’t usually have in-house integrated product development, a gap occurs between research inputs (ergonomic, psychological, anthropological, and sociological) and what needs to be filtered and distilled into something that the product development team can act upon.

Projective Mapping is the second step in the process. It starts when a researcher meets one-on-one with the practitioner and user of the device, preferably after observing them execute a procedure. They're taken through a brief exercise to identify the process, providing them with a large sheet of paper, a pad of sticky notes, and a pen.

 

First, they're asked to write down on each sticky note the important moments in the procedure. They're asked to arrange their notes in a linear or cyclical format depending on the nature of the product usage scenario.

 

Second, the respondents are handed a series of graphic images on stickers (i.e., a rope in a knot, racecar, stop sign, tree, turtle, etc.). Most stickers are generic, but some are customized based on the nature of the project. Each is metaphorical in nature and meant to describe the respondents’ actions and emotions.

 

Then they're asked to review the stickers and select one for each step they’ve outlined on the sticky notes. The purpose of the stickers is to ensure that they are using the visual side of their brain rather than regurgitating conventional wisdom. The process stirs up their emotions and gets them to project their true feelings—hence the term projective mapping.

 

Finally, words on a sticker sheet may also be provided but after the visual images are selected. When asked to explain what the image means, the respondents can quickly associate a word with the graphic image. Although neutral reactions occur, if emotions have been truly stirred, most reactions will be projected as negative or positive.

 

Negative projection. A doctor places a sticker of a rope tied in a knot onto the sticky note. Then he adds another sticker with the word “frustrated.” The researcher then asks why he chose the rope tied in a knot. The doctor tells a story of what part of the procedure is frustrating—the product breaks easily during assembly forcing the physician to open package after package of product.

 

Positive projection. The doctor places a sticker with a picture of a racecar on a sticky note. When asked its meaning, his eyes light up and he says this is why he comes to work every day—to execute a technique-intensive procedure that he considers a specialty, his “art.”

 

Participatory design methodologies can result in both generative and evaluative design inputs. It can be used prior to design development to understand design inputs from a high-level systems perspective. Inputs that take place during device definition or in detailed user-interface feature development.

 

Velcro modeling is an example of a generative method. The researcher provides users (participants) with a kit of schematic parts to assemble with technology constraints as to size and shape. They assemble it (generate its design configuration) with given materials.

 

The opposite is true with evaluative methods. Here the researcher gives users something already in development. A functional model that is highly configurable, specifically designed for the user-research study. The study model is more of a measurement instrument than a design prototype. It enables the researcher to adjust and configure the model to the specific needs of the respondent in a simulated contextual environment (e.g., cadaver lab, training facility, etc.). Although the extensive adjustability enables the user to participate in the development of a fixed design architecture, it would be cumbersome and cost prohibitive for a production design.

 

During such an analysis, because there are now concepts to test, research starts to approach design. A pattern of configurations, from one respondent to another yield very specific geometry input for the development of a user interface, a functionally interactive prop. Like a movie prop it too looks real: buttons depress, knobs turn, and cassettes open and close. Only the full plug-and-play functionality is missing. The researcher provides several variations for evaluation and instructs them in its usage as a final qualitative user preference study.

A variety of user research methods is important to maximize ease-of-use in the design of a medical device and to enhance clinical procedures. The marketing department’s contributions of quantitative research (e.g., voice of customer) helps one to hear what the in-house, external authorities and practitioners are saying. But, what will ensure the optimum fidelity from all professional inputs is the integration of qualitative research methodologies: observing the end user and device in action, mapping their behavior, emotions, workarounds, and finally seeking their participation in a rigorous design process. All elements come together to inspire a true transformative design innovation.