Feature Article

Test Protocols: Accelerate the Process and Keep Customers Happy

Published: August 11, 2011
By: Daniel Santos
Find more content on:

Efficient test protocols remove excess cost and complexity from the product design equation.

Creating effective test protocols can pose many challenges for medical device design engineers. It’s critical to establish test protocols that verify a device will operate safely and reliably from its first through 1,000th use. However, in the competitive world of medical devices, speed to market is important. Test protocols must consider a device’s entire life cycle in the shortest amount of time possible.

Determining what type of tests need to be performed to accelerate design evaluation, analysis, and verification can make a design engineer’s head spin. Over-thinking the parameters can delay product release and result in added cost for the customer. Shortcutting certain tests can lead to failures late in the process, requiring the team to start over at the beginning—or worse, cause failures in the field.
 
However, creating test protocols that verify and validate that a device will be designed and manufactured to meet a customer’s specification and performance requirements doesn’t have to be as complex and costly as it may seem.
 
This article discusses tips and insights that can help design teams avoid unnecessary testing, shorten the development process, prevent nonessential spending, and, most importantly, increase customer satisfaction by meeting customer needs.
 
Customer and Crossfunctional Collaboration
How a medical device company applies its process to designing, developing, and manufacturing an orthopaedic device greatly affects the accuracy and effectiveness of test protocols.
 
If a firm’s process is to assemble key people—especially those involved in testing—in the same room at a kick-off meeting with a customer, important variables that affect the design should be explained.
 
For example, once it’s clear where the product will be released (i.e., the United States versus Europe), the regulatory affairs and quality assurance (RA/QA) team can offer insight on relevant sterilization parameters. The procurement group might have unique knowledge about what materials are available, and the manufacturing team can give input as to whether the components, dimensions, and materials being considered will withstand high production levels. Without the early collaboration between all involved departments, it is difficult for a design engineer to create expeditious and comprehensive test protocols that account for these diverse variables.
 
Siloed processes in which phases of the project are handed off from one department to another (i.e., business development to design engineer to RA/QA) tend to limit the scope and effectiveness of test protocols. For example, if design engineers don’t know that the orthopaedic device will be released in Europe, they aren’t likely to take higher pH levels into account during the design or test for that parameter. As a result, the device will fail in the field or during clinical trials, and design engineers will have to start from the beginning of the process. Timelines are pushed back, costs increase, and customers are left unhappy.
 
Had the QA/RA group been present during the design and testing process, they would have provided important insight to the engineers about the types of sterilizations required or performed in Europe. The result would have been more upfront testing and nominal design issues later in the project. Early communication, comprehension, and collaboration between all departments about what the customer wants can significantly reduce cost, time delays, redundancies, and failures in testing, as well as customer frustrations.
 
Understand Application and Environment
In addition to bringing crossfunctional team members together at the very beginning of a project, understanding how an orthopaedic device will be used in all aspects of the surgical procedure also greatly influences what test protocols are established.
 
If it is unclear to a design engineer how a device will be operated and handled, many important parameters will be overlooked in the testing process. For instance, if limited knowledge leads a designer to think that the device will be turned on and off and then placed directly into the autoclave for sterilization, test protocols are more likely to reflect only those limited parameters—especially if timelines are tight.
 
However, if design engineers continuously communicate with the customer—from the beginning of the project and onward—they may learn that, in reality, the device is exposed to a much harsher cleaning process. It might be that, after the device is turned off, it’s then handed to a nurse or surgical technician who rinses it with water, scrubs it with a brush, dips it into a cleaning chemical, and then puts it into the autoclave. An early awareness of this cleaning process enables design engineers to create more rigorous test protocols that accurately verify and validate safety factors. Customers should provide clarity about how the orthopaedic device will be handled and any potential misuses that can result in failure, because engineers can’t always decipher such factors on their own.
 
Align Customer Wants and Needs
More often than not, customers come to medical device manufacturers with a vague idea of what they want and need from their orthopaedic device. They may need their device to contain a motor with 60 in.-oz of torque, but they want it to be the size of a highlighter. Because test protocols are only as good as the customer specifications, it’s fundamental that the design team is on the same page with what the customer wants and actually needs. Without clear and detailed specifications, protocols can miss the mark for performance, safety, functionality, and reliability testing.
 
To ensure customer needs are established before the design and test protocols are developed, an effective strategy creates a functional prototype while still in the feasibility stage. A functional prototype is a bare bones, primitive model of what the customer is requesting. If the manufacturer has the capability, it can create this mock-up in-house. If not, leveraging the resources of the production floor or an outside prototype house is worth the effort, because this preliminary model ultimately saves time and money.
 
Going back to the motor example, a design engineer may intrinsically know that the customer’s request for a highlighter-sized orthopaedic device to generate 60 in.-oz of torque isn’t feasible and may be overkill for what they really need. However, providing the customer with a rudimentary sample often does a better job of explaining why alternatives must be considered. If the customer can see and feel that to meet their power requirement the device must be the circumference of a baseball, they’ll more likely be open to compromises between what is wanted and what is needed. The sooner the customer and manufacturer can nail down feasible product parameters, the faster the engineer can design the device and focus on testing variables.
 
Understand Variables Associated with Design of Experiment 
The process of solving a Rubik’s cube puzzle is a good example of DOE. When you move one corner color on a Rubik’s cube to align it with another color on the cube, multiple colors on the cube are being shifted in the process. To align all the colors (i.e., solving the puzzle), you must factor in how multiple variables (changes in the color positioning, movement, which face the color ends up on, and opposing faces) interact and affect one another along the way.
Design of experiment (DOE) is a statistical tool for creating effective and efficient test protocols. It takes into account multiple variables simultaneously before test protocols are actually written and carried out.
 
The article “Application of Design of Experiment (DOE) Techniques to Process Validation in Medical Device Manufacture” explains the importance of using DOE during process validation.
 
In DOE all the factors of interest can be investigated in a single trial, minimizing the size of the experimental schedule required and providing information on key process interactions...This latter aspect is of importance since optimal conditions may occur when one factor is high and another low. This type of information on interactions between factors cannot be easily obtained by investigating the effect of each factor separately.
 
A DOE approach permits efficient use of resources (personnel time, machine time, materials, etc.), provides detailed analysis, gives information on reproducibility and errors, and provides a predictive capability...Applying DOE reduces the size and hence the cost of process validation trials. It is a regulatory requirement to run sufficient trials to demonstrate the statistical significance of results, and DOE can assist in this procedural aspect.1
 
Device and process specifications must be established before a piece of equipment can be validated, because this information is used to write the protocol.2
 
 
Table I. These are not actual design values.The high and low limits for temperature, time, and pressure of the autoclave are hypothetical examples for these ranges. The high and low levels of each factor selected would be based on knowledge of the particular system being tested.
Table I illustrates a possible example of DOE process limits for how a sealant in an orthopaedic bone drill might hold up in an autoclave. This DOE process gives a design engineer a holistic understanding of how multiple variables interact with each another in an autoclave environment. Evaluating multiple variables at the same time versus evaluating one variable at a time while all others are held constant reduces the number of iterations required, which accelerates the design and development process.
 
Balance Functional Testing and Sterilization Testing
There’s an art and a science to making sure you’ve adequately tested a device’s functionality and its ability to hold up to harsh sterilization environments. Conducting sterilization tests between every functional test takes time, and in the medical device industry, manufacturers are pressured by time constraints. In order to accelerate the process, many testers separate functional and sterilization testing methods, which, unfortunately, can disregard many important variables.
 
For example, some testers might verify that an orthopaedic hand piece meets autoclave testing, but they won’t run its duty cycle. Sterilizing the device hundreds of times without running it doesn’t use the functionality of the hand piece. The seals are essentially being tested over and over again without wearing out, which is an unlikely scenario in the field.
 
Or, they may test functionality of the device by running it 500 times to ensure that the motor, seals, and internal components are working properly. But if they aren’t running the device through an autoclave, it’s not being exposed to moisture as it would in a real-world application.
 
In fairness, there is no one easy answer to finding a balance between these two important testing methods. Experience conducting other tests and verifying past failures helps because there are some similarities in how certain variables in each hand piece interact. However, if historic data or previous experience is lacking, two important considerations are to:
  • Clearly understand how the device will be used in the field.
  • Conduct component level testing and subsystem testing early.
As noted previously, understanding the operation and handling of an orthopaedic device makes it easier to set up test protocols that replicate actual field usage. You can run a motor all day without stopping until it burns up (fails), but if that’s not how it will be used in the field, you’re wasting valuable testing time and causing unrealistic failures. However, if you conduct a test that closely replicates field usage, verifies that the device lasts through all testing cycles, and has no unintended consequences, you’ve determined a good testing protocol framework.
 
In addition to experience and comprehending the parameters set by the customer, component and subsystem-level testing are crucial. These tests should be started early in the development phases—long before a completed product. Component-level testing analyzes individual aspects of the device. It can involve testing a shaft to verify that it meets strength requirements, testing the gear teeth to ensure they can withstand torque load, or analyzing material properties to assess how it will stand up to high pH levels.
 
After verifying that individual components will survive real-world use, subsystem testing can begin. Subsystem tests bring different components together to evaluate how they interact with one another. Using a motor as an example, you would test the motor separately before putting it into a system to determine its limits and ensuring that it functions properly. Testing the motor first reduces unintended consequences that could make the entire hand piece fail in later testing phases. Once the motor is tested using realistic testing parameters, add another variable—the cable—to see how the two interact. If these two variables interact successfully, the next step would be to add other hand piece components or subsystems to the test. Since you’ve determined that the motor and cable tested well together, if there’s an issue or a failure, your subsystem tests verify that it must be an interaction between the motor, cable, and the other selected component or subsystem.
 
Subsystem testing is a key part of product design. Getting each subsystem tested before it goes to a final system to verify that it can handle the rigors of the field eliminates much of the variable testing later on, further accelerating reliability testing.

 

Recreate Realistic Usage Environments In-House

Another way to accelerate reliability testing is to create a real-world, real-time testing environment in-house. This method provides more control over what and how a device is being tested. Testers have the ability to see how the orthopaedic device is tested, hear how it sounds, and experience how different variables are interacting.
 
This in-house environment might include dishwashers for pH testing, machines for autoclaving, a simulated cadaver lab to test bone density, and a prototype shop to quickly create functional prototypes. If a design engineer doesn’t have adequate in-house capabilities to test components and subsystems, it’s difficult to recreate realistic applications. They are left relying on the material specifications—hoping they are correct—or farming out tests to verify failures, but gaining little understanding for why they occurred.
 
In addition to keeping the whole testing process under one roof, developing proprietary testing equipment specific to the orthopaedic medical device helps create an authentic testing environment. Testing products with equipment that isn’t specific to the device doesn’t always provide accurate results for how it will hold up in the field. For example, using an oven to replicate the autoclave temperature thermal cycling may seem adequate to reduce testing time. In actuality, the lack of steam and pressure cycling eliminate crucial variables in testing the product, leaving results that may not replicate field use.
 
Proprietary equipment can also save a lot of time and produce quicker results than manual testing. For example, creating an automated panel system to test multiple orthopaedic surgical drills eliminates the need for an operator and accelerates a manufacturer’s ability to test more than one unit at a time. Developing proprietary testing equipment to mimic operative handling raises the bar on verifying and validating that the design meets customer specifications.
 
Conclusion
There’s no one-size-fits-all approach to creating effective, proprietary test protocols, especially when it comes to accelerating reliability testing. However, there are fundamentals that, if overlooked, will delay processes. The most important starting point is to ensure that all individuals involved in testing or testing-related activities are on the same page in understanding the customer’s parameters. A transparent understanding of their specifications and the device’s intended field use makes developing accurate, legitimate test protocols an intriguing process of discovery as opposed to one of uncertainty and adversity.
 
References
1. D Dixon et al., “Application of Design of Experiment (DOE) Techniques to Process Validation in Medical Device Manufacture,” Journal of Validation Technology 12, No 2 (2006).
2. GHTF/SG3/N99-10:2004 Edition 2, (PDF) “Quality Management Systems—Process Validation Guidance.”
 
Daniel Santos is engineering manager at Pro-Dex Inc. (Irvine, CA).

Published in Orthotec, September/October, Volume 2, No. 4


Daniel Santos
Featured Imagef: 
Your rating: None Average: 5 (5 votes)

Login or register to post comments