Menu
Menu
In IEC 62366-1:2015, the term “formative evaluation” appears twenty-eight times. “Summative evaluation”, on the other hand, is mentioned a total of fifty-four times. That information alone speaks volumes about the importance of summative evaluation.
The video below is a part of our online Introduction to Usability Engineering for Medical Devices and IEC 62366-1 course and you can register by clicking the link.
The standard defines summative evaluation as
“USER INTERFACE EVALUATION conducted at the end of the USER INTERFACE development with the
intent to obtain OBJECTIVE EVIDENCE that the USER INTERFACE can be used safely” (2015, 10).
In other words, it is the final evaluation of your product’s user interface and confirmation that it does not result in an unacceptable risk to people. Essentially, it is the stage where you get to see if your whole usability engineering process was successful.
It is a requirement to include all parts of the user interface that could lead to hazard-related use scenarios and the information for safety in the summative evaluation. Information for safety includes warnings and information on the product and in the accompanying documentation.
If you consider the summative evaluation as the process when you prove your medical device user interface is without unacceptable risk, it becomes clear why the way in which you conduct the summative evaluation is so important. Therefore, one of the crucial things to remember in summative evaluation is that you need to document the evaluation method and come up with a rationale on how the method will provide objective evidence that the user interface of the device is safe for use. The chief part here is the “objective evidence”, so using rating scales or just asking is not enough. Summative evaluation is commonly done by conducting usability tests in simulated use environments.
Furthermore, you need to define the criteria for determining if the information for safety is perceivable, understandable, and supports the correct use of the medical device. This cannot be done without the accompanying documentation and the provision of training during the summative evaluation. In most cases, the accompanying documentation is tested in one single summative evaluation before testing the user interface, just to make sure everything works and is easy to understand.
Even if you require training of future users, you could consider conducting the summative evaluation with users that have not been trained. This is because untrained users would be the worst case, and if you can show that these users are able to use your product without unacceptable risk, this would be even stronger evidence for a safe user interface. But to be honest, that is something that should only be done if the product has been tested thoroughly with formative evaluations during the development and you are sure that there will be no surprises in the summative evaluation.
Speaking of participants, in the EU, no specific number of people is required for the summative evaluation. Instead, the annex K of IEC 62366-2 provides you with information about the sample size calculation. But the FDA in the USA does have a recommended minimum – if you are conducting an summative evaluation for an FDA submission, you will need at least 15 participants.
If you have multiple user groups, you will need at least 30 participants in each user group. If you are working with lay users and large user groups, this number can increase. And again, the FDA requires you to make a sound sample size calculation and provide a rationale of your decision.
During formative evaluations, one of the most favourite methods in usability engineering is ’’think out loud“, to be able to learn as much as possible about the thoughts of the users. But in summative evaluation, this is not appropriate because talking about what they do makes people change their behavior, and do things more carefully, which means that the testing would not achieve the desired validity.
Speaking of real use situations, what you will need to do as well is describe the test environment. In addition to describing, there needs to be a rationale of how the test environment is adequately representative of the actual conditions of use. This goes hand in hand with the fact that your summative evaluation needs to have final production units or equivalent devices. And if you are using production equivalent devices, it is just like in design validation, you will need to justify why it is relevant to use.
In the end, your summative evaluation can be split into multiple parts. But you need to ensure that you tested all the hazard-related use scenarios that you included in your user interface evaluation for the summative evaluation.
An important part of summative evaluation is who should do it. In the standard, it says that people involved in the design of the medical device should not be allowed to conduct the summative evaluation. The FDA requirements say that for human factors validation testing (which is a synonym for summative evaluation) using employees of your own company as test participants is not permitted.
In usability engineering, summative evaluation is the only mandatory part where you need to provide some sort of data as an output. Summative evaluation is often done through usability testing, but the standard allows you to use pre-existing data too. For example, if there is data from an evaluation of a product with the same user interface, you may write a technical rationale where you explain how the data applies to your product, instead of conducting a completely new summative evaluation.
But what happens if the product has parts of the user interface that the previously collected data does not cover? The great news is that you will only have to evaluate those particular parts of the user interface, not the whole thing.
One of the most important aspects of summative evaluations is testing against acceptance criteria. These criteria are the benchmark your medical device needs to reach to be acceptable for market launch. But, what is that? In simple words, the acceptance criteria are based, more often than not, on the probability of occurrence of harm and the severity of the harm. So, you test the risk.
Other criteria exist too. The MDR requires you to minimize the risk as much as possible, and you need to do this without affecting the benefit-risk ratio.
If it was simple, once you have completed your summative evaluation, the data would confirm that the risk is in the acceptable area. However, it is not that simple, for many reasons. One of them is that there are simply not enough participants to prove the numbers that can reach thousands. A way to solve this is to let the user do the same procedure more than once. Or you could use retrospective data, and combine the testing with expert assessment or even use published research as your evidence.
In conclusion, the only goal of summative evaluation is to show that your medical device is safe to use with regards to the user interface.
If you liked this article, you can explore other articles we have published here.
Get instant access to our online Usability Engineering and IEC 62366-1 course right here. In 8 hours, you can learn how to establish a pragmatic usability engineering process that enables regulatory clearance and great product design during this self-paced, interactive online course.
This course is taken by anyone working in product management, such as R&D and QA engineers, product and application specialists, and product owners.
Dr. Michaela Kauer-Franz is a highly esteemed trainer, consultant, and entrepreneur in the medical device industry. She has vast hands-on experience, having developed, amongst other things, monitoring devices for ICU and NICUs, diabetes care products, and women’s health products.
Michaela Kauer-Franz has received numerous awards including the Red Dot Winner Interface Design Award and being nominated for the UX Design Award. She is a lecturer for the design of medical and critical user interfaces at the Technische Universität Darmstadt and part of the national standardization committees in Germany for Usability and User Experience.
Receive FREE templates and quarterly updates on upcoming courses that can help you in your career! Subscribe to our newsletter now.
When you submit this form, you will be sending personal information to medicaldevicehq.com. To comply with GDPR requirements, we need your consent to store and use the personal data you submit. Take a look at our Privacy policy for more details.
Important! Please be advised that there will be scheduled downtime across our platforms from 13:00 CET Apr 26th to no later than 16.00 CET Apr 28th. During this period you will not be able to access the website or your account. For more information, please contact us at support@medicaldevicehq.com
