Safety-Enhanced Design: Summative Usability Evaluations
User-Centered Design Makes for Effective, Efficient, and Satisfied Users
EHR vendors seeking ONC 2014 Edition (Meaningful Use Certification) or ONC 2015 Edition Certification (MACRA, MIPS) need to follow the requirements for Electronic Health Records (EHRs) issued by The Office of the National Coordinator for Health Information Technology (ONC). Every vendor must include proof of user-centered design and user testing results for the Medical IT user experience from their EHR computer interface in their certification submission. Vendors are also required to follow a formal User Centered Design (UCD) process and perform summative usability testing i.e., Safety-Enhanced Design, on specific product areas.
We recommend that EHR vendors follow the ISO 9241-11 standard. ISO-9241 Part 11: (1998) pertains to the extent to which a product can be used by specified users to achieve specified goals with Effectiveness (Task completion by users), Efficiency (Task on time) and Satisfaction (responded by user in term of experience) in a specified context of use (users, tasks, equipment & environments).
Summative usability testing is an industry standard usability methodology (see http://bit.ly/13WYqTU ) During the testing a trained user experience professional works with the EHR vendor to create a series of representative tasks and then measures the amount of time it takes to complete the task (time on task), how many and what type of errors occur (error rates), and satisfaction with the task interaction (user satisfaction). The results — as measured by things like task times, assisted and un-assisted completion rates and standardized satisfaction scores — can be benchmarked against current industry standards.
In a recent public comment regarding the 2014 Meaningful Use certification process, The Drummond Group (an ONC ATCB) stated that "from our experience .. the usability test effort is not a small undertaking for EHR vendors. In fact, it was essentially one of the most difficult criteria to complete.”
EHR developers can and should perform many iterations of user testing for the Medical IT user experience. These early iterations are often called Formative testing. We also recommend that an Expert (Heuristic) review be performed before any of the more formal EHR usability testing.
This is a cost-saving measure that can significantly reduce the number of iterations required and ultimately reduce the number of “items needed improvement” presented in the ONC submission. The Public submission that is ultimately provided for testing and certification may be the expression of a final iteration in which few areas for improvement would be identified. See Please don’t air our dirty laundry!
How should the Results Presented?
Results from the summative study should be created and presented using the Customized Common Industry Format (CCIF) Template for EHR Usability Testing (NISTIR 7742 see http://1.usa.gov/gAZFDt ) These reports typically include an Executive Summary, an introduction, a method section, discussion of the results and any appendices .
The following are required for submission to the ONC for the 2014 Edition (Stage 2 Meaningful Use):
• Name and version of the product
• Date and location of the test
• Test environment
• Description of the intended users
• Total number of participants
• Description of participants: their experience and demographic characteristics
• Description of the user tasks that were tested
• List of the specific metrics captured during the testing for effectiveness, efficiency and satisfaction
• Data scoring
• Results of the test and data analysis
• Major test findings
• Identified area(s) of improvement(s)
For more information on the ONC Test Procedure for Stage 2 Meaningful Use §170.314(g)(3) Safety-enhanced design see http://bit.ly/YLwrPO