What is your favorite Usability Risk Analysis tool?

Good Afternoon,

I was wondering what everyone's favorite risk tool, or what risk tool your company uses, for assessing usability failures. Every company I have worked for pushes the uFMEA, which does not technically cover normal use failure modes.

It becomes particularly muddy, in regards to the IVD industry, since tests typically have a method associate with them, which is technically a process that a user executes, so do you perform process FMEA or use FMEA or even design FMEA since the method is technically part of the test design?

In my past, I have used uFMEAs formatted with columns to provide input to the usability process. I have also performed process FMEAs with "use error" as a cause. My workaround for normal use vs single fault conditions is to put "normal use" as a failure cause. All of these were bandaid fixes to fit into the quality systems I was hired into. I now have the opportunity to start fresh and would like to hear what other people have used for usability in their past and present or even what you believe could be the next future state of usability risk assessments.

We use UFMEAs. I'm having trouble understanding what a "normal use failure mode" would be. Can you provide some examples?

Usability informs two fundamental aspects of my risk analysis.


Notice that this is explicitly tying the product I intend to make with the circumstances of what it will be prescribed for. The same hazards may present different risk ratings depending on use scenarios. Failure modes are not considered except in the context of how they might inform the (pre-control) ratings (and individual lines of analysis).

Severity ratings for individual UFMEA lines of analysis derive from (and are linked to) identified lines in the Hazard Analysis. This is very often a many-to-many relationship, as the same hazards can present themselves in a varity of use scenarios. Within the UFMEA is where we bluntly consider failure modes (informed by mental models, user physiological factors, skill level, etc.) and consider the implementation of risk controls.

There are some subtleties with this approach. Allow me to try to describe one...

One is that in a "Verification and Validation" development approach, the risk controls for failure modes are initially subjected to implementation (identified by verification of implementation, VoI) and then assessed for effectiveness (identified by verification of effectiveness, VoE). It is common that within a DFMEA that the VoE is simply some single test (possibly challenging boundaries, but often not) during what is called design verification. Within a PFMEA, it is more likely that (with the help of a good process engineer) that there could be an actual process validation to serve as VoE for risk controls.

When it comes to Usability, just about every professionally worth their salt will jump to "User Validation" and start thinking about efforts in terms of "62366" (or "HE75" or some such) and effectively move themselves into a headspace where they may forget that the UFMEA is a subordinate document to the Hazard Analysis and so it needs attention prior to user validation (which is supposed to have summative testing on "finished designs"). This pre-validation attention ought to be motivating the final user validation... and I think an argument can be made that by referencing appropriate formative testing of a design (and documenting such in a UFMEA) it is possible that the UFMEA risk ratings could be low enough that certain possible failure modes wouldn't have to be included in user validation.

It is (in my experience) most appropriate to close the loop on Usability Risks by appropriately referencing the summative testing as VoE for final UFMEA risk ratings. My own preference is that the top level risk document (Hazard Analysis) NOT refer directly to analysis in UFMEA, except via that initial link to the use cases. I definitely think the link is there, it's just my feeling that it is silly to pretend that the design team has any measure of control over the users. A personal, bold statement follows: The design team is obligated to understand how the device is used, summative testing is pretty much just a confirmatory analysis that there are not elements of risk that were unaccounted (or improperly accounted) for.

Normal use hazards would be a better way to word it since these is no fault occurring. It is the hazards that arrive from using your device as intended, so no fault conditions. An FMEA assesses single fault failures.

It is one of the gaps between an FMEA and 14971 risk analysis.

I need a sample or example of real usability engineering file please. I've read the related standards and now I think it is better to study a real case. I know that these would be confidential documents of each organization, but some withdrawn ones may be available.

-PFMEA , DFMEA, poka yoke/error proof , MSA , fault tree & fishbone ,--etc, -- all could be tools of technical risk assessment & preventive measures
-The hazards could be approach occupational risk assessment issues rather than technical assessment of the processes & products ;
- The other business risk assessment should be considered also ; i.e.: COVID 19 & political issues as Ukraine could be issue risks on supply chain , orders , financial issues ; IT risks , lack of shipments & markets risks ,--etc.