Risk Assessment- What to do?

Tidge · May 8, 2024

Spoilers: The hazards, if present, always need to be addressed.

I prefer to see a risk control options analysis (at the Hazard Analysis level) that (for each line) identifies:

Which implemented risk controls are responsible for the documented improvement in the acceptability, or
Why, despite the implementation of document risk controls, there is no improvement in the acceptability rating of a line, or
Why no risk controls were implemented for this line of analysis.

For (1) It is common that every potential failure mode gets traced to some line in a Hazard Analysis (usually to get a "Severity" rating), but it is very unlikely that most of the risk controls in all those traced lines actually have the responsibility for improving the risk profile. Do we really believe that implementing inspections for missing nails in horseshoes is the risk control for preventing the collapse of kingdoms?

For (2) It is typical that numeric ratings in risk management files are qualitative and make no effort to be quantitative (see numerous posts by @Bev D )... but often in the case of quantitative assessments it is impossible to collect enough data to demonstrate the magnitude of improvement that would be reflected in an improvement in ratings. For example: it may be completely justifiable that something happens at a ratio no worse than 1-in-1000.... but to construct a study to demonstrate with confidence that the "improved" rate is no worse than 1-in-10000 (a change of "an order of magnitude"), this is typically not the sort of effort that medical device manufacturers invest in as part of improving a single line of analysis in a RMF. I recommend simply saying something like "based on ____, we believe that the acceptability profile improved, but we cannot justify changing the rating value after implementing the controls."

For (3), this is where you blow off implementing controls for the risk and explain why.

stm55 · May 8, 2024

Ed Panek said:
For example, a battery fails and leaks contents. There are several harms that can result. 1) Chemical burns on skin 2) power failure of the device, 3) smudged labels, 4) inhalation of toxic fumes, 5) eye irritation

However smudged labels can result from more causes than just battery leakage. Things like Shipping, handling, and environmental conditions can also cause smudged labels.

This is another great example. Just trying to visualize exactly how this would look on a Hazard Analysis document where you are intended to assign P1 and P2 and severities for each harm. I'm picking up what everyone is putting down, just trying to actually see how this would easily be documented. The PHA templates that I found online and have been referring to seem inadequate to handle these situations

stm55 · May 8, 2024

Tidge said:
Spoilers: The hazards, if present, always need to be addressed.

I prefer to see a risk control options analysis (at the Hazard Analysis level) that (for each line) identifies:

Which implemented risk controls are responsible for the documented improvement in the acceptability, or

Why, despite the implementation of document risk controls, there is no improvement in the acceptability rating of a line, or

Why no risk controls were implemented for this line of analysis.

For (1) It is common that every potential failure mode gets traced to some line in a Hazard Analysis (usually to get a "Severity" rating), but it is very unlikely that most of the risk controls in all those traced lines actually have the responsibility for improving the risk profile. Do we really believe that implementing inspections for missing nails in horseshoes is the risk control for preventing the collapse of kingdoms?

For (2) It is typical that numeric ratings in risk management files are qualitative and make no effort to be quantitative (see numerous posts by @Bev D )... but often in the case of quantitative assessments it is impossible to collect enough data to demonstrate the magnitude of improvement that would be reflected in an improvement in ratings. For example: it may be completely justifiable that something happens at a ratio no worse than 1-in-1000.... but to construct a study to demonstrate with confidence that the "improved" rate is no worse than 1-in-10000 (a change of "an order of magnitude"), this is typically not the sort of effort that medical device manufacturers invest in as part of improving a single line of analysis in a RMF. I recommend simply saying something like "based on ____, we believe that the acceptability profile improved, but we cannot justify changing the rating value after implementing the controls."

For (3), this is where you blow off implementing controls for the risk and explain why.

I believe I am getting your point. Though for #3, I am thinking this is where you might say, this is acceptable per Risk Acceptance Criteria as defined in the RMP? For instance you may have a hazard that has a very low likelihood of leading to a very low severity of harm; surely you might not dedicate resources to mitigating that?

Again, not to continue beating the dead horse, but while I am understanding all that you are saying, I am just unclear how this would look in a Hazard Analysis document. Either I am not understanding how they are typically filled out, or most/all of the examples I have seen are inadequate. Below is another example I found online. You have two separate distinct levels of Harms associated with "Overdelivery"... You would then presumably look at your Severity and your P2 of each to determine what to do next. You are seemingly instead stating that you may have even more than 2 Harms (which might have their own probabilities and severities)-- you and @Ed Panek have given good examples of this. Looking at the harms individually on a Risk table to see what "area" you are in seems to be against the spirit of what you are saying, so I am still left with the question of how to assess the overall hazard of "Overdelivery" in the below table.

Sincere apologies if it seems like I am saying the same thing, but I am just trying to drive at what a hazard table should look like based on your description (which I agree with).

Bev D · May 8, 2024

I think maybe the issue here is that you are trying to accept what we’ve been saying (only the probability of the hazard is important and in that case you use the worst case severity of the resulting harms) into a form that you’ve seen on the internet that does include the probability of the harms. The solution is to not include or to not pay any attention to the probabilities of the harms.

If you control the hazard of overdosing you should get to the point where there is little real chance of severe overdosing and you may have only a small chance of a little overdosing which of course leaves you with a low severity harm (or residual risk)

In this table a lot of actually useful information is hidden by over-generalization…The info that is helpful is the dose of insulin and the distribution of overdosing that must be controlled. Think about that and how you would drive development to control it. That is where your answer lies. Think first abou the process then about the table: form always follows content

stm55 · May 8, 2024

Bev D said:
I think maybe the issue here is that you are trying to accept what we’ve been saying (only the probability of the hazard is important and in that case you use the worst case severity of the resulting harms) into a form that you’ve seen on the internet that does include the probability of the harms. The solution is to not include or to not pay any attention to the probabilities of the harms.

If you control the hazard of overdosing you should get to the point where there is little real chance of severe overdosing and you may have only a small chance of a little overdosing which of course leaves you with a low severity harm (or residual risk)

In this table a lot of actually useful information is hidden by over-generalization…The info that is helpful is the dose of insulin and the distribution of overdosing that must be controlled. Think about that and how you would drive development to control it. That is where your answer lies. Think first abou the process then about the table: form always follows content

Bev, I will do a 180 and now say that you have provided the most helpful comment with regard to me understanding this-- I did not really put together the bold part of your reply until you just said it. What you're saying seems like a reasonable and intuitive way of looking at things. I guess my only concern with blanketly applying those rules would be that you may over-estimate the risk by quite a bit if you just look at a singular "probability for any harm resulting from Hazard" and "Severity of worst case harm"--- ISO24971 gives an example where there's a very small remote chance of death and a much larger probability of patient discomfort (I'm paraphrasing)... If you look at death and multiply by the overall probability, you will get a super high risk that needs to be mitigated. I suppose 1) a situation like this probably does need increased attention, and 2) this attention will give you a convenient way to document a more robust assessment of the risk where you might be able to look at all the different harms and probabilities and rationalize why the risk is acceptable (i.e. rather than using a small cell in a Hazard table).

I am now able to see a path toward proceduralizing this and creating a form that will work

... That said, if you disagree with anything I am saying in this post or it seems I am not getting your point, please let me know!

Ed Panek · May 8, 2024

We also perform an annual risk review for our products. We gather complaint data, repair data, production data, and we also look at the product as a broad category such as FDA MAUDE data.

We do this because once the product goes live you don’t need to guess. There will be some new types of problems you hadn’t thought about or weren’t anticipating. Some problems you really worried about won’t be an issue.

It’s fine to update your risk files for this and frankly expected. This is why FDA QSIT audits always look at CAPA and feedback.

Maybe your corrugated packaging supplier has been focused on because you lots of customers complaining of DOA. Maybe it’s your freight carrier throwing the product around. Maybe you need to add fragile stickers or this side up. Maybe you must ship everything on a pallet.

This form will be reviewed during regulatory review initial release clearance however there’s no way for any regulatory body to determine the precise values for everything. There will be basic safety and essential performance risks for any product but your actual format or template must work for you.

Bev D · May 8, 2024

OK so now you have a real world example of why you cannot multiply severity times occurrence: when you violate the rules of mathematics you get nonsensical answers. It takes more brain power but you need to apply logic to severity and occurrence not math. Now chew on that for awhile.

d_addams · May 16, 2024

stm55 said:
I believe I am getting your point. Though for #3, I am thinking this is where you might say, this is acceptable per Risk Acceptance Criteria as defined in the RMP? For instance you may have a hazard that has a very low likelihood of leading to a very low severity of harm; surely you might not dedicate resources to mitigating that?

Again, not to continue beating the dead horse, but while I am understanding all that you are saying, I am just unclear how this would look in a Hazard Analysis document. Either I am not understanding how they are typically filled out, or most/all of the examples I have seen are inadequate. Below is another example I found online. You have two separate distinct levels of Harms associated with "Overdelivery"... You would then presumably look at your Severity and your P2 of each to determine what to do next. You are seemingly instead stating that you may have even more than 2 Harms (which might have their own probabilities and severities)-- you and @Ed Panek have given good examples of this. Looking at the harms individually on a Risk table to see what "area" you are in seems to be against the spirit of what you are saying, so I am still left with the question of how to assess the overall hazard of "Overdelivery" in the below table.

Sincere apologies if it seems like I am saying the same thing, but I am just trying to drive at what a hazard table should look like based on your description (which I agree with).

View attachment 30208

one of the problems with this chart is you have two different P1s for the Hazard of Overdelivery. The P1 for a Hazard cannot be different. The P1 in this table should have 1 value for Overdelivery. Having different having severity specific P2s is correct.

stm55 · May 16, 2024

d_addams said:
one of the problems with this chart is you have two different P1s for the Hazard of Overdelivery. The P1 for a Hazard cannot be different. The P1 in this table should have 1 value for Overdelivery. Having different having severity specific P2s is correct.

The common definition I've seen for P1 is probability of hazardous situations occuring.. this particular table may not be the best example, but to go back to your other comment on the other thread you just replied to me on-

If your hazard is "loss of sterility", you could have dozens of different hazardous situations that could all correspond to that hazard with their own probabilities? I.e. you could have product incorrectly reused, clinician doesn't use aseptic technique, product not sealed properly, deterioration of seal, etc.. these dont necessarily need to have the same value for P1? They also may or may not have the same P2 (I e. Some may be more likely than others to actually lead to harm).

Based on the other commenters, I was understanding that a conservative and logical approach would be to essentially focus on the overall hazard and assign a PoH for that, rather than individually estimating all different individual hazardous situations (as you said in your other comment, it is probably not realistic to have good data to individually estimate all-- an overall PoH seems like it could make more sense)

Bev D · May 17, 2024

well, if you heard it on the internet it must be right and if it was in a meme then it is gospel. ‘Cuz opinion trumps science and facts every time!

Seriously remember what others have said and remember that risk assessment isn’t a graduate course in probability; it’s a method for making your device safer (at least less hazardous than the disease or injury it is intended to address). To repeat ourselves: The most effective way to do this is really to address the hazard and not the harm.

Risk Assessment- What to do?

Tidge

stm55

Involved In Discussions

stm55

Involved In Discussions

Bev D

Heretical Statistician

stm55

Involved In Discussions

Ed Panek

QA RA Small Med Dev Company

Bev D

Heretical Statistician

d_addams

Involved In Discussions

stm55

Involved In Discussions

Bev D

Heretical Statistician

Similar threads