PPM calculation of customer complaints for PMS

I'm seeking advise on how to calculate ppm levels of customer complaints for PMS in a B2C market. EU medical and non-medical.

Theoretically (normal) the calculation would be #complaints(from_sales(mthX))/#sales(mthX)*1.000.000, monthly figure

What is preventing us from getting this figure is:
- There is a big delay between complaint date and the sales date. This delay is a distribution therefore we can't use one figure for correction.
- Only from a little fraction of the complaints we get the device serial nr to track back to the sales date.

Solutions that have been proposed are:
- PPM calculation by cumulative numbers -> #complaints(CUM)/#sales(CUM)*1.000.000, monthly figure
- PPM calculation by moving annual total -> #complaints(MAT)/#sales(MAT)*1.000.000, monthly figure
- The three calculations above but with the individual complaints corrected for av (sales->complaint) delay to het the amount of complaints approximately at the right month that correlates to the sales

The problem with the solutions is that they all create their own artifacts in graphs either at the start of sales or at the end of sales.
Needles to say the straight forward PPM calculation of complaints per month against sales per month does not give a realistic number due to the delays between sales and complaints.
The delays can be in the range of a few days up to several months.

Does anyone have encountered such a situation and have implemented a solution to come up with a number that will show if complaints are rising or lowering independent of sales?

I would love to hear about it

It seems to me that you're putting a lot of effort into generating a PPM formula. Like you, our business doesn't work with a typical PPM calculation due to the delay between manufacturing, sales and complaints - it can be many months.

I have abandoned calculating PPM's and have redirected my efforts into reducing complaints and meaningful metrics to me, it has worked wonders.

Johnny Quality said:

It seems to me that you're putting a lot of effort into generating a PPM formula. Like you, our business doesn't work with a typical PPM calculation due to the delay between manufacturing, sales and complaints - it can be many months.

I have abandoned calculating PPM's and have redirected my efforts into reducing complaints and meaningful metrics to me, it has worked wonders.

Click to expand...

I agree with Johnny, if it is not a must have KPI I would change for one more focus to the number of claims or response time.
I work on automotive industry and also have moths of delays between the claim and manufacturing but i must have Customer PPM KPI so we monitor them by month but evaluated in quarters.

Johnny and Pau,
Thank you for your reply. I agree with how you look at this.

However it is an obligation for us to do so. We have to monitor the risk to the consumer in the market and compare it with our risk files and if needed update them or act on the risk in the market if needed if the ppm levels become to high compared to the accepted ppm levels in the risk file.
This way of PMS is obligated for medical devices and we do this also for non-medical devices.
Since I work for a big global company these metrics are also made mandatory by the business.
In our company the business groups need to develop this per business group and our business group is relatively young therefore we are looking into this how to calculate this with the challenges we face.

Therefore I'm not looking for different approaches on way of working (but thank you for the feedback) but I'm looking for the best way to calculate ppm levels with the difficulties we're facing, as explained.

Hmmm...

Would it be possible to split your PPM's your PPM's by product line, process or any sort of classification relevant to you? I admit I'm in Automotive like Pau and I'm also in a small business and have no idea what it's like to be in a global company.

Failing that is it at all feasible to discuss your issue with whoever is responsible for dictating you must use PPM's? You mention your business group is young and you are facing a challenge with the system that has been dictated to you.

Hi Johnny it is mandatory for our type of device on the market to monitor risk by ppm.
Therefore business demands this of us, not because they know how but because they know that it is mandatory, if we don't is is also a business risk. Just to say that an alternative way (not monitoring this and work on the complaint causes only) is not accepted

We do have a device which is split into several sub types with all kind of different features. However for each sub type we still have the same problem for calculating the ppm levels.

ppm is not a requirement of any standard. is this a Customer (?) requirement or an 'internal' requirement?

ppm is just a proportion multiplied by a million just like percentage is a proportion multiplied by 100. that's just fun with math.
ppm is often casually used to refer to 'defect rate'. ppm and percentage etc. are just 'units of measure' of the defect rate.
anybody who really understands monitoring risk to the Customer has to understand that the count of defects (numerator) must come from the area of opportunity (denominator). so the misalignment of the sales data in a time period to the reporting of defects in the same time period are NOT aligned and this variation is not indicative of the true variation and defect occurrence that the Customer is experiencing. so you are misstating the 'risk' by definition.

Bev D,
Indeed ppm is not a requirement. Monitoring risk in the market is.

The figure that is used is ppm.
Yes it is a ratio, anyone with a lab and quality background would know that ppm and % are resp. 10^6 and 10^2 multiplication of a fraction in the range 0-1.

Yes defect count per area of opportunity indeed and yes it is miss aligned.
Please read my post carefully because the whole reason I'm asking the question is to deal with the miss alignment due to the nature of the feedback process (only few cust are sharing back device serial nr info).

01mercy said:

Bev D,
Indeed ppm is not a requirement. Monitoring risk in the market is.

The figure that is used is ppm.
Yes it is a ratio, anyone with a lab and quality background would know that ppm and % are ratios.

Yes defect count per area of opportunity indeed and yes it is miss aligned.
Please read my post carefully because the whole reason I'm asking the question is to deal with the miss alignment due to the nature of the feedback process (only few cust are sharing back device serial nr info).

Click to expand...

It doesn't look like a good metric because of, as you pointed out, the long lag time between the actual trouble and the time you become aware of it (customer complaint) plus the variation in the reporting time that you mentioned. Also, if not every customer who is affected complains, the results are not even accurate.

If the customer wants a metric of this nature (ppm per month or per quarter, possibly broken down by product line), go ahead and report it, but I would be more interested in whether customer complaints result in corrective and preventive action (CAPA) that goes to some kind of conclusion to prevent recurrence; any customer complaint can initiate a CAPA (for which I recommend AIAG's CQI-20 Effective Problem Solving process or the similar 8D process, although anything similar will work).

"Top FDA Form 483 Observations for Medical Device Companies in 2015" cites CAPA, complaint files, and nonconforming product as the 2nd, 3rd, and 4th biggest contributors for a total of 31.3%. 9 Top FDA 483 and Warning Letter Problems for Device Companies in 2015 Complaint files constitute 12% of Form 483 observations. I am not familiar with FDA but it looks like ineffective or no CAPA for complaints is the issue.

1. Complaint Files

• CFR Clause 820.198
• QS Subsystem: CAPA
• Number of 483 Observations: 435
• Percent of total 483s: 12%

CFR - Code of Federal Regulations Title 21 requires a CAPA process.
CFR - Code of Federal Regulations Title 21 for complaint files

Again, this is not my area of experience or expertise (FDA and medical devices) but it looks pretty similar to what ISO 9001 and IATF 16949 require; a process for handling complaints that includes CAPA.

01mercy: simply track the number of complaints. You can look at each ‘defect’ separately prioritized by severity. As long as your sales are not increasing or decreasing steeply (by about 15% up or down) the trends will be fairly accurate - at least to signal that investigation is warranted. You could use a simple c chart to do this. Without any way to align the numerator and denominator this is your only option.

Bill is also correct that some failure modes of high severity dont’ need to be trended - their occurence and report is sufficient to alert you to a serious problem. I use the analogy of seeing a mouse in your kitchen. You really don’t need an “accurate” count of how many mice are in your house or even traendign of the ‘sightings’ to know that you need to begin eradication immediately.

By the way I did read your post carefully. My responses are often general in nature to all readers. You would be surprised that many people in the quality profession are not well trained (not their fault) and the whole ppm thing is quite controversial with some.