Complaints time series significant change

Dear all,

I'm in search for information that describes possible statistical approaches to complaint numbers over time, in order to identify when change is significant or falls in normal variance.
The challenge I see is how to deal with up and downward trends. The techniques I've seen so far are based on the assumption of a stable process.
However complaints rise and drop based on changes in the product or processes that influence the product generating possible more or less complaints.
How to deal with this so that it can be set up in a standard way of monitoring.

Thanks.

There are several approaches to this based on the type of product and what you know about your active use base.

The simplest approach is to just continually work on improving the top complaint rates - if you make the complaints go down you don't have to worry about stability.

a few questions:
What do you know about your products in the field?
Do your products have serial numbers or part versions that you can get when there is a complaint?
Do you know when product are manufactured and can you trace the manufacture date for complaints?
Do you know the active user base?

Hi @Bev D
Thanks for your reply.

The product is a SAAS product and though it has a version, it is considered to be 1 version in the field over multiple instances for different customers.
The customer base does not represent the number of users and the number of users is not exactly known due to privacy regulation so we have this as an estimate. We do not have the frequency of use over the users.

For each complaint I have the type of SAAS product, there are a few, the instance and for the product the total estimate of users (user base).
For now I only have the number of complaints per day which are reported at the end of month as a total for that month (incoming, closed and open).
The complaint rates are taken as the number of complaint divided by the user base as a percentage.

The request I had is to identify in my monthly overviews if there is a trend, or outlier not only by means of visual interpretation of the graphs but by statistical significance. "is the change we see statistical significant or not"

So far my thoughts were
- check the data for seasonal effects by means of ACF
- check if I want to use monthly or weekly or daily data, drill down will give more noise but to high lvl will give not enough to work with
- check how much trend-shift is seen over the data, does the data follow normal distribution
- check how I can visualize any trend shift or outlier more clearly (i.e. taking the delta's and plot them)
- check what statistical techniques I can use and apply to the monitoring to make the interpretation of significant change objective

Please spell out abbreviations on first use. I know that SAAS is Software as a Service, but I'm stumped on ACF.

The only thing I see that you're not doing is categorizing the complaints. It seems to me that a simple line graph will tell you what you need to know, so long as you know the nature of the complaints.

ACF may mean AutoCorrelation Function, a measure used in time series analysis.

I’ve been tracking all types of customer complaint data - including software - for years. The techniques you mention are absolutely not appropriate for this type of data. You simply need to use an I MR chart. (You will probably have to recalculate the limits for each software version as they represent a new population). I usually found that monthly subgrouping was the most effective.

I strongly suggest that you get a copy of Donald Wheeler’s “Understandign Variation; the key to Managing Chaos”. It’s short and fairly cheap - an easy read that should help you immensely.

There is also an article that Steve Prevette posted here awhile ago that I think you will find very helpful. @StevePrevette can you attach the article or post th link?

I’ve used this one in my trining over the years (decades?!) very successfully (always properly attributed of course I’m diligent about attribution and citattions)

The challenge I see is how to deal with up and downward trends.

You want to be able to detect up and downward trends, I would think. To do this - you can use SPC to find out - is this a stable predictable process or not? If there is an increasing trend in complaints detected by SPC, you would want to find out - where is this increase coming from? A certain product / service? If so, what changed in those? A certain customer? A new customer? Your conclusion would be - SPC has said it is not stable, so therefore something has changed - what was it? You would support the SPC with Pareto charts of product / service and by customer. Perhaps it would need to be complaints per $1,000 of business.

If you went on an initiative to reduce complaints, you could tell if you succeed if a decreasing trend is detected.

Now, if indeed the process is stable - you need to ask - is that stable rate of complaints 'acceptable' or at least 'good enough' or not.

Really, SPC does not "assume" a stable process, but if you are going to effectively use SPC, you need to take action on the signals, and the feedback could allow you to reduce complaints to a negligible amount.

@Steve Prevette thank for your answer.
Indeed in that way I'm searching.
@Jim Wynne you're right about tagging complaints, it's also an exercise for medical devices to link risk codes in order to determine risk in market, however not everything is always that perfect it's being worked on

For now I just very "simply" need to answer the request to determine if there is a significant change or not when plotting the amount of complaints per time frequency.

The reason I came up with ACF is to identify seasonal effects or release cadans effects. On a uni timeseries course notes and a thesis on complaint handling I read that technique can be used to identify this.

And of course I analyse the data by the known changes made on product or other changes, but unfortunately this info is very limited our organisation is not that mature yet so it would be more like identifying a significant change and try to look what's behind it instead the other way around.

So any advice on spc methods I can evaluate to use after doing some exploratory analysis would be helpful.

@Steve Prevette
To do this - you can use SPC to find out
Is this a stable predictable process or not?
Partly yes (assumption, I have to check for normality) and partly no, fixes in releases can cause downward trend but releases might also trigger more complaints,at least temporarily.
If there is an increasing trend in complaints detected by SPC, you would want to find out - where is this increase coming from?
Yes indeed but first the audience want to be sure upon review if what they see is significant and not just opinionated decision if so or not and if significant the effort for investigation will be done.