Number of decimal places : Gage R and R

Hi all,

A crossed gage R and r was run recently with 3 operators 10 samples. The data from the programmed vision system has some with 6 decimal places, 4 decimal places and 5 decimal places. The thing is when I enter this data into minitab (not as worksheet), for eg: when I enter 0.0483. then i enter 0.048206, the 0.0483 becomes 0.048300. same for 5 decimal places, it adds an extra 0 at the end. So when I do a Gage r and R crossed study, with these values in minitab , does it affect the %Tolerance?(SV/Toler) and %Study Var (%SV) under Gage Evaluation? Currently for the dimension .0396±.0001 this is causing an unrealistic number
Total Gage R and R: %Tolerance (SV/Toler): 714.33, the %SV is 71.65. Any idea why?

Part Operator Measurement
1 327 0.03992
2 327 0.04001
3 327 0.03982
4 327 0.03978
5 327 0.0395
6 327 0.04004
7 327 0.03999
8 327 0.0399
9 327 0.0399
10 327 0.04018
1 327 0.03979
2 327 0.03981
3 327 0.03987
4 327 0.04016
5 327 0.03951
6 327 0.04022
7 327 0.04007
8 327 0.03985
9 327 0.04011
10 327 0.04021
1 327 0.039514
2 327 0.038991
3 327 0.03985
4 327 0.039767
5 327 0.039559
6 327 0.039825
7 327 0.039857
8 327 0.040328
9 327 0.039979
10 327 0.03984
1 1567 0.039794
2 1567 0.038894
3 1567 0.039488
4 1567 0.039776
5 1567 0.039417
6 1567 0.039869
7 1567 0.0398
8 1567 0.0399
9 1567 0.039885
10 1567 0.040174
1 1567 0.039522
2 1567 0.0387
3 1567 0.039409
4 1567 0.039717
5 1567 0.039509
6 1567 0.039946
7 1567 0.039824
8 1567 0.039832
9 1567 0.039751
10 1567 0.04026
1 1567 0.039516
2 1567 0.038748
3 1567 0.039517
4 1567 0.039662
5 1567 0.039401
6 1567 0.039725
7 1567 0.039793
8 1567 0.0399
9 1567 0.0397
10 1567 0.04
1 1618 0.039648
2 1618 0.039138
3 1618 0.039908
4 1618 0.039744
5 1618 0.03947
6 1618 0.039941
7 1618 0.039468
8 1618 0.039838
9 1618 0.039852
10 1618 0.039924
1 1618 0.039526
2 1618 0.038874
3 1618 0.039534
4 1618 0.039948
5 1618 0.039505
6 1618 0.039654
7 1618 0.039438
8 1618 0.039936
9 1618 0.039815
10 1618 0.039926
1 1618 0.039622
2 1618 0.039551
3 1618 0.039474
4 1618 0.03972
5 1618 0.039384
6 1618 0.039588
7 1618 0.03929
8 1618 0.039931
9 1618 0.039875
10 1618 0.039804

rivariva said:

A crossed gage R and r was run recently with 3 operators 10 samples. The data from the programmed vision system has some with 6 decimal places, 4 decimal places and 5 decimal places. The thing is when I enter this data into minitab (not as worksheet), for eg: when I enter 0.0483. then i enter 0.048206, the 0.0483 becomes 0.048300. same for 5 decimal places, it adds an extra 0 at the end. So when I do a Gage r and R crossed study, with these values in minitab , does it affect the %Tolerance?(SV/Toler) and %Study Var (%SV) under Gage Evaluation? Currently for the dimension .0396±.0001 this is causing an unrealistic number
Total Gage R and R: %Tolerance (SV/Toler): 714.33, the %SV is 71.65. Any idea why?

Click to expand...

The issue that you mentioned (i.e., adding zeroes) should not affect results. The number of decimal places (i.e., resolution of measurement) in general (with the exception of adding zeroes) can have an impact on the results, but not enough to cause the problems that you are seeing. You have issues with repeatability, reproducibility, including operator x part interactions. See the results below.

I recommend reading Dr. Wheeler's "An Honest Gauge R&R Study" particularly page 15, Honest Steps 12 and 13 on Effective measurement increments. This covers how to determine how many decimal places that you should record.

Your data


Your data truncated to four decimal places

Thank you!! This helps to understand the Gabe r and r