Inspection Measurement Analysis

by: Wayne Stansbury

http://www.wfstansbury-riskmgt.com/

Ms Lila Bility is a lawyer in a large firm.  She had three examiners in her department that inspected contracts for defects before sending these contracts to their clients.  She was getting some feedback from some of her clients that there were some mistakes in the contracts.  Lila’s manager asks her how well her inspection process caught defects.  She remembered her Six Sigma instructor telling her about the “Good News / Bad News” aspect of the inspection process.  The bad news is that your inspection system is always worse than you think but the good news is that it is usually easy to fix.  Lila, fresh from Six Sigma training, would use Inspection Measurement Analysis (IMA) to address her manager’s question.  The purpose of IMA is to test the effectiveness of people that will conduct visual inspections of products before giving them to the customer. Lila chose 22 contracts that would be used to test the examiners on their effectiveness of inspecting contracts.  Some of the contracts had defects and should be rejected (R) by the examiners.  Some of the contracts were free of defects and should be accepted (A) by the examiner.  She also believed that the examiners were having a more difficult time inspecting larger contracts.  Contracts 1 – 12 are smaller contracts and contracts 13 to 22 are larger contracts.  The table showing the 22 contracts of the first IMA study is shown below.

The instances where the examiner did not agree with the actual result are bolded.

Contract Comments A/R-Actual Examiner1 Examiner2 Examiner3
1 no signature R A R R
2 amendments in wrong order R R R A
3 Good A A R A
4 Good A R A A
5 2 pages missing R R R R
6 Good A A A A
7 File #’s do not match R R A R
8 Good A A A A
9 Good A A A A
10 signature in wrong line R R R R
11 mismatch on dates R R R R
12 Good A A A A
13 multiple pages missing R A A R
14 mismatch on dates R A A R
15 Good A A R A
16 no signature R R R R
17 Good A A A A
18 Good A A A A
19 amendments in wrong order R A A R
20 File #’s do not match R A A R
21 signature in wrong line R A A R
22 Good A A A A

Analysis was conducted which provided the following results.

Results from IMA – Before Improvements

Parameter Examiner 1 Examiner 2 Examiner 3 Total
Effectiveness 0.68 0.64 0.95 0.76
False Rejects 0.10 0.20 0 0.10
False Accepts 0.50 0.50 0.08 0.36
Bias 0.20 0.40 0 0.28
Inspection Guidelines
Parameter Acceptable Marginal Unacceptable
Effectiveness >=0.90 0.80 to 0.90 <0.80
False Rejects <=0.05 0.05 to 0.10 > 0.10
False Accepts <=0.02 0.02 to 0.05 > 0.05
Bias 0.80 to 1.20 0.50 to 0.80 or 1.2 to 1.5 <0.50 or >1.50

Lila compared the analysis with the inspection guidelines and she saw that there was clearly a problem with the inspection process for this law firm.  A quick explanation of the guidelines is in order.  Effectiveness (E) is a measure of how accurately the examiner rejects are accepts the contracts.  For example, Examiner1 correctly rejected/accepted 15 contracts out of 22 contracts he inspected to give an E=0.68.  False Accepts and false rejects is a measure of how often the examiner incorrectly rejected or accepted a contract.  For example, Examiner 1 falsely rejected 1 contract out of 10 that should have been accepted to give a false rejection rate of 0.10.  Examiner1 falsely accepted 6 contracts out of 12 that should have been rejected to give a false acceptance rate of 0.50.  Bias is simply a measure of determining if examiners favored false rejection over false acceptance or vice versa.  This study clearly showed that the examiners were falsely accepting contracts at a much higher rate.  This was especially troublesome.  False accepts are more troublesome since contacts are reaching customers with errors. False rejects cause the muda or rework.  Also, they were definitely having a harder time with larger contracts.

After some discussions with the examiners, she realized that the examiners received no formal training and there was nothing written down.  Therefore, she worked with the examiners to develop a SOP and then Lila trained the examiners on the new SOP.  She then conducted another round of IMA with following results.

Contract Comments A/R-Actual Examiner1 Examiner2 Examiner3
1 no signature R R R R
2 amendments in wrong order R R R R
3 Good A A A A
4 Good A A A A
5 2 pages missing R R R R
6 Good A A A A
7 File #’s do not match R R R R
8 Good A A A A
9 Good A A A A
10 signature in wrong line R R R R
11 mismatch on dates R R R R
12 Good A A A A
13 multiple pages missing R R R R
14 mismatch on dates R R R R
15 Good A A A A
16 no signature R R R R
17 Good A A A A
18 Good A A R A
19 amendments in wrong order R R R R
20 File #’s do not match R A R R
21 signature in wrong line R R R R
22 Good A A A A

Results from IMA – After Improvements

Parameter Examiner 1 Examiner 2 Examiner 3 Total
Effectiveness 0.95 0.95 1.00 0.97
False Rejects 0.00 0.10 0 0.03
False Accepts 0.08 0.00 0.00 0.03
Bias 0.00 NA NA 1.20

The improvement was significantly better for all three examiners.  She also remembered a quote from long ago, “You can’t inspect quality into a process.”  Therefore, she met again with her examiners to conduct 5-Why analysis to determine root cause of why there are errors in the contracts in the first place.  What are the tools of 5-Why analysis and Root Cause Analysis”, asked the examiners.  “No time to talk about that now,” exclaimed Lila, “I have to tell my manager how awesome our inspection process is!”

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: