Meaning of Precision / Recall in AI

Examples are valid for :
TMflow Software version : TMflow 2.20 or above versions
TM Robot Hardware version : HW3.2 or above versions
Other specific requirements : None 
Note that older or newer software versions may have different results.
 

System Detection Capability Evaluation Metrics

• NG Products: No Good / Defective items.

• OK Products: Good / Non-defective items.

Definitions

• Positive: The category actually being focused on. In this example, it is the NG product.
• Negative: The product opposite to the focused category. In this example, it is the OK product.
• True Positive (TP): An NG product correctly identified by the system as NG.
• True Negative (TN): An OK product correctly identified by the system as OK.
• False Positive (FP): An OK product incorrectly identified by the system as NG (False Alarm).
• False Negative (FN): An NG product incorrectly identified by the system as OK (Leakage).

When system output = NG

When system output = OK

Recall and Precision

• Recall = TP / (TP + FN)
• Precision = TP / (TP + FP)

When Detection Criteria = Moderate

• Recall = TP / (TP + FN) = 8 / 9 = 0.89

=> For every 100 defective items, 11 will leak out.

• Precision = TP / (TP + FP) = 8 / 10 = 0.80

=> For every 100 items judged as defective by the system, only 80 are actually defective.

2. When Detection Criteria = Strict

(Raising criteria to ensure NO NG products leak out)

• Recall = TP / (TP + FN) = 9 / 9 = 1.00

=> 100% of defective products are detected; none will leak out.

• Precision = TP / (TP + FP) = 9 / 12 = 0.75

=> For every 100 items judged as defective by the system, only 75 are actually defective.

Scenarios

Scenario 1

Assumption: Factory yield is 80% (Per 100 items: 80 OK, 20 NG).

System: Recall 89%, Precision 80%.

Result: When producing 100 items, out of 20 NG items, 2.2 (20 * (1-0.89)) will leak out. 25 (20 / 0.80) items will be judged as NG by the system and require manual re-inspection.

System Introduction Benefit:

• 11% of NG products will leak out.
• Manual inspection is reduced from 100% (checking every item) to checking only the 25 identified items. Workload reduced to 25%.

Scenario 2

Assumption: Factory yield is 80% (Per 100 items: 80 OK, 20 NG).

System: Recall 100%, Precision 75%.

Result: When producing 100 items, no defective products will leak out. 26.67 (20 / 0.75) items will be judged as NG by the system and require manual re-inspection.

System Introduction Benefit:

• NG products will not leak out.
• Manual inspection is reduced from 100% to checking only the 26.67 identified items. Workload reduced to 26.67%.

Scenario 3

Assumption: Factory yield is 99% (Per 100 items: 99 OK, 1 NG).

System: Recall 100%, Precision 75%.

Result: When producing 100 items, no defective products will leak out. 1.33 (1 / 0.75) items will be judged as NG by the system and require manual re-inspection.

System Introduction Benefit:

• NG products will not leak out.
• Manual inspection is reduced from 100% to checking only the 1.33 identified items. Workload reduced to 1.33%.