Such examples use a discrete variable, they are a group decision, and the decision has a great influence on business. Despite these difficulties, performing an attribute agreement analysis on bug tracking systems is not a waste of time. In fact, it is (or can be) an extremely informative, valuable and necessary exercise. The analysis of the award agreement must be applied with caution and a certain focus. If the audit is indeed planned and designed, it may reveal enough information about the causes of accuracy issues to justify a decision not to use attribute agreement analysis at all. In cases where the audit does not provide sufficient information, the attribute agreement analysis allows for a more detailed analysis indicating the implementation of safer training and modifications to the measurement system. The accuracy of a measurement system is analyzed by subdividing it into two essential components: repeatability (the ability of a particular evaluator to assign the same value or attribute several times under the same conditions) and reproducibility (the ability of several evaluators to agree among themselves for a number of circumstances). In the case of an attribute measurement system, repeatability or reproducibility problems inevitably cause accuracy problems. In addition, if one knows the overall accuracy, repeatability and reproducibility, distortions can be detected even in situations where decisions are systematically wrong. Since implementing an attribute analysis can be time-saving, expensive, and usually uncomfortable for all parties involved (the analysis is simple compared to execution), it`s best to take a moment to really understand what needs to be done and why. First, the analyst should establish that there is indeed attribute data.

It can be assumed that assigning a code – that is, classifying a code into a category – is a decision that characterizes the error by an attribute. Either a category is correctly assigned to a defect or it is not. Similarly, the defect is either attributed to the right source or not. These are ”yes” or ”no” and ”good assignment” or ”wrong assignment” answers. This part is quite simple. For example, if the accuracy rate calculated with 100 samples is 70 percent, the margin of error is about +/- 9 percent. At 80 percent, the margin is about +/- 8 percent, at 90 percent, the margin is +/- 6 percent. Of course, more and more samples can be collected to check if more accuracy is needed, but the reality is that if the database is less than 90 percent exactly, the analyst probably wants to understand why. An attribute agreement analysis allows the impact of repeatability and reproducibility on accuracy to be assessed simultaneously.

It allows the analyst to study the responses of multiple auditors, while examining multiple scenarios. It compiles statistics that assess the ability of evaluators to agree with themselves (repeatability), with each other (reproducibility) and with a well-known mastery or accuracy value (overall precision) for each characteristic – again and again. . . .