Abstract—Detection of outlying ratings of samples is a primary step in statistical analysis and classification. A novel rule-based method is presented for automatically detecting and removing outlying ratings in order to improve the quality of sample classification and to increase the degree of agreement between raters. The effectiveness of our method in improving the degree of agreement, assessed using a modified Fleiss' kappa, is demonstrated through a practical example. Our method is conceptually transparent, computationally simple and easy to apply in practice. It is expected to be a useful tool in many real world applications.