Confidence in classification: a Bayesian approach

W.J. Krzanowski, J.E. Fieldsend, T.C. Bailey, R.M. Everson, D. Partridge and V. Schetinin

Abstract

Bayesian classification is currently of considerable interest. It provides a strategy for eliminating the uncertainty associated with a particular choice of classifier-model parameters, and is the optimal decision theoretic choice under certain circumstances when there is no single true classifier for a given data set. Modern computing capabilities can easily support the Markov chain Monte Carlo sampling that is necessary to carry out the calculations involved, but the information available in these samples is not at present being fully utilised. We show how it can be allied to known results concerning the `reject option' in order to produce an assessment of the confidence that can be ascribed to particular classifications, and how these confidence measures can be used to compare the performances of classifiers. Incorporating these confidence measures can alter the apparent ranking of classifiers as given by straightforward success or error rates. Several possible methods for obtaining confidence assessments are described, and compared on a range of data sets using the Bayesian probabilistic nearest-neighbour classifier.

Gzipped postscript  (229 kb)     PDF  (175 kb)