Contributions to Nonparametric Predictive Inference: Classification and Performance Evaluation

Abstract

Nonparametric predictive inference (NPI) is a statistical methodology which uses imprecise probability to quantify uncertainty. In imprecise probability, lower and upper probabilities are assigned to events to represent uncertainty about the events. NPI provides lower and upper probabilities for future events based on observed data.

Since NPI uses imprecise probability, methods based on the NPI approach provide predictions which are, by nature, imprecise, making it challenging to compare their performance directly with methods based on classical probability. This highlights the importance of studying the performance of the methods based on the NPI approach. Examining their performance when predictions take the form of a single value, an interval or a more general set of values, is important. This thesis contributes to NPI-based methods by investigating their performance across different scenarios of predictive inference.

In classification, the Direct Nonparametric Predictive Inference (D-NPI) method is based on the NPI approach. This method uses a splitting criterion, Correct Indication (CI), which relies on NPI lower and upper probabilities. Imprecise classification and multi-label classification are two important problems within the area of classification. In imprecise classification, a classifier, which is known as imprecise classifier, predicts a set of class labels rather than a single class label. In multi-label classification, a single data point in a dataset, which is known as an instance, can be associated with multiple labels simultaneously. This thesis applies D-NPI to ensemble methods, imprecise classification and multi-label classification, and investigates its performance across different classification problems. Experimental studies are conducted to evaluate the performance of the proposed methods using several measures and statistical tests. Further studies are carried out to compare the performance of the proposed methods to other methods from the literature.

The results obtained from the proposed ensemble methods, bagging and random forest, indicate their effectiveness compared to the D-NPI method. The D-NPI-based random forest method performs well compared to the well known random forest method from the literature. In imprecise classification, some of the proposed imprecise classifiers have strong performance compared to other methods. In multi-label classification problems, the results, according to various evaluation measures, show that the best performance is observed for methods which are based on the NPI approach for most of the datasets considered.

This thesis further contributes to the new use of performance evaluation measures for imprecise probability inferences focusing on an NPI-based method for bivariate data. Performance measures for imprecise probability methods are required to consider both the accuracy and the imprecision of predictions. These measures include loss functions as well as a new interval score measure, designed with three weights to evaluate the performance of prediction intervals. This thesis investigates the performance of an NPI method for bivariate data in different scenarios using the introduced performance measures, and compares its performance with an alternative method. The results show that the performance measures are effective in assessing the performance of the method, enabling investigation of the accuracy and the imprecision of intervals, and comparing different scenarios. Using the interval score measure enables evaluating the performance in terms of both accuracy and precision, selecting its weights depends on the requirements of the application.