Nonparametric Predictive Inference
for Selection and Ranking

Abstract

This thesis introduces Nonparametric Predictive Inference (NPI) for selection and
ranking, based on a single future observation from each group, and focuses on two
main contributions. First, the development of NPI procedures for various selection
and ranking events. Secondly, the application of different loss functions to quantify
the loss incurred from non-optimal selection and ranking decisions.
Initially, NPI is applied to rank the best groups within subsets. A selected subset
refers to one or more independent groups that are better than the rest, where better
means that all future observations from the groups in the selected subset exceed
all the future observations from the non-selected groups. The ’independent group’
means that information about the random quantities for one group does not provide
any information about the random quantities for another group. Two special cases
are considered: selecting a ranked subset of the two best groups and the three best
groups. For the subset consisting of two groups, the exact NPI lower and upper
probabilities are derived for the event that these are the two best groups, while for
the subset consisting of three groups, approximate NPI lower and upper probabilities
are derived for the event that these are the three best groups.
The thesis further explores a more general case of selection and ranking, addressing
how to rank buckets containing one or more independent groups. Here, ’bucket’
refers to a cluster or grouping of independent groups. This approach tackles two
key questions: how to allocate groups to buckets, and how to determine the optimal
number of buckets—defined as the allocation that maximises or minimises the NPI lower and upper probabilities for a given event. Various allocation methods are
evaluated, including those based on measures such as the median. Additionally, the
NPI-Bootstrap method is used to estimate probabilities, to approximate the probability
of the event of interest itself, rather than its lower and upper probabilities.
Throughout the thesis, data from the literature illustrate and support the methods.
In this thesis, the NPI method is applied across various selection and ranking
events, using different loss functions to quantify the loss incurred from non-optimal
selection and ranking decisions. Uncertainty is quantified by calculating the NPI
lower and upper expected losses for the events corresponding to these scenarios.
In the selection scenario, zero-one, linear, and quadratic loss functions are used in
both pairwise and multiple comparisons. Several selection events are considered,
including selecting the best group, selecting the subset of best groups, and selecting
the subset that includes the best group. In ranking scenarios, zero-one and general
multi-level loss functions are applied to ranked subsets of best groups. The
zero-one loss function provides a binary measure of whether the ranking is correct,
while the general multi-level loss function allows for a more nuanced evaluation by
assigning penalties based on the specific ranking of groups according to the next
future observation per group. For the general event of selection and ranking, linear
and quadratic loss functions are used to evaluate the ranking of groups assigned to
different buckets. The effect of the use of different loss functions on the selection
and ranking decisions is illustrated by examples.