Contributions to Statistical Reproducibility and Small-Sample Bootstrap

Abstract

This thesis consists of three contributions: an investigation of bootstrap methods for small samples, an overview of reproducibility, and advances on the topic of test reproducibility. These contributions are inspired by statistical practice in preclinical research.

Small samples are a common feature in preclinical research. In this thesis, an extensive simulation study is carried out to explore whether bootstrap methods can perform well with such samples. This study compares four bootstrap methods: nonparametric predictive inference bootstrap, Banks bootstrap, Hutson bootstrap, and Efron bootstrap. The thesis concludes that bootstrap methods can provide a useful estimation and prediction inference for small samples. Some initial recommendations for practitioners are provided.

There are no standardised definitions for reproducibility. This work further contributes to the existing literature by classifying reproducibility definitions from the literature into five types, and providing an overview of reproducibility with a focus on issues related to preclinical research, and on statistical reproducibility and its quantification.

This research explores the variability of statistical methods from the statistical reproducibility perspective. It considers reproducibility as a predictive inference problem. The nonparametric predictive inference (NPI) method, which is focused on the prediction of future observations based on existing data, is applied. In this work, statistical reproducibility is defined as the probability of the event that, if the test was repeated under identical circumstances and with the same sample size, the same test outcome would be reached. This thesis presents contributions to NPI reproducibility for the t-test and the Wilcoxon-Mann Whitney test. As one of the prevailing patterns, a test statistic falling close to the test threshold leads to low reproducibility. In a preclinical test scenario, reproducibility of a final decision involving multiple pairwise comparisons is studied.