Towards Fair Face Recognition: Mitigating Racial Bias via Generative Deep Learning

Abstract

Facial recognition is one of the most academically studied and industrially developed areas within computer vision, where we readily find associated applications deployed globally. This widespread adoption has uncovered significant performance variation across subjects of different racial profiles leading to focused research attention on racial bias within face recognition spanning both current causation and potential future solutions. However, still the use of ill-defined racial categorisations, a lack of both consideration of the broader context of historical and social factors and contemporary evaluation methods hinder collaborative efforts towards mitigation of racial bias within face recognition. In support, this thesis firstly provides an extensive taxonomic review of research on racial bias within face recognition, covering topics from problem definition and racial grouping strategies to every aspect and all stages of the face recognition processing pipeline. Moreover, a comprehensive discussion within the review reveals the potential pitfalls and limitations of contemporary mitigation strategies that need to be considered within future research endeavours or commercial applications alike.



Accordingly, the prior literature has identified a need for alternative evaluation methodologies, particularly in the context of assessing racial bias. In response to this need, a phenotype-based racial bias analysis methodology is introduced via the use of a set of observable characteristics of an individual face where a race-related facial phenotype is hence specific to the human face and correlated to the racial profile of the subject. Subsequently, a commonplace lossy image compression algorithm impact at the initial stage of face recognition processing pipeline, image and dataset acquisition, concerning the racial characteristics of the subject, is investigated by adopting the proposed evaluation methodology. The results reveal the disparate performance decrease on specific racial phenotype categories and show improvement of the use of compressed imagery during training and removing chroma subsampling on the performance of specific racial phenotype categories more affected by lossy compression. Furthermore, a novel adversarial-derived data augmentation methodology is presented by aiming to enable dataset balance at a per-subject level via image-to-image transformation for the transfer of sensitive racial characteristic facial features to improve performance variation among racial and phenotype-based categories. The proposed approach decreases the performance variations between four racial groups by 15.81%.

Consequently, a novel GAN framework to enable fine-grained control over individual race-related phenotype attributes of the facial images is introduced. The proposed framework achieves both higher image quality and controllability on race-related facial phenotype attributes without requiring any synthetic or 3D data. Within the chapter, we introduce the CelebA-HQ-Augmented-Cleaned dataset, which is the first semi-synthesised, manually-cleaned, high-quality dataset encompassing over 26,500 images with a diverse distribution. Finally, this thesis concludes with an extensive discussion with insights draw from the literature, proposed approaches, and experiments presented throughout the thesis and outlines future directions for addressing racial bias within face recognition.