On Semantic Segmentation for Vehicle Sensing in Adverse Conditions

Abstract

Automotive scene understanding and segmentation has become increasingly popular in recent years as it meets the high demand for automotive computer vision to support future vehicle autonomy. Understanding and analysing the entire scene for tasks such as image classification and segmentation for automotive applications is extremely challenging and is made significantly more challenging by variable illumination and weather changes in outdoor environments. While most contemporary scene understanding approaches are applied under ideal weather conditions, there is a lack of research on automotive scene understanding under extreme weather conditions. Indeed, these existing approaches are unlikely to provide optimal performance compared to using established insights into extreme-weather understanding.

This thesis provides a significant contribution to a contemporary automotive semantic scene understanding by proposing a model capable of performing semantic segmentation from a monocular image input taken in adverse illumination and weather conditions. Firstly, we investigate the use of illumination-invariant image pre-transformations to reduce illumination variations of scenes when shadows are present; we do this by assessing their impact as an initial pre-process before using an existing fully deep Convolutional Neural Network (CNN). Within this context, we propose a novel approach based on illumination-invariant image representation that is combined with the chromatic component of a perceptual colour-space to improve contemporary automotive scene understanding and segmentation. Our subsequent proposed approach takes advantage of a multi-modal architecture that leverages both recent advances in adversarial training and domain adaptation as a method to correct for the
degraded visibility present under foggy weather conditions. As an end-to-end pipeline, our framework includes a competitive encoder-decoder architecture for semantic segmentation, with low computational complexity, thus enabling real-time performance. We employ distinct encoders with dense connectivity and features fusion to effectively exploit information from different inputs such as RGB colour, depth and luminance images, which contributes to more comprehensive feature extraction to better facilitate the representation of different key features. To avoid information loss and share high-resolution features in the latter reconstruction stages of CNN upsampling, we leverage skip connections. We expand this approach to perform multi-task learning as well as utilise depth as a complementary information source for the semantic segmentation task. Furthermore, we outline the process of training two domains ( and ) simultaneously with shared weights, where each model is trained on each weather condition. Experimental evaluation of our proposed approach is performed on the challenging benchmark datasets including synthetic and real-world examples illustrating competitive quantitative and qualitative performance when compared to contemporary state-of-the-art approaches.