Improving Embedding Spaces in Deep Neural Networks for Computer Vision Tasks

Seeing is believing, and computer vision is enabling the next step of human-computer interface by allowing the machines to interact with the world much like humans do, i.e., by enabling machines to see the world around them. Computer vision has several applications that directly impact the everyday lives of humans, from helping us navigate our environment to improving our entertainment experiences.
A significant application of computer vision is in the field of autonomous driving, where vision-based sensors detect obstacles and navigate the vehicle safely.
Computer vision algorithms also play a significant role in healthcare, where they are used to assist with medical imaging analysis and diagnosis.
Additionally, computer vision has revolutionized the field of entertainment, which is used to create realistic special effects in movies and video games.
As computer vision technology continues to improve and become more accessible, we will likely see more and more applications in our daily lives, improving safety, efficiency, and overall quality of life.

One of the most significant breakthroughs in computer vision has been the development of convolutional neural networks (CNNs), a deep learning algorithm specifically designed for image recognition tasks. CNNs consist of multiple layers of interconnected nodes that are trained to recognize patterns and features in images. By leveraging CNNs, computer vision systems can accurately identify objects and scenes in images and even detect fine-grained details such as textures and shapes. The impact of CNNs on computer vision has been enormous. Before their development, computer vision algorithms relied on handcrafted features and heuristics, which were limited in their ability to identify and classify objects in images accurately. With the advent of CNNs, however, computer vision systems can now achieve state-of-the-art performance on various tasks, including object detection, image segmentation, and image captioning. In this work, we will focus on the major components of Deep Neural Networks, namely, the dataset space, the embedding space, and the loss function to elucidate performance gains on computer vision tasks. A holistic look at these components will enable us to advance the current knowledge of the community in this space and enable future work to build upon our efforts.

Starting from the embedding space, We will present ConvMix, a novel approach to combining latent embedding of a CNN to create embeddings that are informed by the multiple stages of the network. The intent here is to create an embedding space that leverages the spatial and context information of an image for the downstream task. Combining latent spaces is under-explored in the literature. A dedicated study of this area will enable the community to refocus its efforts on the embedding space. We experimentally showcase our performance gains and highlight the need to reconsider the embedding space by showing that ConvMix can approximate the performance of more complex networks without additional parameters. We also demonstrate the improvement in training behavior by using ConvMix and the improvement it yields in the final accuracy for classification tasks.

Next, we will present our work on the dataset space with multi-instance learning titled Phantom Embeddings. Multi-instance learning aims to utilize more than one instance to train in each step to ensure model generalization. Phantom Embeddings creates an approximate training instance via a convex combination of multiple training samples in some ϵ neighborhood of the original training samples and trains the model using these approximations rather than the original images. This training regime enables the network to self-regularize due to the `phantom' images it sees. We empirically show that this training regime enables better model performance and outperforms the base versions of the networks.

Having looked at the data space and embedding spaces, we also look at the loss function of deep neural networks to extract performance gains. We target the proxy Intersection Over Union (IOU) in the object detection task. IOU for the detected bounding boxes in an Object Detection task dictates the training of the feature extractor, and traditionally, it is approximated with Huber Loss, which is a regression loss that treats the errors outside a threshold region as relaxation between L1 and L2 loss, encouraging heavy penalty for the outliers. Inside that threshold, the loss reduces to L2 loss, encouraging smoother convergence. While this proxy loss has shown great results, we present a relaxation on top of this loss, which enables us to substitute the L2 portion of the loss inside the threshold with the actual IOU loss. Doing so allows us to optimize for the true target of the regression loss in object detection tasks.

Finally, we will present applied research work that was done for the Volkswagen Financial Services (VWFS) project. The use case under consideration was designed for the automatic detection of damages on leased vehicles and price estimation for the cost of repair. This use case entailed with it the entire MLOPs pipeline. Presented with the raw dataset, we preprocessed the data, annotated it, designed several machine learning solutions for the individual requirements of the project partner, and deployed models on cloud infrastructure.