Fadi Boutros

Machine Vision and Applications

Iris Presentation Attack Detection (PAD) algorithms address the vulnerability of iris recognition systems to presentation attacks. With the great success of deep learning methods in various computer vision fields, neural network-based iris PAD algorithms emerged. However, most PAD networks suffer from overfitting due to insufficient iris data variability. Therefore, we explore the impact of various data augmentation techniques on performance and the generalizability of iris PAD. We apply several data augmentation methods to generate variability, such as shift, rotation, and brightness. We provide in-depth analyses of the overlapping effect of these methods on performance. In addition to these widely used augmentation techniques, we also propose an augmentation selection protocol based on the assumption that various augmentation techniques contribute differently to the PAD performance. Moreover, two fusion methods are performed for more comparisons: the strategy-level and the score-level combination. We demonstrate experiments on two fine-tuned models and one trained from the scratch network and perform on the datasets in the Iris-LivDet-2017 competition designed for generalizability evaluation. Our experimental results show that augmentation methods improve iris PAD performance in many cases. Our least overlap-based augmentation selection protocol achieves the lower error rates for two networks. Besides, the shift augmentation strategy also exceeds state-of-the-art (SoTA) algorithms on the Clarkson and IIITD-WVU datasets.

Image and Vision Computing

With the widespread use of mobile devices, iris recognition systems encounter more challenges, such as the vulnerability of Presentation Attack Detection (PAD). Recent works pointed out the contact lens attacks, especially images captured under the uncontrolled environment, as a hard task for iris PAD. In this paper, we propose a novel framework for detecting iris presentation attacks that especially for detecting contact lenses based on the normalized multiple micro stripes. The classification decision is made by the majority vote of those micro-stripes. An in-depth experimental evaluation of this framework reveals a superior performance in three databases compared with state-of-the-art (SoTA) algorithms and baselines. Moreover, our solution minimizes the confusion between textured (attack) and transparent (bona fide) presentations in comparison to SoTA methods. We support the rationalization of our proposed method by studying the significance of different pupil-centered eye areas in iris PAD decisions under different experimental settings. In addition, extensive cross-database and cross-attack (unknown attack) detection evaluation experiments are demonstrated to explore the generalizability of our proposed method, texture-based method, and neural network based methods in three different databases. The results indicate that our Micro Stripes Analyses (MSA) method has, in most experiments, better generalizability compared to other baselines.

IET Biometrics

Face recognition is an essential technology in our daily lives as a contactless and convenient method of accurate identity verification. Processes such as secure login to electronic devices or identity verification at automatic border control gates are increasingly dependent on such technologies. The recent COVID‐19 pandemic has increased the focus on hygienic and contactless identity verification methods. The pandemic has led to the wide use of face masks, essential to keep the pandemic under control. The effect of mask‐wearing on face recognition in a collaborative environment is currently a sensitive yet understudied issue. Recent reports have tackled this by using face images with synthetic mask‐like face occlusions without exclusively assessing how representative they are of real face masks. These issues are addressed by presenting a specifically collected database containing three sessions, each with three different capture instructions, to simulate real use cases. The data are augmented to include previously used synthetic mask occlusions. Further studied is the effect of masked face probes on the behaviour of four face recognition systems—three academic and one commercial. This study evaluates both masked‐to‐non‐masked and masked‐to‐masked face comparisons. In addition, real masks in the database are compared with simulated masks to determine their comparative effects on face recognition performance.

IJCB 2021. IEEE/IARP International Joint Conference on Biometrics

IEEE International Joint Conference on Biometrics (IJCB) <2021, online>

Liveness Detection (LivDet)-Face is an international competition series open to academia and industry. The competition’s objective is to assess and report state-of-the-art in liveness / Presentation Attack Detection (PAD) for face recognition. Impersonation and presentation of false samples to the sensors can be classified as presentation attacks and the ability for the sensors to detect such attempts is known as PAD. LivDet-Face 2021 * will be the first edition of the face liveness competition. This competition serves as an important benchmark in face presentation attack detection, offering (a) an independent assessment of the current state of the art in face PAD, and (b) a common evaluation protocol, availability of Presentation Attack Instruments (PAI) and live face image dataset through the Biometric Evaluation and Testing (BEAT) platform. The competition can be easily followed by researchers after it is closed, in a platform in which participants can compare their solutions against the LivDet-Face winners.

IJCB 2021. IEEE/IARP International Joint Conference on Biometrics

IEEE International Joint Conference on Biometrics (IJCB) <2021, online>

Iris presentation attack detection (PAD) plays a vital role in iris recognition systems. Most existing CNN-based iris PAD solutions 1) perform only binary label supervision during the training of CNNs, serving global information learning but weakening the capture of local discriminative features, 2) prefer the stacked deeper convolutions or expert-designed networks, raising the risk of overfitting, 3) fuse multiple PAD systems or various types of features, increasing difficulty for deployment on mobile devices. Hence, we propose a novel attention-based deep pixel-wise bi-nary supervision (A-PBS) method. Pixel-wise supervision is first able to capture the fine-grained pixel/patch-level cues. Then, the attention mechanism guides the network to automatically find regions that most contribute to an accurate PAD decision. Extensive experiments are performed on LivDet-Iris 2017 and three other publicly available databases to show the effectiveness and robustness of proposed A-PBS methods. For instance, the A-PBS model achieves an HTER of 6.50% on the IIITD-WVU database outperforming state-of-the-art methods.

IJCB 2021. IEEE/IARP International Joint Conference on Biometrics

IEEE International Joint Conference on Biometrics (IJCB) <2021, online>

This paper presents a summary of the Masked Face Recognition Competitions (MFR) held within the 2021 International Joint Conference on Biometrics (IJCB 2021). The competition attracted a total of 10 participating teams with valid submissions. The affiliations of these teams are diverse and associated with academia and industry in nine different countries. These teams successfully submitted 18 valid solutions. The competition is designed to motivate solutions aiming at enhancing the face recognition accuracy of masked faces. Moreover, the competition considered the deployability of the proposed solutions by taking the compactness of the face recognition models into account. A private dataset representing a collaborative, multisession, real masked, capture scenario is used to evaluate the submitted solutions. In comparison to one of the topperforming academic face recognition solutions, 10 out of the 18 submitted solutions did score higher masked face verification accuracy.

IJCB 2021. IEEE/IARP International Joint Conference on Biometrics

IEEE International Joint Conference on Biometrics (IJCB) <2021, online>

In this paper, we present a set of extremely efficient and high throughput models for accurate face verification, Mix-FaceNets which are inspired by Mixed Depthwise Convolutional Kernels. Extensive experiment evaluations on Label Face in the Wild (LFW), Age-DB, MegaFace, and IARPA Janus Benchmarks IJB-B and IJB-C datasets have shown the effectiveness of our MixFaceNets for applications requiring extremely low computational complexity. Under the same level of computation complexity (≤ 500M FLOPs), our MixFaceNets outperform MobileFaceNets on all the evaluated datasets, achieving 99.60% accuracy on LFW, 97.05% accuracy on AgeDB-30, 93.60 TAR (at FAR1e-6) on MegaFace, 90.94 TAR (at FAR1e-4) on IJB-B and 93.08 TAR (at FAR1e-4) on IJB-C. With computational complexity between 500M and 1G FLOPs, our MixFaceNets achieved results comparable to the top-ranked models, while using significantly fewer FLOPs and less computation over-head, which proves the practical value of our proposed Mix-FaceNets. All training codes, pre-trained models, and training logs have been made available https://github.com/fdbtrs/mixfacenets.

BIOSIG 2021

Conference on Biometrics and Electronic Signatures (BIOSIG) <20, 2021, Online>

GI-Edition - Lecture Notes in Informatics (LNI), P-315

The recent Covid-19 pandemic and the fact that wearing masks in public is now mandatory in several countries, created challenges in the use of face recognition systems (FRS). In this work, we address the challenge of masked face recognition (MFR) and focus on evaluating the verification performance in FRS when verifying masked vs unmasked faces compared to verifying only unmasked faces. We propose a methodology that combines the traditional triplet loss and the mean squared error (MSE) intending to improve the robustness of an MFR system in the masked-unmasked comparison mode. The results obtained by our proposed method show improvements in a detailed step-wise ablation study. The conducted study showed significant performance gains induced by our proposed training paradigm and modified triplet loss on two evaluation databases.

Darmstadt, TU, Bachelor Thesis, 2021

Biometrics is a rapidly growing technology that aims to identify or verify people’s identities based on their physical or behavioral properties. With the rapid growth of smartphone users, the interest in secure authentication using biometric technology to authorize and identify the application user has been increased. With its high social acceptability, face recognition is one of the most convenient and accurate biometric recognition technologies which is used more and more in mobile and embedded systems for unlocking, application login and mobile payment. To enable face recognition on low computational powered devices, the model needs to be accurate, small and fast. With the rise of deep learning technology, face recognition systems were able to achieve notable verification performances. However, most high-accurate face recognition models rely on very deep Convolutional Neural Networks (CNNs) and therefore, require a high amount of computational resources, which makes them unfeasible for mobile and embedded systems. Recently, a great progess has been made in designing efficient face recognition systems by utilizing lightweight deep learning network architectures designed for common computer vision tasks for face recognition. However, none of these works designed a network specifically for the face recognition task, rather than adopting existing architectures designed for common computer vision tasks. With the development of AutoML, Neural Architecture Search (NAS) has shown excellent performance in many computer vision tasks and was able to automatically design highly efficient network architectures that outperform existing manually designed architectures. This thesis utilizes Neural Architecture Search (NAS) to automate the process of designing highly efficient neural architectures for face recognition. While other works focused on manually designing efficient architectures, this process has not been made automatic for the face recognition setting. This is the first work that utilizes Differentiable Architecture Search (DARTS) for face recognition and introduces a new search space. Based on DARTS, this thesis introduces a new network architecture named DartFaceNet. Evaluation on a variety of large-scale databases proves that DartFaceNet is able to achieve high performance on major face recognition benchmarks. Using DartFaceNet architecture and different embedding sizes, this thesis introduces three face recognition models with less than 2 million parameters. Trained with ArcFace loss on MS1MV2 dataset, DartFaceNet-256 achieves 99.5% on LFW with only 0.991 million parameters and 587.11 FLOPs which is comparable to the efficient light-weight architecture from MobileFaceNets. With less than 2 million parameters, DartFaceNet-512 outperforms existing light-weight models under 5 million parameters on CA-LFW with 95.333%. Also this thesis provides evaluation on the large-scale databases IJB-B, IJB-C and the MegaFace challenge. With only 0.925 million parameters and a memory footprint of 3.7 Megabytes, DartFaceNet-128 achieves the best trade-off between performance and model size among all DartFaceNet models

IJCB 2021. IEEE/IARP International Joint Conference on Biometrics

IEEE International Joint Conference on Biometrics (IJCB) <2021, online>

For iris recognition in non-cooperative environments, iris segmentation has been regarded as the first most important challenge still open to the biometric community, affecting all downstream tasks from normalization to recognition. In recent years, deep learning technologies have gained significant popularity among various computer vision tasks and also been introduced in iris biometrics, especially iris segmentation. To investigate recent developments and attract more interest of researchers in the iris segmentation method, we organized the 2021 NIR Iris Challenge Evaluation in Non-cooperative Environments: Segmentation and Localization (NIR-ISL 2021) at the 2021 International Joint Conference on Biometrics (IJCB 2021). The challenge was used as a public platform to assess the performance of iris segmentation and localization methods on Asian and African NIR iris images captured in non-cooperative environments. The three best-performing entries achieved solid and satisfactory iris segmentation and localization results in most cases, and their code and models have been made publicly available for reproducibility research.

Advances in Visual Computing. 16th International Symposium, ISVC 2021, Proceedings, Part I

International Symposium on Visual Computing (ISVC) <16, 2021, Online>

Lecture Notes in Computer Science (LNCS), 13017

A face morphing attack image can be verified to multiple identities, making this attack a major vulnerability to processes based on identity verification, such as border checks. Various methods have been proposed to detect face morphing attacks, however, with low generalizability to unexpected post-morphing processes. A major post-morphing process is the print and scan operation performed in many countries when issuing a passport or identity document. In this work, we address this generalization problem by adapting a pixel-wise supervision approach where we train a network to classify each pixel of the image into an attack or not, rather than only having one label for the whole image. Our pixel-wise morphing attack detection (PW-MAD) solution proved to perform more accurately than a set of established baselines. More importantly, PW-MAD shows high generalizability in comparison to related works, when evaluated on unknown re-digitized attacks. Additionally to our PW-MAD approach, we create a new face morphing attack dataset with digital and re-digitized samples, namely the LMA-DRD dataset that is publicly available for research purposes upon request.

Darmstadt, TU, Bachelor Thesis, 2021

Deep neural network have become one of the popular solutions for face recognition. However, the memory footprint of such models are still quite large for deploying to certain embedded devices or in application with limited memory requirement. In this thesis, post training static quantization and quantize aware training are utilized to reduce the memory footprint of face recognition models together with analyzing the effect of quantization on the performance of those models. Both quantization methods will be implemented with built-in functions from Pytorch to generate quantization models using 8 bits integer to represent model parameters. The result showed that both quantization methods can help to reduce almost 4 times the model memory footprint of big face recognition models like Resnet18, Resnet34, Resnet50 and Resnet100 [17, 8] with the average accuracies of quantized model still within 99% of original un-quantized models. For compact face recognition models like VarGFaceNet [44], MixFaceNet [3] and MobileFaceNet [5], these two quantization methods can make the model memory footprint 2-3 times smaller than the original size, but model accuracy will be affected significantly. Specifically, post training static quantization models of these three models showed significant drop in accuracy, around 5%, 9% and 40% for VarGFaceNet, MobileFaceNet and MixFaceNet respectively with MegaFace as evaluation metric, but quantize aware training models displayed much better accuracy, only 2%, 4% and 13% drop in accuracy for VarGFaceNet, MobileFaceNet and MixFaceNet respectively also with MegaFace as evaluation metric. ShuffleFaceNet [26] is also a compact face recognition model, but its quantized models accuracy is very close to the accuracy of original model, less than 1% drop for all evaluation metrics, and the model memory footprint is reduced from 10.64MB to 3.18MB.

Advances in Visual Computing. 16th International Symposium, ISVC 2021, Proceedings, Part I

International Symposium on Visual Computing (ISVC) <16, 2021, Online>

Lecture Notes in Computer Science (LNCS), 13017

Face morphing attacks aim at creating face images that are verifiable to be the face of multiple identities, which can lead to building faulty identity links in operations like border checks. While creating a morphed face detector (MFD), training on all possible attack types is essential to achieve good detection performance. Therefore, investigating new methods of creating morphing attacks drives the generalizability of MADs. Creating morphing attacks was performed on the image level, by landmark interpolation, or on the latent-space level, by manipulating latent vectors in a generative adversarial network. The earlier results in varying blending artifacts and the latter results in synthetic-like striping artifacts. This work presents the novel morphing pipeline, ReGenMorph, to eliminate the LMA blending artifacts by using a GAN-based generation, as well as, eliminate the manipulation in the latent space, resulting in visibly realistic morphed images compared to previous works. The generated ReGenMorph appearance is compared to recent morphing approaches and evaluated for face recognition vulnerability and attack detectability, whether as known or unknown attacks.

Pattern Recognition

Using the face as a biometric identity trait is motivated by the contactless nature of the capture process and the high accuracy of the recognition algorithms. After the current COVID-19 pandemic, wearing a face mask has been imposed in public places to keep the pandemic under control. However, face occlusion due to wearing a mask presents an emerging challenge for face recognition systems. In this paper, we present a solution to improve masked face recognition performance. Specifically, we propose the Embedding Unmasking Model (EUM) operated on top of existing face recognition models. We also propose a novel loss function, the Self-restrained Triplet (SRT), which enabled the EUM to produce embeddings similar to these of unmasked faces of the same identities. The achieved evaluation results on three face recognition models, two real masked datasets, and two synthetically generated masked face datasets proved that our proposed approach significantly improves the performance in most experimental settings.

BIOSIG 2020

Conference on Biometrics and Electronic Signatures (BIOSIG) <19, 2020, Online>

GI-Edition - Lecture Notes in Informatics (LNI), P-306

Despite the wide use of deep neural network for periocular verification, achieving smaller deep learning models with high performance that can be deployed on low computational powered devices remains a challenge. In term of computation cost, we present in this paper a lightweight deep learning model with only 1.1m of trainable parameters, DenseNet-20, based on DenseNet architecture. Further, we present an approach to enhance the verification performance of DenseNet-20 via knowledge distillation. With the experiments on VISPI dataset captured with two different smartphones, iPhone and Nokia, we show that introducing knowledge distillation to DenseNet-20 training phase outperforms the same model trained without knowledge distillation where the Equal Error Rate (EER) reduces from 8.36% to 4.56% EER on iPhone data, from 5.33% to 4.64% EER on Nokia data, and from 20.98% to 15.54% EER on cross-smartphone data.

FUSION 2020

International Conference on Information Fusion (FUSION) <23, 2020, Online>

Iris presentation attack detection (PAD) algorithms are developed to address the vulnerability of iris recognition systems to presentation attacks. Taking into account that the deep features successfully improved computer vision performance in various fields including iris recognition, it is natural to use features extracted from deep neural networks for iris PAD. Each layer in a deep learning network carries features of different level of abstraction. The features extracted from the first layer to the higher layers become more complex and more abstract. This might point our complementary information in these features that can collaborate towards an accurate PAD decision. Therefore, we propose an iris PAD solution based on multi-layer fusion. The information extracted from the last several convolutional layers are fused on two levels, feature-level and score-level. We demonstrated experiments on both, off-theshelf pre-trained network and network trained from scratch. An extensive experiment also explores the complementary between different layer combinations of deep features. Our experimental results show that feature-level based multi-layer fusion method performs better than the best single layer feature extractor in most cases. In addition, our fusion results achieve similar or better results than the state-of-the-art algorithms on the Notre Dame and IIITD-WVU databases of the Iris Liveness Detection Competition 2017 (LivDet-Iris 2017).

FUSION 2020

International Conference on Information Fusion (FUSION) <23, 2020, Online>

The growing popularity of Virtual Reality and Augmented Reality (VR/AR) devices in many applications also demands authentication of users. As the devices inherently capture the eye image while capturing the user interaction, the authentication can be devised using the iris and periocular recognition. While both iris and periocular data being non-ideal unlike the data captured from standard biometric sensors, the authentication performance is expected to be lower. In this work, we present and evaluate a fusion framework for improving the biometric authentication performance. Specifically, we employ score-level fusion for two independent biometric systems of iris and periocular region to avoid expensive feature-level fusion. With a detailed evaluation of three different score-level fusion after the score normalization on a dataset of 12579 images, we report the performance gain in authentication using score-level fusion for iris and periocular recognition.

Image and Vision Computing

Augmented and virtual reality deployment is finding increasing use in novel applications. Some of these emerging and foreseen applications allow the users to access sensitive information and functionalities. Head Mounted Displays (HMD) are used to enable such applications and they typically include eye facing cameras to facilitate advanced user interaction. Such integrated cameras capture iris and partial periocular region during the interaction. This work investigates the possibility of using the captured ocular images from integrated cameras from HMD devices for biometric verification, taking into account the expected limited computational power of such devices. Such an approach can allow user to be verified in a manner that does not require any special and explicit user action. In addition to our comprehensive analyses, we present a light weight, yet accurate, segmentation solution for the ocular region captured from HMD devices. Further, we benchmark a number of well-established iris and periocular verification methods along with an in-depth analysis on the impact of iris sample selection and its effect on iris recognition performance for HMD devices. To the end, we also propose and validate an identity-preserving synthetic ocular image generation mechanism that can be used for large scale data generation for training purposes or attack generation purposes. We establish the realistic image quality of generated images with high fidelity and identity preserving capabilities through benchmarking them for iris and periocular verification.

IJCB 2020. IEEE/IARP International Joint Conference on Biometrics

IEEE/IARP International Joint Conference on Biometrics (IJCB) <2020, online>

Launched in 2013, LivDet-Iris is an international competition series open to academia and industry with the aim to assess and report advances in iris Presentation Attack Detection (PAD). This paper presents results from the fourth competition of the series: LivDet-Iris 2020. This year's competition introduced several novel elements: (a) incorporated new types of attacks (samples displayed on a screen, cadaver eyes and prosthetic eyes), (b) initiated LivDet-Iris as an on-going effort, with a testing protocol available now to everyone via the Biometrics Evaluation and Testing (BEAT)* open-source platform to facilitate reproducibility and benchmarking of new algorithms continuously, and (c) performance comparison of the submitted entries with three baseline methods (offered by the University of Notre Dame and Michigan State University), and three open-source iris PAD methods available in the public domain. The best performing entry to the competition reported a weighted average APCER of 59.10% and a BPCER of 0.46% over all five attack types. This paper serves as the latest evaluation of iris PAD on a large spectrum of presentation attack instruments.

Darmstadt, TU, Master Thesis, 2020

This thesis explores the effectiveness of shape variation in the context of occlusion type augmentations by utilizing a shape generation policy, called Shapeshifting, for the construction of different occlusion shapes. In this approach, a reference point is randomly initialized within the image and several polygon vertices are then randomly placed around the reference point for the construction of the occlusion area. Further improvements, by applying the proposed approach multiple times in unstructured and structured ways, in what is referred to as K-Shapeshifting and Structured Shapeshifting, were also explored. This thesis also explores a segmentation-based occlusion approach, called Semantic Occlusion. The proposed approach constructs an occlusion mask using the regions inferred through an unsupervised semantic segmentation approach. The occlusion mask is constructed by selecting an arbitary region from the inferred segmentation. This approach was extended by further evaluating the performance of occluding multiple regions from the semantic segmentation. The proposed approaches were evaluated using the widelyused augmentation policy of random crop alongside with random flip as baseline. On the benchmark dataset CIFAR10, ResNet18 with Shapeshifting achieved an accuracy of 0.9574, an improvement over the baseline accuracy of 0.9528. On SVHN, the multi-component variant of Shapeshifting, K-Shapeshifting, achieved an accuracy of 0.9731, an improvement over the baseline accuracy of 0.9631. On STL10, the same policy, K-Shapeshifting, achieved 0.9729 on STL10 over the baseline accuracy of 0.9704. The proposed approach, Shapeshifting, that uses polygon generation algorithm for the construction of the occlusion mask achieved competitive performances. While it is shown that adding more polygon vertices for the construction of the occlusion polygon contributes to a significant improvment in performance in the proposed setting, the experiment results also provide empirical evidence that the improvement in performance is mainly attributed to the underlying increase in occlusion ratio. As such, it is concluded with empirical evidence that shape variation of the occlusion mask does not provide significant contribution to the model accuracy. The proposed Semantic Occlusion approach, which uses a single region of the inferred segmentation for the construction of the occlusion mask achieved 0.9444 on CIFAR10, 0.9649 on SVHN and 0.9637 on STL10. Improvements were achieved in the extension, K-Semantic Occlusion, with 0.9502 on CIFAR10, 0.9678 on SVHN and 0.967 on STL10. The proposed segmentation-based approach only achieved improvements over the baseline on SVHN and did not outperform the previous approach, Shapeshifting.

IJCB 2020. IEEE/IARP International Joint Conference on Biometrics

IEEE/IARP International Joint Conference on Biometrics (IJCB) <2020, online>

Augmented and virtual reality is being deployed in different fields of applications. Such applications might involve accessing or processing critical and sensitive information, which requires strict and continuous access control. Given that Head-Mounted Displays (HMD) developed for such applications commonly contains internal cameras for gaze tracking purposes, we evaluate the suitability of such setup for verifying the users through iris recognition. In this work, we first evaluate a set of iris recognition algorithms suitable for HMD devices by investigating three well-established handcrafted feature extraction approaches, and to complement it, we also present the analysis using four deep learning models. While taking into consideration the minimalistic hardware requirements of stand-alone HMD, we employ and adapt a recently developed miniature segmentation model (EyeMMS) for segmenting the iris. Further, to account for non-ideal and non-collaborative capture of iris, we define a new iris quality metric that we termed as Iris Mask Ratio (IMR) to quantify the iris recognition performance. Motivated by the performance of iris recognition, we also propose the continuous authentication of users in a non-collaborative capture setting in HMD. Through the experiments on a publicly available OpenEDS dataset, we show that performance with EER = 5% can be achieved using deep learning methods in a general setting, along with high accuracy for continuous user authentication.

IWBF 2020. Proceedings

International Workshop on Biometrics and Forensics (IWBF) <8, 2020, online>

Virtual and augmented reality technologies are increasingly used in a wide range of applications. Such technologies employ a Head Mounted Display (HMD) that typicallyincludes an eye-facing camera and is used for eye tracking.As some of these applications require accessing or transmittinghighly sensitive private information, a trusted verification ofthe operator’s identity is needed. We investigate the use ofHMD-setup to perform verification of operator using periocularregion captured from inbuilt camera. However, the uncontrollednature of the periocular capture within the HMD results inimages with a high variation in relative eye location and eyeopening due to varied interactions. Therefore, we propose a newnormalization scheme to align the ocular images and then, a newreference sample selection protocol to achieve higher verificationaccuracy. The applicability of our proposed scheme is exemplifiedusing two handcrafted feature extraction methods and two deeplearning strategies.We conclude by stating the feasibility of sucha verification approach despite the uncontrolled nature of thecaptured ocular images, especially when proper alignment andsample selection strategy is employed.

Darmstadt, TU, Bachelor Thesis, 2020

Current face recognition models have benefited from the recent advanced development of deep learning techniques achieving very high verification performances. However, most of the recent works pay less attention to the computational efficiency of these models. Hence, deploying such models on low computational powered mobile devices is challenging. Nevertheless, recent studies have also shown an increasing demand for mobile user identity authentication using biometrics modalities i.e. face, fingerprint, iris, etc. As a consequence, large well-performing face recognition models have to become smaller to be deployable on mobile devices. This thesis proposes a solution to enhance the verification performance of small face recognition models via knowledge distillation. Conventional knowledge distillation transfers knowledge from a large teacher network to a small student network by mimicking the classification layer. In addition to that, this thesis adapts the knowledge distillation method to be applicable to the feature level which the used teacher ArcFace tries to optimize. The verification results of this thesis prove that knowledge distillation can enhance the performance of a small face recognition model compared to the same model trained without knowledge distillation. Applying conventional knowledge distillation to a ResNet- 56 model increased the accuracy from 99.267% to 99.3% on LFW and from 93.767% to 93.867% on AgeDB. This accuracy of the ResNet-56 student is only 0.117% below the accuracy of its twelve times larger ResNet-18 teacher on LFW and even higher on AgeDB by 0.067%. Moreover, when matching the objective function of ArcFace with knowledge distillation, the performance of a ResNet-56 model could be further increased to 99.367% on LFW. This implies it exceeded the accuracy of the same face recognition model trained without knowledge distillation by a margin of 0.1%. At the same time, it decreased the FMR on LFW compared to the model trained without knowledge distillation.

IJCB 2020. IEEE/IARP International Joint Conference on Biometrics

IEEE/IARP International Joint Conference on Biometrics (IJCB) <2020, online>

The paper presents a summary of the 2020 Sclera Segmentation Benchmarking Competition (SSBC), the 7th in the series of group benchmarking efforts centred around the problem of sclera segmentation. Different from previous editions, the goal of SSBC 2020 was to evaluate the performance of sclera-segmentation models on images captured with mobile devices. The competition was used as a platform to assess the sensitivity of existing models to i) differences in mobile devices used for image capture and ii) changes in the ambient acquisition conditions. 26 research groups registered for SSBC 2020, out of which 13 took part in the final round and submitted a total of 16 segmentation models for scoring. These included a wide variety of deep-learning solutions as well as one approach based on standard image processing techniques. Experiments were conducted with three recent datasets. Most of the segmentation models achieved relatively consistent performance across images captured with different mobile devices (with slight differences across devices), but struggled most with low-quality images captured in challenging ambient conditions, i.e., in an indoor environment and with poor lighting.

BIOSIG 2020

Conference on Biometrics and Electronic Signatures (BIOSIG) <19, 2020, Online>

GI-Edition - Lecture Notes in Informatics (LNI), P-306

Face recognition has become essential in our daily lives as a convenient and contactless method of accurate identity verification. Process such as identity verification at automatic border control gates or the secure login to electronic devices are increasingly dependant on such technologies. The recent COVID-19 pandemic have increased the value of hygienic and contactless identity verification. However, the pandemic led to the wide use of face masks, essential to keep the pandemic under control. The effect of wearing a mask on face recognition in a collaborative environment is currently sensitive yet understudied issue. We address that by presenting a specifically collected database containing three session, each with three different capture instructions, to simulate realistic use cases.We further study the effect of masked face probes on the behaviour of three top-performing face recognition systems, two academic solutions and one commercial off-the-shelf (COTS) system.

2019 Conference Proceedings

IEEE International Conference on Imaging Systems and Techniques (IST 2019) <2019, Abu Dhabi, UAE>

Recent efforts in biometrics have focused on crossdomain face recognition where images from one domain are either transformed or synthesized. In this work, we focus on a similar problem for cross spectral periocular recognition where the images from Near Infra Red (NIR) domain are matched against Visible (VIS) spectrum images. Specifically, we propose to adapt a cascaded image transformation network that can produce NIR image given a VIS image. The proposed approach is first validated with regards to the quality of the image produced by employing various quality factors. Second the applicability is demonstrated with images generated by the proposed approach. We employ a publicly available cross-spectral periocular image data of 240 unique periocular instances captured in 8 different capture sessions. We experimentally validate that the proposed image transformation scheme can produce NIR like images and also can be used with any existing feature extraction scheme. To this extent, we demonstrate the biometric applicability by using both hand-crafted and deep neural network based features under verification setting. The obtained EER of 0.7% indicates the suitability of proposed approach for image transformation from the VIS to the NIR domain.

The 12th IAPR International Conference On Biometrics

IAPR International Conference on Biometrics (ICB) <12, 2019, Crete, Greece>

Face synthesis from thermal to visible spectrum is fundamental to perform cross-spectrum face recognition as it simplifies its integration in existing commercial face recognition systems and enables manual face verification. In this paper, a new solution based on cascaded refinement networks is proposed. This method generates visible-like colored images of high visual quality without requiring large amounts of training data. By employing a contextual loss function during training, the proposed network is inherently scale and rotation invariant. We discuss the visual perception of the generated visible-like faces in comparison with recent works. We also provide an objective evaluation in terms of cross-spectrum face recognition, where the generated faces were compared against a gallery in visible spectrum using two state-of-the-art deep learning based face recognition systems. When compared to the recently published TV-GAN solution, the performance of the face recognition systems, OpenFace and LightCNN, was improved by a 42.48% (i.e. from 10.76% to 15.37%) and a 71.43% (i.e. from 33.606% to 57.612%), respectively.

2019 International Conference on Computer Vision Workshops. Proceedings

International Conference on Computer Vision (ICCV) <17, 2019, Seoul, Korea>

Training functionality-demanding AR/VR systems require accurate and robust gaze estimation and tracking solutions. Achieving such a performance requires the availability of diverse eye image data that might only be acquired by the means of image generation. Works addressing the generation of such images did not target realistic and identity-specific images, nor did they address the practicalrelevant case of generation from semantic labels. Therefore, this work proposes a solution to generate realistic and identity-specific images that correspond to semantic labels, given samples of a specific identity. Our proposed solution consists of two stages. In the first stage, a network is trained to transform the semantic label into a corresponding eye image of a generic identity. The second stage is an identity-specific network that induces identity details on the generic eye image. The results of our D-ID-Net solutions shows a high degree of identity-preservation and similarity to the ground-truth images, with an RMSE of 7.235.

Pattern Recognition

German Conference on Pattern Recognition (GCPR) <40, 2018, Stuttgart, Germany>

Lecture Notes in Computer Science (LNCS), 11269

Face morphing attacks create face images that are verifiable to multiple identities. Associating such images to identity documents lead to building faulty identity links, causing attacks on operations like border crossing. Most of previously proposed morphing attack detection approaches directly classified features extracted from the investigated image. We discuss the operational opportunity of having a live face probe to support the morphing detection decision and propose a detection approach that take advantage of that. Our proposed solution considers the facial landmarks shifting patterns between reference and probe images. This is represented by the directed distances to avoid confusion with shifts caused by other variations. We validated our approach using a publicly available database, built on 549 identities. Our proposed detection concept is tested with three landmark detectors and proved to outperform the baseline concept based on handcrafted and transferable CNN features.

FUSION 2019

International Conference on Information Fusion (FUSION) <22, 2019, Ottawa, Canada>

Soft biometrics identify certain traits of individuals based on their sampled biometric characteristics. The automatic identification of traits like age and gender provides valuable information in applications ranging from forensics to service personalization. Color images are stored within a color space containing different channels. Each channel represents a different portion of the information contained in the image, including these of soft biometric patterns. The analysis of the age and gender information in the different channels and different color spaces was not previously studied. This work discusses the soft biometric performances using these channels and analyzes the sample error overlap between all possible channels to successfully prove that different information is considered in the decision making from each channel. We also present a multi-channel selection protocols and fusion solution of the selected channels. Beside the analyzes of color spaces and their channels, our proposed multi-channel fusion solution extends beyond state-of-the-art performance in age estimation on the widely used Adience dataset.

2019 International Conference on Computer Vision Workshops. Proceedings

International Conference on Computer Vision (ICCV) <17, 2019, Seoul, Korea>

Segmentation of the iris or sclera is an essential processing block in ocular biometric systems. However, humancomputer interaction, as in VR/AR applications, requires multiple region segmentation to enable smoother interaction and eye-tracking. Such application does not only demand highly accurate and generalizable segmentation, it requires such segmentation model to be appropriate for the limited computational power of embedded systems. This puts strict limits on the size of the deployed deep learning models. This work presents a miniature multi-scale segmentation network consisting of inter-connected convolutional modules. We present a baseline multi-scale segmentation network and modify it to reduce its parameters by more than 80 times, while reducing its accuracy by less than 3%, resulting in our Eye-MMS model containing only 80k parameters. This work is developed on the OpenEDS database and is conducted in preparation for the OpenEDS Semantic Segmentation Challenge.

Darmstadt, TU, Bachelor Thesis, 2019

Human monitoring is a major research direction in computer vision, with application in smart living assistants, human-computer interaction, surveillance, health monitoring, etc. This variety of applications has led to the design of many human monitoring systems in order to extract information about environment inhabitances based on different technologies. In computer vision, this task can be achieved by generating a 2D skeleton representing the human body. However, users do not favor constant camera monitoring. This thesis investigates the use of ultrasonic sensors for human pose estimation, which have a very low cost and require only minimalistic infrastructure. We do this by establishing a framework for data collection of human poses, using an ultrasound sensor and a camera for labeling the data. We collected data from 25 people, performing 4 different activities and evaluated the collected data on 4 different artificial neural network architectures by training them and comparing their performance against each other, showing that an LSTM architecture achieved results up to 67% accuracy. The use of a non-visual input stream for pose estimation is also motivated by the less privacy intrusive nature of ultrasound data, compared with videos of homes and people inside them with the application in smart living environments.

IEEE 10th International Conference on Biometrics: Theory, Applications and Systems

IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS) <10, 2019, Tampa, Florida, USA>

The quality of images produced by generative adversarial networks (GAN) is commonly a trade-off between the model size, its training data needs, and the generation resolution. This trad-off is clear when applying GANs to issues like generating face morphing attacks, where the latent vector used by the generator is manipulated. In this paper, we propose an image enhancement solution designed to increase the quality and resolution of GAN-generated images. The solution is designed to require limited training data and be extendable to higher resolutions. We successfully apply our solution on GAN-based face morphing attacks. Beside the face recognition vulnerability and attack detectability analysis, we prove that the images enhanced by our solution are of higher visual and quantitative quality in comparison to unprocessed attacks and attack images enhanced by state-of-the-art super-resolution approaches.

Darmstadt, TU, Master Thesis, 2019

Automated face recognition has gained wider deployment ground after the recent accuracy gains achieved by deep learning techniques. Despite the rapid performance gains, face recognition still suffers from very critical issues. One of the recently uncovered, and very sensitive, challenges is the ethnicity bias in face recognition decision. This is the case, unfortunately, even in the latest commercial and academic face recognition system. In 2018, the National Institute of Standards and Technology (NIST) published the latest report regarding the evaluation result of commercial face recognition solutions from several major biometric vendors. The report specifically evaluated and demonstrated the variance of the error rates of the evaluated solutions based on demographic variations. This thesis is one of the first research efforts in addressing the decision bias challenge in biometrics. It builds its hypothesis on the strong assumption that ethnicity bias is caused by the relative under-representation of certain ethnicities in training databases. This work introduces a novel ethnicity-driven data augmentation approach to reduce biometric bias. The proposed approach successfully utilize a generative image model to generate new face images that preserve the identity of their source images while partially transforming their ethnicity to the targeted ethnicity group. A large-scale ethnicity-labeled face images database is created in order to develop and evaluate the proposed approach. To achieve that, part of this thesis focused on creating an ethnicity classifier to annotate face images, achieving accuracy in the state-of-the-art range. The proposed ethnicity-driven face generative model is developed based on the ethnicity labeled images to generate realistically and high-resolution face images, depending on a limit amount of training data. More importantly, the thesis proves that the proposed augmentation approach strongly preserves the identity of the input images and partially transforms the ethnicity. The augmented images are used as part of the training data of a face recognition model. The achieved verification results prove that the proposed ethnicity augmentation methods significantly and consistently reduced the ethnicity bias of the face recognition model. For examples, the ERR was reduced from 0.159 to 0.130 when verifying inter-ethnicity samples of Black individuals on a model trained on Asian individual images, respectively before and after applying the proposed training data augmentation. Moreover, the overall performance of the face recognition model was improved. However, this improvement was more significant, as intended, in the targeted ethnicity groups.

FUSION 2019

International Conference on Information Fusion (FUSION) <22, 2019, Ottawa, Canada>

We introduce, in this paper, a new scheme of score level fusion for face authentication from visible and thermal face data. This proposed scheme provides a fast and straightforward integration into existing face recognition systems and does not require recollection of enrollment data in thermal spectrum. In addition to be used as a possible countermeasure against spoofing, this paper investigates the potential role of thermal spectrum in improving face recognition performances when employed under adversarial acquisition conditions. We consider a context where individuals have been enrolled solely in visible spectrum, and their identity will be verified using 2 sets of probes: visible and thermal. We show that the optimal way to proceed is to synthesis a visible image from the thermal face in order to create a synthetic-visible probe; and then to fuse scores resulting from comparisons between visible gallery with both visible probe and synthetic-visible probe. The thermal-to-visible face synthesis is performed using a Cascaded Refinement Network (CRN) and face features were extracted and matched using LightCNN and Local Binary Patterns (LBP). The fusion procedure is performed based on several quality measures computed on both visible and thermal-to-visible generated probes and compared to the visible gallery images.

Darmstadt, TU, Bachelor Thesis, 2019

The recent advances in ubiquitous computing and the Internet of Things induce the awareness of smart environments and enhance the interaction between the system and the users. This enables energy savings, improvements in human comfort and assistance, and many other convenience services. However, it requires abilities to detect, count and identify current occupying invidiuals within the smart environment. Person detection and identification with devices like cameras is a well-addressed topic in literature. However, this vision-based sensing is not socially acceptable in a home setting. Person detection based on contact sensors, such as wearable devices, relies too much on correct behavior of its users, and can be regarded as inconvenient especially for older adults, as it requires a constant contact with the users. This works aims at using ambient sensors that can be installed in existing indoor environments to detect and identify individuals in smart environments. Ambient sensors can mitigate disadvantages of other sensing methods: (a) ambient sensors can be seamlessly integrated into homes, (b) they can sense without direct interaction from their users, (c) they are more socially acceptable than video surveillance. These benefits make it realistic to capture ambient sensor information constantly, which could make it possible to detect and identify people with context-aware environments. In order to achieve person detection and identification, three different tasks are investigated: Single Human Occupancy Detection, Multiple Human Occupancy Detection, and Human Identification. This thesis investigates the use of different machine learning methods for aforementioned tasks, including neural networks, SVM, kNN, Discriminant Analysis and CART, trains and evaluates on three different databases of ambient sensor measurements, and compares with the current methods proposed in literature. A bidirectional recurrent model that uses GRU cells is proposed to extract patterns from time series data. On a data set specifically intended for occupancy detection, the state-of-the-art is outperformed with neural network models, achieving up to 99,44% accuracy. Another utilized database is a composition of sensor data collected from 30 different apartments, annotated with daily life activities. These activity annotations are useful enough to gain knowledge for all three detection/identification tasks. While not enough information is provided to fully explore multiple person detection and identification, it is shown that (a) a system can predict whether the environment is not occupied, or that one person, or multiple people are present, and (b) ambient sensor measurement patterns are sufficient to distinguish two similar apartments that have one resident each, so indirectly two persons can be identified.

2018 Proceedings of the 26th European Signal Processing Conference (EUSIPCO)

European Signal Processing Conference (EUSIPCO) <26, 2018, Rome, Italy>

Normalization is an important step for different fusion, classification, and decision making applications. Previous normalization approaches considered bringing values from different sources into a common range or distribution characteristics. In this work we propose a new normalization approach that transfers values into a normalized space where their relative performance in binary decision making is aligned across their whole range. Multi-biometric verification is a typical problem where information from different sources are normalized and fused to make a binary decision and therefore a good platform to evaluate the proposed normalization.We conducted an evaluation on two publicly available databases and showed that the normalization solution we are proposing consistently outperformed state-of-the-art and best practice approaches, e.g. by reducing the false rejection rate at 0.01% false acceptance rate by 60- 75% compared to the widely used z-score normalization under the sum-rule fusion.