Fraunhofer Institute for Computer Graphics Research IGD
Mitigating Soft-Biometric Driven Bias and Privacy Concerns in Face Recognition Systems
Biometric verification refers to the automatic verification of a person’s identity based on their behavioural and biological characteristics. Among various biometric modalities, the face is one of the most widely used since it is easily acquirable in unconstrained environments and provides a strong uniqueness. In recent years, face recognition systems spread world-wide and are increasingly involved in critical decision-making processes such as finance, public security, and forensics. The growing effect of these systems on everybody’s daily life is driven by the strong enhancements in their recognition performance. The advances in extracting deeply-learned feature representations from face images enabled the high-performance of current face recognition systems. However, the success of these representations came at the cost of two major discriminatory concerns. These concerns are driven by soft-biometric attributes such as demographics, accessories, health conditions, or hairstyles.
Philipp Terhörst, research assistant in the department "Smart Living and Biometric Technologies", addressed these concerns in the course of his doctorate and successfully defended his dissertation on April 20, 2021 - congratulations!
The public defense of Philipp Terhörst's dissertation took place on April 20 at Fraunhofer IGD in Darmstadt and online. Supervisors of the work were Prof. Dr. Arjan Kuijper (TU Darmstadt), Prof. Dr. techn. Dieter W. Fellner (TU Darmstadt) and Prof. Dr. Vitomir Štruc (University of Ljubljana).
Abstract
The first concern is about bias in face recognition. Current face recognition solutions are built on representation-learning strategies that optimize total recognition performance. These learning strategies often depend on the underlying distribution of soft-biometric attributes in the training data. Consequently, the behaviour of the learned face recognition solutions strongly varies depending on the individual’s soft-biometrics (e.g. based on the individual’s ethnicity).
The second concern tackles the user’s privacy in such systems. Although face recognition systems are trained to recognize individuals based on face images, the deeply-learned representation of an individual contains more information than just the person’s identity. Privacy-sensitive information such as demographics, sexual orientation, or health status, is encoded in such representations. However, for many applications, the biometric data is expected to be used for recognition only and thus, raises major privacy issues. The unauthorized access of such individual’s privacy-sensitive information can lead to unfair or unequal treatment of this individual.
Both issues are caused by the presence of soft-biometric attribute information in the face images. Previous research focused on investigating the influence of demographic attributes on both concerns. Consequently, the solutions from previous works focused on the mitigation of demographic-concerns only as well. Moreover, these approaches require computationally-heavy retraining of the deployed face recognition model and thus, are hardly-integrable into existing systems.
Unlike previous works, this thesis proposes solutions to mitigating soft-biometric driven bias and privacy concerns in face recognition systems that are easily-integrable in existing systems and aim for more comprehensive mitigation, not limited to pre-defined demographic attributes. This aims at enhancing the reliability, trust, and dissemination of these systems.
The first part of this work provides in-depth investigations on soft-biometric driven bias and privacy concerns in face recognition over a wide range of soft-biometric attributes. The findings of these investigations guided the development of the proposed solutions. The investigations showed that a high number of soft-biometric and privacy-sensitive attributes are encoded in face representations. Moreover, the presence of these soft-biometric attributes strongly influences the behaviour of face recognition systems. This demonstrates the strong need for more comprehensive privacy-enhancing and bias-mitigating technologies that are not limited to pre-defined (demographic) attributes.
Guided by these findings, this work proposes solutions for mitigating bias in face recognition systems and solutions for the enhancement of soft-biometric privacy in these systems. The proposed bias-mitigating solutions operate on the comparison- and score-level of recognition system and thus, can be easily integrated. Incorporating the notation of individual fairness, that aims at treating similar individuals similarly, strongly mitigates bias of unknown origins and further improves the overall-recognition performance of the system.
The proposed solutions for enhancing the soft-biometric privacy in face recognition systems either manipulate existing face representations directly or changes the representation type including the inference process for verification. The manipulation of existing face representations aims at directly suppressing the encoded privacy-risk information in an easily-integrable manner. Contrarily, the inference-level solutions indirectly suppress this privacy-risk information by changing the way of how this information is encoded.
To summarise, this work investigates soft-biometric driven bias and privacy concerns in face recognition systems and proposed solutions to mitigate these. Unlike previous works, the proposed approaches are (a) highly effective in mitigating these concerns, (b) not limited to the mitigation of concerns origin from specific attributes, and (c) easily-integrable into existing systems. Moreover, the presented solutions are not limited to face biometrics and thus, aim at enhancing the reliability, trust, and dissemination of biometric systems in general.