Template Recovery Attack on Encrypted Face Recognition Systems with Unprotected Decision using Synthetic Faces

Amina Bassit ^1,2* Florian Hahn ² Zohra Rezgui ² Hatef Otroshi Shahreza ^3,4 Raymond Veldhuis ^2,5 Andreas Peter ^2,6

• ¹ Michigan State University, East Lansing, United States
• ² University of Twente, Enschede, Netherlands
• ³ Idiap Research Institute, Martigny, Switzerland
• ⁴ École polytechnique fédérale de Lausanne (EPFL), Lausanne, Switzerland
• ⁵ Norwegian University of Science and Technology Gjøvik, Gjøvik, Norway
• ⁶ University of Oldenburg, Oldenburg, Germany

Homomorphically encrypted face recognition systems protect face information by hiding facial embeddings using homomorphic encryption schemes as a privacy-preserving mechanism. These systems conduct comparisons of encrypted face templates without the need for decryption, producing recognition scores as in an embedding space. For efficiency considerations, face recognition systems often tolerate revealing these scores after performing a comparison under encryption, yet this practice introduces a vulnerability. Accessing comparison scores in the cleartext introduces the risk of reconstructing the target embedding and inferring demographic information, which compromises the privacy and security of face recognition systems. In this paper, we highlight the susceptibility of face recognition systems to these risks, particularly for systems employing the inner product as a similarity measure commonly used in modern deep learning-based systems. We propose a training-less face template recovery attack based on the Lagrange multiplier optimization method. Our attack requires only a few random facial images and their comparison scores relative to the target facial template. To show the applicability of our attack in real-world scenarios, we assume that the synthetic facial images represent the spoofed faces. This assumption aligns with scenarios where attackers lack direct access to face recognition systems or avoid being caught, necessitating external approaches using spoofed faces to compromise these systems. We evaluate our attack on synthetic faces by verifying whether the recovered template is deemed similar to the target template held by face recognition systems set to accept strict thresholds. Our empirical study shows that an attacker needs between 50 and 192 scores and synthetic faces for a template recovery with a 100% success rate. We then analyze the impact of recovered templates by measuring the amount of soft biometrics they contain, and their resemblance to the reconstructed images of their target templates.