Learning Multi-Scale Knowledge-Guided Features for Text-Guided Face Recognition

Abstract

Text-guided face recognition (TGFR) aims to improve the performance of state-of-the-art face recognition (FR) algorithms by incorporating auxiliary information, such as distinct facial marks and attributes, provided as natural language descriptions. Current TGFR algorithms have been proven to be highly effective in addressing performance drops in state-of-the-art FR models, particularly in scenarios involving sensor noise, low resolution, and turbulence effects. Although existing methods explore various algorithms using different cross-modal alignment and fusion techniques, they encounter practical limitations in real-world applications. For example, during inference, textual descriptions associated with face images may be missing, lacking crucial details, or incorrect. Furthermore, the presence of inherent modality heterogeneity poses a significant challenge in achieving effective cross-modal alignment. To address these challenges, we introduce CaptionFace, a TGFR framework that integrates GPTFace, a face image captioning model designed to generate context-rich natural language descriptions from low-resolution facial images. By leveraging GPTFace, we overcome the issue of missing textual descriptions, expanding the applicability of CaptionFace to single-modal FR datasets. Additionally, we introduce a multi-scale feature alignment (MSFA) module to ensure semantic alignment between face-caption pairs at different granularities. Furthermore, we introduce an attribute-aware loss and perform knowledge adaptation to specifically adapt textual knowledge from facial features. Extensive experiments on three face-caption datasets and various unconstrained single-modal benchmark datasets demonstrate that CaptionFace significantly outperforms state-of-the-art FR models and existing TGFR approaches.

Overview

CaptionFace extends our earlier WACV 2024 work, TGFR, with a deeper alignment module, a knowledge-distillation mechanism between modalities, an attribute-aware objective, and an end-to-end captioning model that removes the framework's dependence on having a caption available at inference time. It is evaluated on three face-caption datasets (Multi-Modal CelebA-HQ, Face2Text, CelebA-Dialog), on single-modal benchmark FR datasets (LFW, CALFW, AgeDB-30) using captions generated on the fly, and on a fine-grained image classification benchmark (CUB-200-2011) to show the alignment module generalizes beyond faces.

• Removes TGFR's dependence on ground-truth captions at inference time, via GPTFace.
• Local + global contrastive alignment (MSAM), instead of one coarse image–text contrastive loss.
• Caption Knowledge Adaptation distills image-specific detail into text embeddings without polluting them with a direct identity loss.
• An attribute-aware loss adds a modality-invariant, fine-grained supervisory signal on top of identity loss.
• Consistently outperforms ArcFace, AdaFace, MagFace, and our own TGFR baseline across three face-caption datasets and three FR benchmarks.
• GPTFace matches BLIP / BLIP-2 captioning quality at roughly one-third the input resolution and a fraction of the parameter count (177M vs. 446M / 1.1B).

Architecture

Four components make up the framework:

1. GPTFace

   A lightweight face-captioning model (frozen AdaFace image encoder + 12-block GPT-2 decoder) that generates a natural-language description directly from a low-resolution face image, so CaptionFace can be applied to single-modal FR datasets that have no associated text.

2. Multi-Scale Alignment Module (MSAM)

   Aligns image and caption embeddings at both the global level (global semantic alignment loss) and the local, region-to-token level (local semantic alignment loss).

3. Caption Knowledge Adaptation (CKA)

   A distillation mechanism that enriches the caption embedding with image-specific detail it might be missing, instead of forcing an identity loss directly onto noisy text features.

4. Cross-Modal Fusion Module (CMFM)

   A transformer block with multi-head cross- and self-attention that fuses local image features with caption token embeddings for the final identity decision, trained with a focal loss.

An attribute-aware loss is applied on both modalities using a 40-attribute vector extracted from the caption with TextBlob, adding a modality-invariant, fine-grained supervisory signal on top of the identity loss.

Face image captioning: GPTFace

GPTFace pairs a frozen AdaFace image encoder with a 12-block GPT-2 language decoder. Since GPT-2 has no native mechanism for consuming encoder features, a cross-attention scheme is incorporated into every GPT-2 block. A projection network combines local and global image features extracted by AdaFace, with a self-attention layer added to capture long-range fine-grained dependencies before passing the combined representation into GPT-2's cross-attention.

GPTFace optimizes a language-modeling loss for next-token prediction, plus a multi-label classification loss (L_attr-text) that penalizes the decoder for missing or misrepresented facial attributes in the generated caption, judged against attributes extracted from the ground truth using TextBlob.

Results

1:1 verification on face-caption datasets

TPR (%) at FPR = 1e-6 / 1e-5 / 1e-4, both ArcFace and AdaFace as the image encoder, BLIP as the text encoder.

CaptionFace is the top-performing method on all three datasets with both image encoders. With AdaFace, it outperforms TGFR by 1.56 / 1.18 / 0 points on MMCelebA, 1.79 / 1.0 / 1.08 on Face2Text, and 1.34 / 1.56 / 1.47 on CelebA-Dialog at TAR@FAR = 1e-6 / 1e-5 / 1e-4.

Benchmark FR accuracy

Verification accuracy (%) on LFW / CALFW / AgeDB-30, captions generated on the fly by GPTFace (no ground-truth text at test time).

*Our implementation. Backbones are kept frozen during CaptionFace training (paper Section IV.A.3). LFW and AgeDB-30 are near saturation, but CaptionFace still consistently improves over every corresponding FR model.

GPTFace vs. multimodal foundation models

Captioning quality on MMCelebA. GPTFace runs at 112×112 input, roughly a third the resolution of BLIP / BLIP-2.

GPTFace with AdaFace outperforms BLIP and BLIP-2 by 25.34% and 12.54% in B@4, while using a third the input resolution and far fewer trainable parameters (177M vs. 446M / 1.1B).

Fine-grained image classification (CUB-200-2011)

Rank-1 accuracy (%), caption supervision only — no bounding boxes or part annotations used.

CaptionFace improves 5.23 points over plain ResNet18 and 5.88 points over plain ResNet50, surpassing TGFR by roughly 2 points on both backbones — confirming the alignment module generalizes beyond faces.

Ablation studies

Objective function

TAR@FAR (%) on MMCelebA with AdaFace; identity loss and focal loss are present in every row as the baseline.

Every loss term contributes positively. Combined, the full objective surpasses the identity + fusion baseline by 9.11 / 5.12 / 5.65 points at TAR@FAR = 1e-6 / 1e-5 / 1e-4.

Robustness to atmospheric turbulence

EER and TAR@FAR (%) on MMCelebA as image quality degrades with simulated atmospheric turbulence (D/r₀ ratio).

At the most severe degradation level (3), CaptionFace improves 2.38 / 1.28 / 3.13 points over TGFR and 15.11 / 17.06 / 20.0 points over plain AdaFace. The performance gap widens as image quality worsens.

Qualitative analysis

Grad-CAM++ class activation maps show that CaptionFace's attention shifts toward the facial regions actually mentioned in the caption, rather than spreading uniformly across the face or background the way the plain AdaFace encoder does.

Code

The repository includes CaptionFace training and evaluation, the standalone GPTFace captioning model, the fine-grained classification extension on CUB-200-2011, and the legacy BERT-encoder baseline used for comparison in the paper.

Train CaptionFace on CelebA with the AdaFace backbone and Cross-Modal Fusion Module:

python3 src/train_captionface.py \
    --dataset celeba \
    --model_type adaface \
    --fusion_type CMF_FR \
    --batch_size 8 \
    --max_epoch 18 \
    --is_itc --is_KD --is_attr_loss

Evaluate 1:1 verification on the held-out face-caption test split:

python3 src/test_captionface.py --model_type adaface --dataset celeba

Full installation steps, dataset layout, pretrained-weight paths, and the complete command reference for GPTFace training, benchmark FR evaluation, and the CUB-200-2011 extension are in the repository README.

Citation

If you use this code or build on this work, please cite:

This paper extends two earlier works on text-guided face recognition:

Related work

• WACV 2024
  TGFR — Text-Guided Face Recognition Using Multi-Granularity Cross-Modal Contrastive Learning
  Our earlier two-stage text-guided FR framework that this work extends to an end-to-end design.
• BMVC 2025
  FaceCPT — cross-modal face recognition and vision-language pre-training.

This work was supported by the Center for Identification Technology Research (CITeR) and the National Science Foundation under Grant 1650474. Image encoders build on ArcFace, AdaFace, and MagFace; the text encoder builds on BLIP; the captioning decoder builds on GPT-2.