ECCV 2026 Workshops

EXPL-FR: Explaining Face Recognition Models via Vision-Language Alignment

Guray Ozgur^1,2, Mustafa Efe Tamyapar^1,2, Naser Damer^1,2, Fadi Boutros¹

¹Fraunhofer IGD, Darmstadt · ²Technische Universität Darmstadt

A deployed face recognition model returns a similarity score and no reason for it. EXPL-FR reads which named attribute that score relied upon directly off the frozen matcher's own embedding space, not off a separate commentator. A ~1M-parameter adapter, trained on face images alone and never on text, turns any written prompt into a direction in FR space; cosine similarities to those directions form a face's semantic signature.

Method

Align the VLM image encoder with a frozen FR encoder on images alone; the text encoder inherits the mapping, so every written prompt becomes an anchor in FR space.

EXPL-FR pipeline: frozen VLM text and vision encoders, a frozen FR model, and the trained adapter.

Figure 1. The adapter is the only learned component (~1M parameters).

Read a Face

Pick a verification pair, then drag the blend slider: the differential signature grows from reference minus reference (flat) toward reference minus compared.

0.71 cosine similarity · match

morph blend 100%

Differential signature · reference − genuine

Figure 2. Per-image and differential explanations on a morph case. The reference-genuine difference is near-flat; the morph residual names inherited attributes. The values shown are illustrative.

Signatures

One bar per prompt, read off the FR model's own embedding space.

signature

look up an attribute nearest match in the released vocabulary, not live inference

above the row's own mean below it

The Obstacle

A VLM and an FR encoder are trained separately. Their embeddings live in unrelated coordinate systems, so cross-encoder verification sits at chance until the adapter bridges them.

50.29% unaligned · chance

94.56% aligned cross-encoder (ours)

97.44% FR self-verification ceiling

unaligned 50.29%

prompt anchors face embeddings

1:1 verification, mean over LFW, AgeDB-30, CFP-FP, CPLFW, CALFW
Representation pair	Mean (%)
FR self-verification (upper bound)	97.44
VLM self-verification	82.33
Aligned-VLM self-verification	92.58
Unaligned cross-encoder	50.29
Aligned cross-encoder (ours)	94.56
Vocabulary projection, in VLM space	51.98
Vocabulary projection, in FR space (ours)	71.66

Within 2.88 points of the FR ceiling; vocabulary projection gains +19.68 when moved from VLM space into FR space.

What Survives the Mapping

Eyewear, hair colour and facial hair come through into FR space; distance, scene context and lighting are discounted.

Label-free detectability per concept in VLM space versus FR space.

Figure 3. The 100 most detectable prompts separate identities better than the full vocabulary.

Auditing Without Labels

Sec. 3.5 builds the same FR-space attribute axis three ways, with decreasing supervision. Right column: cost of extending the audit by one new attribute.

Supplementary Fig. S1. Supervision levels for attribute-level FR auditing.

(1) Labels prior practice

attribute labels + face images → group-mean axis in FR space

re-annotate the dataset

(2) VLM-proxy no labels

written prompt + unlabeled images + VLM ranking → pseudo group-mean axis

re-rank the collection

(3) Ours prompts only

written prompt + adapter g_θ → prompt axis (no images)

write one sentence

Label-free ethnicity dependence against real ten-fold verification error.

Figure 4. Label-free dependence vs. real verification error (τ = 0.92).

Finding	Statistic	Against
Ethnicity dependence vs. per-group error (RFW)	τ = 0.92	real ten-fold verification error
Matched-axis sensitivity vs. per-attribute EER (GAN-Control)	ρ = 0.90 / 0.83 / 0.95	labeled / VLM-proxy / prompt-only axes