Audio-Driven Adversarial Defense for 3D Talking Face Generation with totally Visual Fidelity Preservation

Audio-driven talking face generation (TFG) aims to synthesize realistic talking portraits by animating facial motions according to speech signals. Early methods mainly relied on 2D image generation or warping-based pipelines (Chen et al., 2019; Chen et al., 2018; Prajwal et al., 2020), which achieved plausible lip synchronization but often struggled to model natural head movement, fine-grained facial dynamics, and multi-view consistency.

Recent advances in 3D representations have substantially improved the realism and controllability of TFG. In particular, Neural Radiance Fields (NeRF) (Athar et al., 2022; Bi et al., 2024; Guo et al., 2021; Tang et al., 2025; Li et al., 2023; Ye et al., 2023; Peng et al., 2024) and 3D Gaussian Splatting (3DGS) (Cho et al., 2024; Li et al., 2024; Li et al., 2025a) enable the reconstruction of subject-specific 3D portraits from monocular reference videos, leading to stronger identity consistency, more accurate lip synchronization, and better view-consistent rendering. Compared with generalized one-shot portrait animation methods (Li et al., 2025b; Ye et al., 2024), these personalized 3D-field approaches are particularly powerful because they can recover reusable identity-aware 3D representations and support highly realistic speech-driven reenactment.

However, this strong generation capability also introduces significant privacy risks. Once a target person’s reference video is collected, current 3D-field TFG models can reconstruct a controllable digital portrait and synthesize realistic videos under arbitrary speech input. This makes them especially concerning in public multimedia sharing scenarios, where user portraits and voices can be easily harvested and reused. In this work, we focus on defending against such audio-driven 3D-field TFG systems, which pose a more serious threat than conventional 2D talking face generation due to their stronger realism, controllability, and reusability.

With the rapid progress of portrait generation models, recent studies have begun to explore proactive defenses against talking face generation and related portrait manipulation systems. Most existing methods are developed in the visual domain, where protective perturbations are injected into facial regions to disrupt identity extraction, facial landmark estimation, reconstruction, or downstream editing and animation models (Liang et al., 2023; Liang and Wu, 2023b; Salman et al., 2023b; Jeong et al., 2025; Wang et al., 2025). This line of work is intuitive because the face image is the most direct source for identity modeling and visual synthesis.

Despite their effectiveness, visual-domain defenses face two major limitations in the context of talking face protection. First, human faces are highly structured and socially sensitive visual objects, so even subtle perturbations may noticeably reduce visual quality or introduce undesirable facial artifacts. Second, such perturbations are often fragile under common real-world transformations, including resizing, resampling, and platform-side compression, which are almost unavoidable during online media sharing and redistribution. These issues are especially problematic for 3D-field TFG, where accurate visual priors are often reconstructed through long video preprocessing pipelines and may be partially restored or purified during fitting.

Compared with prior work, our goal is not to further optimize visual perturbations, but to revisit where protection should be imposed. We argue that, for audio-driven talking face generation, the driving audio itself provides a more natural and practical intervention point. Instead of perturbing visually sensitive facial regions, we shift protection to the audio modality and study how imperceptible audio perturbations can suppress reliable facial animation while preserving portrait appearance. To the best of our knowledge, this perspective remains largely underexplored in proactive defense for talking face generation, especially for modern 3D-field TFG systems.

Our work is also related to recent efforts on voice clone defense and adversarial protection for speech generation systems. Prior studies have shown that speech models, including automatic speech recognition (ASR), speaker verification, and voice cloning systems, are vulnerable to carefully designed perturbations in the audio domain (Carlini and Wagner, 2018; Qin et al., 2019; Schönherr et al., 2018; Neekhara et al., 2019; Yakura and Sakuma, 2018). To improve perceptual stealth, several works further incorporate psychoacoustic principles and human auditory masking models to constrain perturbations below perceptual thresholds (Qin et al., 2019; Schönherr et al., 2018). These studies demonstrate that the audio modality provides a principled path to achieving both attack effectiveness and human imperceptibility.

However, defending against audio-driven talking face generation is fundamentally different from defending against pure speech or speaker models. In our setting, the audio signal is not the final target itself, but a cross-modal control signal that drives facial geometry and lip motion. Therefore, the objective is not merely to alter linguistic content or speaker identity, but to disrupt the audio-to-geometry mapping required for realistic facial animation. This introduces a new challenge: the perturbation must remain imperceptible to listeners while still being strong enough to interfere with the downstream visual generation process. Our work bridges this gap by introducing a psychoacoustically guided audio defense specifically designed for 3D talking face generation, connecting voice-side imperceptible perturbation design with portrait privacy protection.

Unlike 2D-based talking head synthesis that operates on image pixels, 3D-field TFG models represent the human head as a continuous or discrete dynamic 3D structure. This approach ensures multi-view consistency and realistic geometric deformation. The common pipeline involves mapping acoustic features to a 3D spatial representation, which is then rendered into 2D images.

• •

  Volumetric Fields: (NeRF-based) The scene is represented as a continuous function $f:(\mathbf{x},\mathbf{d})\to(\sigma,\mathbf{c})$. To accelerate rendering, Tri-planes ($\mathcal{P}_{xy},\mathcal{P}_{xz},\mathcal{P}_{yz}$) are often employed, where features at coordinate $\mathbf{x}$ are aggregated from orthogonal projections:

  (1)

  $$\boldsymbol{f}_{\mathbf{x}}=\operatorname{Agg}_{i\in\{xy,xz,yz\}}\left(\mathcal{I}(\mathcal{P}_{i},\pi_{i}(\mathbf{x}))\right)$$

• •

  Point-based Fields: (3DGS-based) The scene is explicitly modeled by a collection of 3D Gaussians, each defined by its center $\boldsymbol{\mu}$, opacity $\alpha$, and covariance matrix $\boldsymbol{\Sigma}$. Audio features drive the facial motion by predicting residual offsets $\Delta\boldsymbol{\mu}$ or modulating a canonical deformation field.

• •

  Hybrid Fields: These models leverage 3D Morphable Models (3DMM) to provide explicit structural priors ($\boldsymbol{\alpha},\boldsymbol{\beta},\boldsymbol{\delta}$), integrating parametric facial geometry with neural feature volumes to achieve both high-fidelity rendering and precise expression control.

To achieve imperceptible protection, we must ensure that the adversarial noise added to the audio remains beneath the detection threshold of the human auditory system. This is governed by the Psychoacoustic Masking Effect.

• •

  Adversarial Intent: The adversary seeks to exploit the victim’s acoustic identity to drive a digital avatar, ensuring that the synthesized lip movements are perfectly synchronized with the victim’s speech to facilitate social engineering or misinformation.

• •

  Black-box Assumptions: We assume a stringent black-box setting where the adversary has no prior knowledge of our defensive mechanism, its parameters, or the underlying psychoacoustic constraints. This forces the adversary to rely solely on the perceived quality of the audio for synthesis.

• •

  Channel Robustness: Considering the practical transmission of audio over the Internet, we assume the adversary operates on a version of $x$ that has undergone standard lossy compression (e.g., MP3 or AAC transcoding) and resampling. Our threat model stipulates that a successful defense must remain effective even after such signal distortions, which are typical of social media pipelines.

The key idea of our method is to shift the defense from the visual domain to the audio domain. Instead of directly injecting perturbations into facial pixels, we optimize a frequency-domain perturbation on the driving speech such that the resulting audio remains perceptually natural to humans, but induces misleading semantic cues for the downstream talking-face generator. Once the perturbed audio is used to drive facial reconstruction, the generated mouth motion becomes structurally inconsistent with the expected speech articulation, thereby breaking the learned audio-to-lip correspondence required by personalized 3D talking face generation.

As shown in Fig. 4, the upper branch first transforms the clean waveform into the time-frequency domain, computes a psychoacoustic capacity bound from the masking threshold, and optimizes a learnable perturbation variable within this admissible region. The perturbed spectrum is then converted back to the time domain to obtain the adversarial waveform. The lower branch feeds the adversarial waveform into a frozen audio feature extractor and a frozen pretrained audio-driven 3D-field renderer. A differentiable landmark extractor is finally applied to the rendered frames, and the resulting mouth closure loss is back-propagated to update the perturbation. In this way, the optimized perturbation does not visibly alter the shared video, yet it poisons the semantic driving signal used by the talking-face model.

We first construct the perturbation in the frequency domain, since psychoacoustic masking constraints are naturally defined over time-frequency bins. Given a clean waveform $x\in[-1,1]^{T}$, we compute its short-time Fourier transform:

where $S_{x}\in\mathbb{C}^{F\times T_{s}}$ denotes the complex spectrum, with $F$ frequency bins and $T_{s}$ temporal frames. We further decompose it into magnitude and phase, and define the phase template as

Based on the clean spectrum, we estimate the psychoacoustic masking threshold map $\Theta_{x}$, which specifies the maximum imperceptible perturbation energy at each time-frequency location. From $\Theta_{x}$, we derive a perturbation capacity map $C$, which provides an upper bound for the perturbation amplitude. To ensure that the optimization always remains within the admissible psychoacoustic region, we introduce a learnable variable $B$ and parameterize the perturbation amplitude as

where $\odot$ denotes element-wise multiplication. The $\tanh(\cdot)$ mapping constrains each element of $B$ to $(-1,1)$, so that the resulting perturbation amplitude $A$ never exceeds the psychoacoustic budget defined by $C$.

We then construct the complex perturbation

and add it to the clean spectrum to obtain the adversarial spectrum

Finally, the adversarial waveform is reconstructed by inverse STFT:

This design corresponds to the upper branch of Fig. 4. It also matches the first several lines of Algorithm 1: we compute $S_{x}$, the phase template $P$, the threshold map $\Theta_{x}$, the perturbation bound $C$, and optimize the learnable variable $B$ through iterative updates.

After obtaining the adversarial waveform $x^{adv}$, we use it to drive a frozen audio-driven 3D-field talking face model. This stage corresponds to the lower branch of Fig. 4. Specifically, the adversarial waveform is first mapped into latent audio features:

where $\mathcal{E}$ denotes the frozen audio feature extractor. These features are then fed into a pretrained audio-driven renderer $\mathcal{R}$ together with the visual conditioning of the target identity to synthesize talking-face frames:

where $d_{i}$ denotes the visual or camera-related condition of frame $i$, and $a_{i}$ is the audio condition derived from $F^{adv}$.

The central objective of our method is not to suppress the visual content itself, but to establish a semantic conflict between the perturbed audio cues and the expected mouth articulation. To quantify this conflict, we apply a differentiable landmark detector $\mathcal{D}$ to each rendered frame and obtain the mouth landmarks:

Let $\phi_{upper,i}^{(y)}$ and $\phi_{lower,i}^{(y)}$ denote the vertical coordinates of the selected upper and lower inner-lip landmarks in frame $i$, respectively. We define the mouth deconstruction loss as

Minimizing $L_{mouth}$ encourages the rendered mouth to collapse toward a closed state, even when the original clean speech should correspond to open-mouth articulations. This mismatch forces the model into an inconsistent audio-to-geometry mapping regime, thereby weakening its ability to learn stable and accurate lip motion patterns from the protected data.

This process is exactly reflected in the middle part of Algorithm 1: after generating $x^{adv}$ and extracting $F^{adv}$, the renderer synthesizes frames one by one, the landmark detector analyzes each frame, and the mouth-related loss is accumulated over all rendered frames.

To jointly enforce attack effectiveness, perceptual naturalness, and spectral smoothness, we optimize the perturbation using four complementary objectives.

The full optimization procedure is summarized in Algorithm 1. Since the attack objective and the perceptual objective have inherently different priorities, we adopt an asynchronous dual-loop strategy instead of optimizing all losses with a single static weighting scheme.

To ensure a fair and practical evaluation, we collect the demonstration videos used in prior works and construct a subset of the HDTF (Zhang et al., 2021) dataset consisting of 11 identities. This setup balances computational cost and evaluation fairness, while covering diverse speaking styles, facial structures, and mouth motion patterns.

We compare our method against two representative baselines: antifake (Yu et al., 2023), a visual-domain proactive defense method, and silencer (Gan et al., 2025), an audio-removal based defense strategy. These baselines represent two fundamentally different protection paradigms: adding perturbations to the visual signal versus directly suppressing the driving audio.

For evaluation, we adopt both generation-quality and perceptual-audio metrics. We denote metrics with the prefix “M” to indicate that they are computed over the mouth region, which is most relevant to speech-driven lip dynamics.

Specifically, we use M-LMD, M-SSIM, and M-PSNR to evaluate the impact of different defenses on generated talking-face videos, where all metrics are computed with respect to the outputs produced from clean reference videos. M-LMD measures the distance between mouth landmarks and reflects lip motion consistency, with lower values indicating better preservation of natural articulation. M-SSIM and M-PSNR measure structural similarity and pixel-level fidelity within the mouth region, respectively, where higher values indicate better visual quality.

In addition, we adopt CDPAM (Manocha et al., 2021) to measure the perceptual distance between the defended audio and the clean audio, where lower values indicate higher perceptual similarity.

Table 1 reports the quantitative comparison across different defense methods. Since the objective of proactive defense is to prevent the protected media from serving as a reliable driving signal for talking-face generation, stronger defense performance should lead to generated results that deviate more from those produced using clean reference inputs. Therefore, larger M-LMD and smaller M-SSIM/M-PSNR indicate stronger disruption of speech-driven facial synthesis, while lower CDPAM indicates higher perceptual similarity between the defended and clean audio.

Compared with antifake, our method achieves a substantially lower CDPAM score, indicating that the introduced perturbation is significantly less perceptible in the audio domain. At the same time, it maintains a comparable level of defense effectiveness in terms of M-LMD, M-SSIM, and M-PSNR, showing that our method can effectively interfere with downstream talking-face generation without noticeably compromising perceptual quality.

Although silencer achieves the strongest disruption according to M-LMD, M-SSIM, and M-PSNR, this result mainly stems from directly degrading the protected visual input itself. Since the quality of the visual input is severely compromised, the downstream generated videos also deviate more significantly from those driven by clean reference inputs, which naturally leads to stronger disruption metrics. However, this comes at the cost of substantially worse visual appearance and user experience. As also evidenced by the qualitative comparisons in Fig. 5, such a strategy introduces obvious perceptual degradation and produces visually less acceptable protected content.

In contrast, our method achieves effective defense while preserving the natural appearance of the shared visual content. This makes it more suitable for realistic deployment scenarios, where proactive protection should not require sacrificing the viewing experience of benign users. Overall, these results demonstrate that our method provides a more practical trade-off between defense effectiveness and perceptual quality.

Figure 5 presents qualitative comparisons between our method and representative visual-domain defense approaches (Liang and Wu, 2023a; Salman et al., 2023a; Liang et al., 2023; Gan et al., 2025; Xue et al., 2023). As shown in the figure, existing visual perturbation methods inevitably introduce visible artifacts on the face, including grainy noise, color inconsistency, and corrupted local textures. These distortions are particularly evident in identity-sensitive regions such as the cheeks, lips, and facial contours, which significantly degrade visual realism and user experience.

In contrast, our method achieves visually lossless protection, where the facial appearance remains clean, natural, and free of noticeable artifacts across different identities. The overall texture, color distribution, and facial structure are well preserved, making the defended content perceptually indistinguishable from clean data for human observers.

Despite this high visual fidelity, our method can still effectively disrupt the speech-driven synthesis process, demonstrating that strong protection does not necessarily require sacrificing visual quality. This property is crucial for real-world deployment, where user-uploaded content should remain visually acceptable while being resistant to unauthorized talking-face generation.