HyperStyler: Low-resource Authorship Style Transfer via Context-aware Style Navigation and Hypernetworks

Let $E^{(t)}=[e_{1};\dots;e_{N_{t}}]\in\mathbb{R}^{N_{t}\times d_{e}}$ be a learnable embedding table associated with a modulation target $t$ (e.g., cross-attention prefix keys). Here, $N_{t}$ is the number of embeddings for target $t$, and each row $e_{j}$ corresponds to a distinct modulation target indexed by a tuple (layer, type, position). Prefix embeddings are indexed by (layer, key/value, prefix position) and adapter embeddings are indexed only by projection type (layer, up/down).

We compute compatibility scores between $z$ and each layer embedding through a multi-head bilinear interaction:

where $(\cdot)^{(h)}$ is the $h$-th head, $W_{e}$ and $W_{s}$ are trainable projection matrices, and $\bar{e}_{j}$ and $\bar{z}$ denote layer-normalized vectors. Each $b_{j}^{(h)}$ is a style-dependent and embedding-specific signal that determines the relative contribution of the corresponding subspace of $z$ to embedding $j$. We then form the offset $o_{j}$ by projecting $\bar{z}$ with trainable matrix $W_{v}$, scaling each head with its corresponding score, and concatenating all $N_{h}$ heads:

Finally, we project the offset $o_{j}$ with trainable matrix $W_{o}$ and apply layer normalization to obtain a stable style-conditioned update $\Delta e_{j}$, which is added to the original embedding $e_{j}$:

We map the style-conditioned layer embeddings $\tilde{E}^{(t)}$ to modulation parameters using two-layer MLPs. We use dedicated generators for different modulation targets, as each target operates on distinct parameter spaces with different dimensionalities and functional roles. Here, $d_{\text{model}}$ denotes the hidden dimensionality of the underlying model.

Cross-attention prefixes modulate how the decoder references the source context during generation Li and Liang (2021). We generate key and value prefix vectors using two independent MLPs. The generated vectors are grouped into length-$p$ prefixes per layer, $P^{\ell}_{K},P^{\ell}_{V}\in\mathbb{R}^{p\times d_{\text{model}}}$, and concatenated to the original keys and values along the sequence dimension, $K^{{}^{\prime}\ell}=[P^{\ell}_{K};K^{\ell}]$ and $V^{{}^{\prime}\ell}=[P^{\ell}_{V};V^{\ell}]$.

Low-rank adapters target FFN layers, which have been shown to store substantial linguistic information Geva et al. (2021); Geva et al. (2022), making them well-suited for controlling surface realization such as lexical choice and syntactic patterns. For each layer $\ell$, we generate low-rank down- and up-projection weights using separate MLPs, $W^{\ell}_{down}\in\mathbb{R}^{d_{\text{model}}\times r}$ and $W^{\ell}_{up}\in\mathbb{R}^{r\times d_{\text{model}}}$. The resulting branch is added to the FFN output:

where $h_{\text{in}}$ and $\sigma(\cdot)$ denote the FFN input hidden states and the activation function, respectively. In our implementation, each MLP outputs a vector of length $r\cdot d_{\text{model}}$, which is reshaped into the corresponding low-rank matrix for each layer.

For each author, we collect an author-specific corpus consisting of a sentence set $X=\{x_{i}\}_{i=1}^{K}$. Using a pretrained paraphraser, we generate a synthetic paraphrase $x^{\prime}_{i}$ for each sentence $x_{i}$, thereby constructing synthetic pairs $\{(x_{i},x^{\prime}_{i})\}_{i=1}^{K}$. To mitigate stylistic bias inherited from the pretrained paraphraser and to improve both diversity and semantic preservation, we train the underlying model with a bidirectional reconstruction objective over $(x_{i}\leftrightarrow x^{\prime}_{i})$ pairs Sjöblom et al. (2020); Ma et al. (2021). The goal of this stage is not to acquire any specific style, but to establish a semantically reliable paraphrasing backbone.

We freeze the underlying paraphraser and integrate it with the Stylo-navigator and Stylo-hypernet to be trained simultaneously through an unsupervised reconstruction task. For each author, the reference set $R=X$ is embedded into the style embeddings $S\in\mathbb{R}^{K\times d}$. The Stylo-navigator is trained to identify the stylistic target within $S$ that best matches $x_{i}$ given the source context $x^{\prime}_{i}$. We use the index $i$ of the target sentence as a ground-truth label and minimize the negative log-likelihood of the predicted selection probabilities $\alpha$:

To prevent the navigator’s prediction errors from propagating to the Stylo-hypernet, we use teacher-forced style conditioning, where the Stylo-hypernet is conditioned on the ground-truth style embedding $s_{i}$ (obtained by encoding $x_{i}$ with the STYLE embedder) rather than the predicted coordinate $z$. This isolates hypernetwork optimization from navigator errors and allows the Stylo-hypernet to focus on learning precise parameter modulation. The Stylo-hypernet is optimized to maximize the reconstruction likelihood of $x_{i}$:

Finally, we optimize the model for style transfer beyond reconstruction, conditioning the Stylo-hypernet on the stylistic coordinate $z$ from the Stylo-navigator. Since parallel data between source and target authors is unavailable, we construct a high-quality parallel dataset via self-distillation Zhang et al. (2019). Specifically, we generate style-transferred outputs using the Stage 2 model and filter them following the Rerank and Filtering procedure from Horvitz et al. (2024b). Unlike prior work that uses a mean-pooled reference embedding, we use predicted $z$ to evaluate style fidelity during filtering. Using the pseudo-parallel data, we jointly train the Stylo-navigator and Stylo-hypernet.

We conduct experiments on three datasets with distinct genres: Reddit Khan et al. (2021), Blog Schler et al. (2006), and News (All-the-news). Following prior works for LAST Patel et al. (2024); Horvitz et al. (2024a); Horvitz et al. (2024b), we segment each author’s corpus into sentences and randomly sample 10 sentences per author. We filter out any samples exceeding 60 tokens and split the data by author into training, validation, and test sets with a 0.9/0.05/0.05 ratio. More details of the datasets are described in the Appendix A.

We use three evaluation sets of Reddit from Patel et al. (2024): Random, Single, and Diverse. Each split comprises 15 source and 15 target authors with 16 samples each, totaling 225 transfer directions and 3,600 transformations. We apply the same configuration to Blog and News by randomly selecting hold-out authors from the test dataset.

To ensure fair comparison, we follow the evaluation protocol established in prior LAST studies Patel et al. (2024); Horvitz et al. (2024b). Away and Towards measure the degree to which the set of transferred texts moves away from the source author’s style and toward the target author’s style, respectively. These metrics are computed using a held-out UAR embedder Rivera-Soto et al. (2021) trained via contrastive learning for authorship verification. To evaluate semantic preservation, we use the Mutual Implication Score Babakov et al. (2022) as Sim. Finally, the Joint score summarizes overall performance of style transfer: $\textsc{Joint}=G(G(\textsc{Towards},\textsc{Away}),\textsc{Sim})$, where $G(\cdot)$ denotes the geometric mean. (see details in C.1)

We compare against baselines across three categories for the LAST task.

In-context learning methods include STYLL Patel et al. (2024) with Qwen2.5, and models prompted with the instructions from Horvitz et al. (2024b), including Llama3.1, GPT4-turbo, and the reasoning enabled GPT5-mini and GPT5.4.

Inference-time control methods include ParaGuide Horvitz et al. (2024a), a diffusion-based model, and StyleMC Khan et al. (2024), which performs Metropolis-Hastings sampling guided by a future regressor.

Unsupervised alignment methods include TinyStyler Horvitz et al. (2024b), which conditions on a mean-pooled style embedding for references and utilizes self-distillation, and ASTRAPOP Liu et al. (2024), a policy optimization conditioning on the entire reference sentences. Beyond its original length-based reward, we also train ASTRAPOP using the Joint as a reward.

To ensure fair comparison, we apply two principles: (1) we unify the backbone to T5-large Raffel et al. (2020) across all trainable baselines so that performance reflects methodological rather than capacity differences, and (2) for baselines that require a style guide, we provide a mean-pooled style embedding rather than the UAR embedding used for evaluation to prevent baselines from directly optimizing the evaluation metric. We utilize off-the-shelf paraphrasing PEGASUS Zhang et al. (2020), following Horvitz et al. (2024b), and set the adapter rank to 32 and the prefix length to 5. Following TinyStyler, we apply reranking Suzgun et al. (2022) at inference time, but use the predicted $z$ as the target style instead of a mean-pooled embedding. Other details are provided in the Appendix B. Our implementation code for HyperStyler is available at https://github.com/JK-SHIN-PG/HyperStyler.

Style transfer requires simultaneously achieving high style fidelity and semantic preservation, as a model biased toward preservation fails to transfer style, while one biased toward style fidelity risks distorting meaning Fu et al. (2018); Horvitz et al. (2024b). As shown in Table 1, HyperStyler strikes the best balance between the two objectives, improving Towards while maintaining competitive Sim scores, resulting in the highest Joint scores across all three domains. Further improvements are observed when reranking is applied.

Most baselines have relatively low performance on the News, as news articles are more formal and exhibit lower stylistic variability, making it more difficult to capture distinctive author-specific styles Eder et al. (2021); Wang and Riddell (2022) (Table 8). Despite this, HyperStyler outperforms all baselines, including LLMs. Additional experimental results including qualitative analysis are provided in Appendix D.

We compare the generalization capability of HyperStyler with TinyStyler, the strongest baseline in our experiments. As shown in Table 2, TinyStyler consistently degrades when transferring from News to Reddit and Blog, where the inter-author distances to the target authors are approximately 2.04$\times$ and 1.68$\times$ larger than those in the in-domain setting, respectively (Figure 5). Given these larger distances and the high style variation in Blog and Reddit, this degradation suggests that a single mean embedding fails to provide sufficiently fine-grained style signals for such large stylistic shifts. In contrast, HyperStyler achieves consistently strong performance across most domain pairs, as its context-aware style selection and parameter modulation enable more precise stylistic adaptation across diverse domains.

We also evaluate in-domain authorship style transfer under out-of-domain training. As shown in Table 3, TinyStyler performs well when trained and evaluated within the same domain, but its performance degrades substantially when applied to other domains. Since the training domain shapes the range and granularity of styles a model learns, and each target domain may require a different level of stylistic precision, some degree of degradation under domain shift is expected. Nevertheless, HyperStyler exhibits relatively limited degradation. Notably, HyperStyler trained only on News achieves Blog$\rightarrow$Blog transfer performance close to that of TinyStyler trained directly on Blog, highlighting HyperStyler’s robust generalization capability.

We examine whether HyperStyler navigates the style space in a context-dependent manner. Specifically, we paraphrase the original texts and have the model reconstruct them in their corresponding styles. As shown in Figure 2, TinyStyler, which is conditioned on a single static embedding, fails to faithfully reproduce the original stylistic distribution, whereas HyperStyler closely matches it. Furthermore, the predicted $z$ achieves a cosine similarity of 0.82 with the original style embedding and a Mean Reciprocal Rank (MRR) of 0.80, substantially outperforming mean pooling (cosine similarity:0.58, MRR: 0.21). These results demonstrate that the Stylo-navigator accurately identifies the context-appropriate style target. We also analyze performance under target-author style variation. Figure 3 shows that HyperStyler remains robust as variation increases, whereas the baselines degrade. This highlights that context-aware style selection enables HyperStyler to effectively handle high stylistic variation within an author’s style.

We compare the Stylo-navigator against two alternative strategies. (1) Mean pooling: We inject the mean-pooled reference embedding into the Stylo-hypernet. (2) Implicit selection: We provide all reference embeddings and perform layer-wise style selection implicitly via cross-attention. The results in Table 4 show that both alternatives exhibit substantially lower Towards scores compared to the Stylo-navigator, demonstrating that explicit style selection conditioned on the source context is effective in improving style fidelity.

We compare parameter modulation against two hidden-state injection strategies. (1) Global: We concatenate predicted $z$ to the encoder hidden states, providing the same style signal to all decoder layers. (2) Layer-wise: We generate layer-specific style embeddings and concatenate them to the encoder hidden states for each decoder layer. The global strategy fails to induce style changes, confirming that layer-wise style control is necessary. More importantly, while layer-wise injection shows some improvement, the Towards/Sim ratio of HyperStyler (0.263) is approximately 37% higher than that of layer-wise injection (0.192), indicating that parameter modulation achieves higher style fidelity for the same semantic cost compared to hidden-state injection. This demonstrates the effectiveness of parameter modulation in realizing style while preserving content.

We compare different combinations of modulation targets. Modulating only the cross-attention (w/o Adapter in FFN) achieves high performance but falls short in style fidelity, while modulating only the FFN (w/o Prefix in CrossAttn) improves style fidelity but degrades content preservation. Modulating the self-attention layers results in broken sentence structures, interfering with the decoder’s generation process. These results suggest that jointly modulating FFN and cross-attention achieves the best balance between style fidelity and content preservation.

We further analyze the effects of varying rank and prefix length. As shown in Figure 4, increasing the rank yields only limited additional benefit, and a longer prefix does not necessarily yield further gains. Across all combinations of rank and prefix length, using both modulation components consistently outperforms variants that remove either the FFN adapter or the cross-attention prefix, further supporting the structural effectiveness of dual modulation.

We evaluate the parameter efficiency of our approach by testing a constrained configuration with a reduced rank and prefix length. As summarized in Table 6, decreasing the number of parameters leads to a slight degradation in performance. Nevertheless, even with the rank and prefix length set to 1, our model still outperforms TinyStyler while adding only 2.4% of the underlying model’s parameters. This result demonstrates that HyperStyler maintains robust performance even under parameter constraints.

HyperStyler achieves approximately one second per inference on a single A100 GPU, with a modest overhead over TinyStyler (Table 17). Compared to open-source LLMs, HyperStyler is over 1.8$\times$ faster and uses less than one-eighth of the VRAM, and over 2.0$\times$ faster than API-based LLMs. Notably, even with reranking applied, HyperStyler remains faster than LLM-based baselines. These results demonstrate that high style transfer performance can be achieved without compromising computational efficiency, highlighting its suitability for time- and memory-constrained practical applications.