PLUME: Parameter-Efficient Personalization of Large Language Models via Low-Rank User Modulation in Shared Subspaces

We consider the task of personalized text generation, where the goal is to find a unique adapter for each individual to generate user-specific textual responses conditioned on both the current query and contextual information.

Data Construction. Let there be $N$ users denoted by $\{u_{1},u_{2},\dots,u_{N}\}$. Each user $u_{i}$ has $M_{i}$ pairs of history of query–response:

where $q_{i,j}$ represents the $j$-th query describing a specific task or instruction, and $a_{i,j}$ denotes the corresponding ground-truth response. Each response is tokenized as:

where $a_{i,j}^{(t)}$ denotes the $t$-th token and $T_{i,j}$ its token-wise length.

To both emulate the real deployment regime and conform to the established best practice of few-shot in-context prompting, for each query $q_{i,j}$, we retrieve its top-$k$ similar historical queries $\mathcal{T}_{i,j}=\text{TopK}\big(\text{Sim}(q_{i,j},q_{i,\cdot})\big)$ from the same user and pair them with their corresponding responses to construct the personalized context:

where $\text{Sim}(\cdot)$ could be any function that measures query-level similarity. Each instance is therefore represented as a triplet $(q_{i,j},\mathcal{C}(q_{i,j}),a_{i,j})$.

Training Objective. We adopt Decoder-Only LLMs for training where a pre-trained large language model parameterized by $\Theta$ serves as the shared backbone, and each user $u_{i}$ is associated with a lightweight personalized adapter parametrized by $\theta_{i}$. The full model for user $u_{i}$ is denoted as $f_{\Theta,\theta_{i}}$, which conditions generation on both the shared backbone and user-specific adaptation.

For user $u_{i}$, the model is trained to predict each token in the user’s response given all previous tokens, the current query, and the retrieved context:

where $\mathcal{A}_{\text{other}}$ represents auxiliary information from other users (Note: We formulate a general LLM personalization task in Eq.1, where $\mathcal{A}_{\text{other}}$ could be in any form, such as other user’s textual histories. In this work, we encode $\mathcal{A}_{\text{other}}$ into global Space $A^{(l)}$ and $B^{(l)}$ as a learned task-specific representation shared by all users). During optimization, $\Theta$ is shared or fixed, while $\theta_{i}$ is optimized to minimize the user-specific loss:

Inference. At test time, given a new query $q_{i}^{\text{test}}$ for user $u_{i}$, we retrieve its top-$k$ similar query–response pairs from $\mathcal{H}_{i}$ to form $\mathcal{C}(q_{i}^{\text{test}})$. The shared backbone combined with the learned personalized parameters is then used to generate the predicted response:

and the output $\hat{a}_{i}^{\text{test}}$ is compared with the ground-truth $a_{i}^{\text{test}}$ using standard evaluation metrics.

We follow the One PEFT per User (OPPU Tan et al. (2024)) training framework, which assigns an independent PEFT module to each user for personalized fine-tuning.

Concretely, OPPU first ignores user differences and trains a task-specific global LoRA adapter using the union of all users’ training data $\bigcup_{i}\mathcal{H}_{i}$, resulting in a shared adapter $\Delta\Theta$. This stage corresponds to optimizing

The Second step simply follows Eq  2, but with merged model $\Theta\leftarrow\Theta+\Delta\Theta^{*}$.

If we further examine this paradigm through the lens of a specific target module, e.g., $q_{\text{proj}}$, the resulting parameter composition becomes more explicit. Typically, the effective weight matrix on a layer can be expressed as:

where $W_{0}^{(l)}$ denotes the shared base model weight, $(A^{(l)},B^{(l)})$ represents the global LoRA adapter trained from the aggregated non-personalized data $\bigcup_{i}\mathcal{H}_{i}$, and $(A^{(l)}_{u},B^{(l)}_{u})$ denotes the user-specific LoRA adapter paramterized from $\theta_{i}$.

Vanilla OPPU provides a straightforward way to incorporate user-specific knowledge, nevertheless, it faces two major limitations. First, maintaining a separate LoRA module for every user leads to prohibitively high storage and memory costs, especially when the number of users is large. Second, since each user’s training data can vary drastically in scale and quality, independently optimizing a full-rank LoRA for every user often causes overfitting or unstable personalization performance. As illustrated in Figure 2(a), the performance of OPPU drops significantly when the individual LoRA rank is reduced.

These issues highlight the need for a more parameter-efficient and stable approach to capture user-specific preferences without requiring a full LoRA module per user.

Table 1 summarizes the key symbols used in this paper, covering both the preceding problem formulation and the following method sections.

Revisiting Task Space. From a linear algebraic perspective, the task-specific LoRA update on layer $l$ can be expressed as $\Delta W^{(l)}=s\,B^{(l)}A^{(l)}$, where $A^{(l)}$ projects inputs into a low-dimensional task subspace, and $B^{(l)}$ expands the projected representation back to the model’s hidden space. Hence, for any input $x$, the LoRA output $\Delta W^{(l)}x=s\,B^{(l)}A^{(l)}x$ always resides in the column space $\mathrm{col}(B^{(l)})$. OPPU extends this by assigning each user an individual adapter $(A_{u}^{(l)},B_{u}^{(l)})$, which effectively introduces a new output subspace basis $\mathrm{col}(B^{(l)}_{u})$ specific to each user. From this view, OPPU increases the expressive capacity of the model by expanding the dimensional coverage of $\mathrm{col}(B^{(l)})$ across users, thereby enhancing personalization.

However, this completely “separates” the shared task space $\mathrm{col}(B^{(l)})$ and the augmented individual space $\mathrm{col}(B^{(l)}_{u})$, leaving the shared adapter only learns how to project tokens into the task subspace, while not being able to control how different users behave within the collaboratively trained existing shared task subspace $\mathrm{col}(B^{(l)})$. This motivates us to allow user-dependent modulation of the shared subspace.

User-Conditioned Subspace Mixer ($c_{u}$). Intuitively, since users may differ in style, intent, or preference, the way they combine or emphasize task-subspace directions should vary. To capture user-specific task space utilization, we insert a lightweight matrix $c_{u}^{(l)}\in\mathbb{R}^{r_{g}\times r_{g}}$ between $A^{(l)}$ and $B^{(l)}$:

We term $c_{u}^{(l)}$ a User-Conditioned Subspace Mixer (USM). It can be interpreted as a re-indexing or re-weighting operator that re-combines latent task directions inside the shared LoRA subspace. Thus, $c_{u}^{(l)}$ enables personalized modulation of how a user exploits the common task subspace $\mathrm{col}(B^{(l)})$, offering additional expressivity with negligible parameter cost.

PLUME. Before we move on to a more compact model, we first assess how much expressivity the USM alone provides, we introduce a lightweight per-layer residual term aiming to approximate the representational power of high-rank OPPU:

Here $(a_{u}^{(l)},b_{u}^{(l)})$ follow the same formulation as $(A_{u}^{(l)},B_{u}^{(l)})$, but with a reduced individual rank $r_{\text{ind}}$. Empirically, we find that when $r_{\text{ind}}\!\approx\!4$, PLUME already achieves performance comparable to OPPU with rank 64, as shown in Fig 2(a). Interestingly, further increasing $r_{\text{ind}}$ provides no additional gains—indicating that once the shared task subspace is efficiently utilized via $c_{u}^{(l)}$, a large individual subspace becomes redundant and inefficient, leading to overfitting. This observation suggests that there may still exist parameter redundancy even within the reduced adapters. Thus, we further push PLUME to the extreme—seeking the most compact yet expressive form of individual representation. As illustrated in Fig 2(b), our proposed novel shared subspace mechanism enables PLUME to reduce the individual parameter while maintaining highly compatible performance.

CKA Analysis for Layer-Wise Representation Similarity for Personalized Model Inspired by prior work He et al. (2025); Kopiczko et al. (2023); Zhou et al. () that sharing weights across layers can enhance model expressiveness while reducing parameter usage, we investigate whether different layers in personalized LoRA models actually learn similar user-specific representations. To this end, we perform a Centered Kernel Alignment (CKA), a widely used approach for comparing neural representations across layers and models  Kornblith et al. (2019); Liu et al. (2025b) to quantify layer-wise representational similarity across the personalized adapters (See Appendix A.3 for more about CKA).

As shown in Figure 3, OPPU-trained adapters exhibit remarkably high inter-layer CKA values (often exceeding $0.9$), indicating that the personalized LoRA subspaces learned by different layers are strongly aligned. This suggests that the layer-wise residuals $\{a_{u}^{(l)},b_{u}^{(l)}\}$ tend to encode similar directions of user-specific variation, leading to redundant parameterization across depth. Motivated by this finding, we hypothesize that a shared subspace across layers could capture the dominant personalized factors more efficiently.

PLUME-s. Building on this insight, we observe that the per-layer residuals $(a_{u}^{(l)},b_{u}^{(l)})$ often act similarly to layer-specific biases that adjust the shared projection directions. Accordingly, we decompose $(a_{u}^{(l)},b_{u}^{(l)})$ into two functional components: (a) a Personalized Cross-Layer Shared Subspace (PCLS), $(\mathcal{A}_{u},\mathcal{B}_{u})$) capturing cross-layer user-specific residuals in a low-rank shared form, and (b) a rank-1 layer-wise Residual (Resid), $(\alpha_{u}^{(l)},\beta_{u}^{(l)})$ that enables fine-grained local adaptation. The resulting formulation becomes:

Here $(\mathcal{A}_{u},\mathcal{B}_{u})$ with rank $r_{\textbf{sh}}$ parameterize the Layer-Shared Personalized Subspace reused across all layers, while $(\alpha_{u}^{(l)},\beta_{u}^{(l)})$ serve as the rank-1 layer refinements for fine adjustment. Extensive experiments demonstrate that leveraging a shared residual subspace allows substantial reduction of individual residual ranks without compromising performance, and can even outperform OPPU in certain settings (see Sec. 4).

Summary and Parameter Efficiency. Let $L$ be the number of layers and $r_{g}$ the global LoRA rank, and $d$ the embedding vector dimension. In conclusion, PLUME adaptively unifies four complementary components: a global task-specific adapter $(A^{(l)},B^{(l)})$ shared by all users, and three lightweight personalized modules. (1) The User-Conditioned Subspace Mixer (USM) $c_{u}^{(l)}$ adaptively reuses the global task space with $O(Lr_{g}^{2})$ parameters. (2) The Personalized Cross-Layer-Shared Personalized Subspace (PCLS) $(\mathcal{A}_{u},\mathcal{B}_{u})$ captures cross-layer personalization compactly with $O(r_{\text{sh}}d)$ parameters. (3) The rank-1 personalized residuals $(\alpha_{u}^{(l)},\beta_{u}^{(l)})$ restore fine local flexibility using $O(Ld)$ parameters. In total, the per-user complexity is $O(Lr_{g}^{2}+r_{\text{sh}}d+Ld),$ compared to $O(Ldr_{\text{oppu}})$ for OPPU’s layer-wise independent adapters. When $r_{\text{sh}}\!\ll\!r_{\text{oppu}}$, the parameter reduction becomes particularly significant, see Table 3 for practical analysis.

Datasets We adopt the widely used benchmarks LongLaMP Kumar et al. (2024) and LaMP Salemi et al. (2023) to evaluate our model’s ability across both short-form and long-form content personalized content generation tasks. Specifically, we select 3 datasets: Abstract Generation, Product Review, and Topic Writing from LongLaMP, and 2 datasets: News Headline generation, Scholarly Title generation from LaMP. We leave details on data preprocessing and data statistics to Appendix B

Baselines. We compare our methods against a series of strong parameter-efficient fine-tuning baselines under both non-personalized training and personalized training settings. For the non-personalized setting, we compare against the base model, and LoRA Hu et al. (2021), AdaLoRA Zhang et al. (2023), PiSSA Meng et al. (2024), and QLoRA Dettmers et al. (2023) and LoRA-One Zhang et al. (2025b) which all adapt model parameters through a shared low-rank adapter across users. For the personalized setting, we consider retrival-based approach such as RAG Tan et al. (2024), training-based baseline OPPU Tan et al. (2024), and a more recent work CoPE Bu et al. (2025). Finally, we evaluate our two variants: PLUME and PLUME-s.

Implementation To ensure fairness, all models are evaluated with a consistent LoRA configuration (rank = 64) across both non-personalized and personalized settings, including the OPPU and CoPE baselines. To demonstrate the generalizability and robustness of our approach, we evaluate PLUME on two distinct open-source large language models: LLaMA2-7B Touvron et al. (2023) and Mistral-7B-Instruct-v0.2 Jiang et al. (2023). As illustrated in Figure 4(b) in the later discussion, PLUME already reaches competitive results with a very small rank, so we report PLUME results with residual rank $4$, and PLUME-s with residual rank $1$ and shared rank as low as $8$. Due to space constraints, we present the results using the Mistral-7B backbone in the main text and leave more experiment results and details in Appendix C.

Evaluation Metrics In accordance with established practices in prior work Tan et al. (2024); Kumar et al. (2024), we employ a standard set of automatic metrics ROUGE-1, ROUGE-L Lin (2004), and METEOR Banerjee and Lavie (2005) to quantitatively assess the lexical overlap, fluency, and semantic correspondence between generated outputs and reference responses (See more details in Appendix C.1)

Overall Perfomance (RQ1) Table 2 reports the overall performance across five generation tasks. Overall, personalized methods outperform non-personalized ones across all metrics, confirming the advantage of user-specific adaptation. Within the personalized group, despite using far fewer parameters than strong baselines such as OPPU and CoPE, the PLUME series delivers comparable or better results than other baseline methods. For instance, PLUME attains the highest ROUGE-L on Abstract Generation (0.2395) and comparable performance on other long-text tasks, demonstrating that the personalized module USM would substantially enhances expressiveness. When incorporating the shared subspace, PLUME-s performance slightly improves, e.g., ROUGE-1 rises from 0.2150 to 0.2427 on News Headline Generation Tasks. This indicates that the shared subspace not only compresses parameters but also mitigates redundancy and overfitting issues observed in OPPU.

In summary, PLUME achieves a favorable balance between expressiveness and efficiency. It retains nearly identical generation quality to strong personalized baselines OPPU and CoPE while using only a fraction of their parameters, and even surpasses them on several short-text tasks.

Efficiency Comparison (RQ2) One of our main contributions is that PLUM substantially improves parameter efficiency for LLM personalization adapters. Here, we quantify the extent to which this compression frontier can be pushed under practical setting. As in Table 3, compared to OPPU, which requires 168M parameters per user, PLUME and PLUME-s reduce the cost to only 6.8% and 3.1% respectively, achieving over $15\times$–$30\times$ compression while preserving personalization quality and requiring less training time.

Ablation Study: Effectiveness of Components (RQ3)

Figure 4(a) compares the effects of three core modules: USM, PCLS, and Resid on both long and short text generation tasks.

Across both tasks, removing any single module consistently leads to a drop in ROUGE-L and METEOR, suggesting that all three work jointly to enhance expressiveness, coherence, and efficiency. This consistent trend highlights that the modules are complementary rather than redundant. The relative impact of each module remains consistent across both tasks (Resid > PCLS > USM). Moreover, the effect of the Resid module is more pronounced in long-form generation tasks. Notably, even though Resid employs only a rank-1 update, it still contributes sufficiently to representational richness, showing that lightweight residual paths can complement USM’s user-specific modulation effectively. These results demonstrate that each component is indispensable for achieving a strong balance between personalization, coherence, and parameter efficiency.

Parameter Efficiency VS Model Performance (RQ4)
Sensitivity of residual rank. To investigate the sensitivity of residual dimensionality, we vary the rank of the residual module. As shown in Figure 4(b), performance on Abstract Generation (left) first increases with larger residual ranks and peaks around rank 4, after which both ROUGE-L and METEOR scores gradually decline. This trend indicates that a small residual rank is sufficient to capture user-specific variations, while larger ranks introduce redundancy and overfitting. In contrast, the Scholarly Title task (right) shows much smaller variance across ranks, with performance already strong at rank 1 and exhibiting a slight downward trend thereafter. This validates our conjecture that short-form generation requires fewer personalized parameters and benefits more from PLUME’s shared-subspace regularization.

Sensitivity on PCLS rank. To further examine the influence of the shared subspace capacity, we vary the rank of PCLS module. As illustrated in Figure 4(c), the results show an opposite trend between the two settings. For Abstract Generation, performance steadily improves as the shared rank increases, indicating that long-form generation benefits from a richer shared subspace capable of modeling broader contextual dependencies and semantic consistency across layers. In contrast, the Scholarly Title task exhibits an inverted trend—performance peaks at a small shared rank and then declines as the rank grows—suggesting that short-form generation requires only limited shared capacity. These contrasting patterns highlight PLUME’s flexibility that its shared subspace can effectively balance generalization and personalization.