Prompt Compression and Contrastive Conditioning for
Controllability and Toxicity Reduction in Language ModelsThanks:   Work done while at Nvidia, Inc.Thanks:    Work done while at Brigham Young University

Language models (LMs), such as GPT-2 Radford et al. (2018); Radford et al. (2019a), BERT Devlin et al. (2018), T5 Raffel et al. (2020), or GPT-3 Brown et al. (2020), exhibit a remarkable ability to capture patterns of grammar, vocabulary, cultural knowledge, and conversational rhythms present in natural language. Formally, a LM is a conditional distribution over tokens $p(x_{t}|x_{1},\cdots,x_{t-1})$, with each token $x_{t}\in\mathcal{V}$ for some vocabulary $\mathcal{V}$. Throughout this paper, we will refer to $x_{h}=x_{1},\cdots,x_{t-1}$ as the prompt.

This paper explores prompt compression: the idea that the text $x_{h}$ used to condition a LM can be approximately represented by a much smaller set of carefully chosen weights, using the framework of soft prompts Lester et al. (2021). We begin by establishing some basic properties of compressed prompts, and importantly show that while highly compressed prompts lose fine-grained information about the prompt, they can retain general, abstract information. This motivates our central application: to use such compressed prompts in a Bayesian attribute framework to steer text generation, with specific application to toxicity reduction.

To motivate this more deeply, we briefly sketch how compressed prompts can be used in toxicity reduction. Efforts to reduce toxicity and bias generally follow one of two strategies: the first is to train or fine-tune LMs on carefully curated data, either tagging or labelling it in special ways Keskar et al. (2019a); Lu et al. (2022) or using data known to be “clean”. The second is to "steer" the generation of token probabilities away from toxic generations Krause et al. (2020); Liu et al. (2021), and towards text with known, desirable properties.

Following previous work, we steer LM probabilities by using a Bayesian attribute classifier framework that involves scoring candidate tokens with different experts. As an independent contribution, we explore the idea of simply using conditioning text to construct such experts by leveraging the few-shot modeling abilities of LMs Radford et al. (2019a); Brown et al. (2020): given a few examples of text containing a pattern of interest, language models are capable of “analyzing” such examples and assign high probability to subsequent text exhibiting the same pattern. Thus, in the same way that language model can, for example, classify the sentiment of a tweet, we use LMs to analyze the toxicity of candidate generations in real-time. Our method can be considered an exemplar-based method of defining experts that capture desirable and undesirable attributes of generated text. We term this technique contrastive contexts, and note that it reduces the problem of creating experts to one of prompt engineering Reynolds and McDonell (2021).

However, our conditioning contexts are quite large, which motivated this work. We use prompt compression to mimic an uncompressed prompt (hereafter referred to as "hard" prompt) as closely as possible, thereby saving both computation and space in the context window. Our results demonstrate that this can be very effective, and, in a very surprising finding, that complex prompts can be reduced to a single token and still be useful for toxicity reduction, often with better fluency compared to hard prompts.

The contributions of this paper are three-fold: first, we introduce and formalize the idea of prompt compression; second, we introduce and formalize the method of contrastive contexts in the Bayesian attribute framework; third, we experimentally evaluate our methods, and refine the technique based on various empirical observations, and contribute a careful study of effectiveness as model size varies.

To the best of our knowledge, this is the first work to directly explore prompt compression. However, our work is based on the original soft prompt ideas of (Lester et al., 2021). It is also somewhat related to distillation, where one model is trained to mimic another by matching logits Gou et al. (2021).

Usually, LMs take text as input, which is then tokenized into discrete tokens by a tokenizer. Each token is then mapped to a learned embedding, which is used as input to a transformer Vaswani et al. (2017). The idea of soft prompts Lester et al. (2021) is to bypass the need to use discrete tokens with pre-trained embeddings and instead directly learn a series of embeddings via backpropagation. These learned embeddings are then fed directly to the transformer and do not need to correspond to any actual language tokens.

As the centerpiece application of prompt compression, we explore generative controllability Keskar et al. (2019b) and toxicity reduction in language models.

Our method is most closely related to decode-time algorithms, such as GEDI Krause et al. (2020), which uses Bayes’ rule and discriminative models to steer the generation towards a certain attribute; and PPLM Dathathri et al. (2019), which uses an estimated gradient with respect to the desired attribute to steer the LM’s internal representation at generation time.

Other methods are based on fine-tuning language models with the classical language modeling objective to steer generation. DEXPERTs Liu et al. (2021) combines experts and anti-experts in a product of experts model to reduce toxicity of LMs.

Additionally, reinforcement learning approaches show strong performance at steering language models Stiennon et al. (2020). By providing rewards, methods such as PPO Schulman et al. (2017) and Quark Lu et al. (2022) represent the current best performance at reducing LM toxicity while maintaining fluency. These methods, however, require a predetermined reward function, which may or may not be feasible depending on the context.

Here, we introduce and explore the idea of prompt compression, whereby the parameters of a soft prompt Lester et al. (2021) are trained to mimic a fixed hard prompt as closely as possible.

The intuition of our idea is simple: conditioning a LM on a hard prompt $x_{h}$ induces a distribution $p(x_{t},\cdots,x_{t+k}|x_{h})$ over all possible subsequent sequences of tokens $x_{t},\cdots,x_{t+k}$ for all $k$. To simplify notation, let $x_{t:k}=x_{t},\cdots,x_{t+k}$. The schematic of the idea is shown in Fig. 1. Formally, a soft prompt is a block of weights $\theta_{n}$ that is prepended to the embeddings of the tokenized sequence $x_{t:k}$, and which is then fed through the transformer layers of the language model. The soft prompt induces a modified distribution over $x_{t:k}$, which we represent as $q(x_{t:k}|\theta_{n})$. Here, $n$ is the number of tokens in the soft prompt (which do not necessarily correspond to natural language tokens).

To compress prompt $x_{h}$, we train the soft prompt weights to minimize the following objective:

where the sequences $x_{t:k}$’s are sentences of various lengths and content drawn from a diverse training set. We optimize this objective using the Adam optimizer for 75,000 steps of training with a linear learning rate schedule starting at 0.1, and $x_{t:k}$’s drawn randomly from The Pile Gao et al. (2021), requiring about 1-4 GPU-hours to train a single prompt, depending on computational complexity of running the LM. All prompt training was done using either a single A100 or V100 GPU.

While training a compressed prompt can be expensive, the gains are found at inference time. Using a compressed prompt over a hard prompt reduces the length of the context. This scales down the needed computation according to the transformer’s attention mechanism, which is $O(n^{2})$. This also could allow long contexts to be compressed and appended to longer inputs than was previously possible. Once trained, these compressed prompts could be shared to create a library of efficient contexts.

We begin by exploring the properties of compressed prompts. First, we show that conditioning on a hard prompt and its compressed prompt generate qualitatively similar generations, although this equivalence degrades as the prompt is compressed more and more. Second, we qualitatively explore what happens to fine-grained information as a prompt is compressed more and more.

Models and codebase. All experiments were conducted using the Huggingface¹¹ 1 https://github.com/huggingface/transformers Wolf et al. (2019) implementation of GPT-2 (117M parameters), GPT-2 medium (345M), GPT-2 large (774M) and GPT-2 xl (1.5B) models.

As an application of compressed prompts, we now turn our attention to toxicity reduction and controllability. Following previous work Dathathri et al. (2019); Krause et al. (2020) we adopt the Bayesian attribute classifier framework for decode-time controllability. Our goal is to generate text that exhibits some attribute $a$; by conditioning generations on this attribute and using Bayes law, we arrive at

where the prior $p(x_{t}|x_{h})$ is simply the vanilla distribution over generations from the LM, and the likelihood term $p(a|x_{1},\cdots,x_{t})^{\omega}$ is known as an attribute classifier. Here, we have also introduced the temperature parameter $\omega$ that modulates the strength of the effect of the attribute classifier.

There are multiple ways to construct the attribute classifier $p(a|x_{h},x_{t})$. If the desired attribute $a$ is, for example, “does not contain profanity”, then the attribute classifier could simply scan $x_{h}$ for words on a blacklist and output $p(a|x_{h},x_{t})=1$ if none of the words are present.

More sophisticated approaches are possible; our approach centers on the careful construction of this classifier. Our work is most similar technically to the GEDI framework Krause et al. (2020), which uses two language models to construct the classifier in a contrastive manner. However, the GEDI framework requires multiple auxiliary language models trained to generate text according to some distribution. Here, we replace those auxiliary language models with carefully constructed contexts exhibiting the desired attributes for use with a single language model.

To empirically assess prompt compression in the context of toxicity reduction, we follow the experimental protocol outlined in the RealToxicityPrompts (RTP) paper Gehman et al. (2020). The RTP paper contributes both a dataset and a variety of metrics for assessing toxicity; we briefly summarize those here. Also following the RTP paper, all toxicity measurements are done with the PerspectiveAPI Jigsaw and the Google Counter Abuse Team (2015), an imperfect Sap et al. (2019) but standard tool for assessing toxicity along a variety of dimensions.

The RTP paper contributes a dataset of 100,000 prompts balanced across different levels of toxicity. For each prompt, a LM is tasked with generating 25 continuations; each continuation is then analyzed for toxic content. There are two primary metrics of interest. The first is the expected maximum toxicity, where the max toxicity is taken over the 25 generations, and second is the average toxicity.

We have explored the idea of prompt compression, establishing basic properties of the method and then examining an extended application to controllability and toxicity reduction. Based on our experiments, we conclude that prompts can be significantly compressed and still retain some useful information. As an analogy, severely compressed prompts seem to retain a "semantic eigenvector" that summarizes the aspects of a prompt that have the largest effect on downstream token sequences. This suggests that representing information as basic tokenized sentences is inefficient, and that more general prompt compression strategies may be possible (for example, by training a prompt-compressing deep neural network). We see that compressed prompts generally exhibit the properties we would expect them to, such as naturally retaining the most important information as they are compressed further and further.

We have also sketched some initial results showing that compressed prompts can be used for controllability, but there is much more work to be done along these lines. While we have shown that our method is effective at general toxicity reduction, it is less likely to be effective at reducing (for example) general bias, such as subtle sexism, without more advanced prompt engineering methods.

Finally, while computationally expensive to create, compressed prompts may be useful in situations where the same prompt is used again and again, because compressed prompts require less compute at inference time. Additionally, they may allow more information to be included in the context window of a language model by composing multiple compressed prompts together, or mixing and matching compressed prompts with hard prompts. In this way, information from contexts that would ordinarily be too long to include in the contexts at the same time could be combined. Ultimately, however, the possibilities and limitations of the method are an open question.

This material is based upon work supported by the National Science Foundation under Grant No. 2141680. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

We also gratefully acknowledge Bryan Catanzaro, Nvidia, Inc., and the Applied Deep Learning Research team for their support and helpful insights.

The method of prompt compression has a variety of limitations. Here, we summarize a few of the most noteworthy.

Computational limitations: compressing a single prompt currently requires several hours of compute on state-of-the-art hardware. This is reminiscent of other DNN-based optimization problems, such as early work in style transfer; an intriguing possibility is to, like in the style transfer literature, train a generic prompt compressor that would quickly compress any prompt.

Theoretical limitations: we have sketched a variety of properties and applications for compressed prompts, but there is currently no theoretical characterization of the method. Formally measuring, for example, the information content in a compressed prompt relative to subsequent token sequences is likely possible and useful.

Application limitations: our method yields state-of-the-art toxicity reduction for decode-time methods. Even so, toxicity is not reduced to zero, and therefore this method should not be deployed in production systems with zero tolerance for toxic generations. In addition, other types of methods (e.g., Quark Lu et al. (2022)) provide better overall toxicity reduction, at the cost of modifying the LM’s weights; in cases where modifying the weights is possible (and no dynamic changes are needed, or where no compositionality of multiple steering directions is needed), those methods should be preferred.

While we hope that our work can enable positive downstream applications, such as toxicity reduction, we realize that the method can be trivially applied to increase toxicity or any other undesirable characteristic. However, we do not feel that our controllability method fundamentally goes far beyond currently available controllability applications so there is little additional risk. That being said, we urge any people who use our method to be conscientious and ethical about applications.

Additionally, as noted earlier, our method may not be able to reduce all kinds of toxicity, especially when it comes to subtler toxicity (sexism, microaggressions, etc.) Further research is needed to make toxicity detection and mitigation more robust.