arXiv is now an independent nonprofit! Learn more
License: CC BY 4.0
arXiv:2607.00083v1 [cs.CL] 30 Jun 2026

Harnessing the Latent Space:
From Steering Vectors to Model Calibrators for Control and Trust

Nishant Subramani [Uncaptioned image] [Uncaptioned image] Affiliation: [Uncaptioned image] [Uncaptioned image]: Carnegie Mellon University, Language Technologies Institute Email: $nishant2@cs.cmu.edu$
Abstract

Language models have changed from unreliable text generators to highly-capable large models with trillions of parameters. Capability increases come hand-in-hand with increases in scale, making understanding the internal representations of models more challenging. Since millions of users increasing rely on language models to interact with external tools or make decisions in medium or high-stakes scenarios, we need to establish control over model behavior and know when to trust model outputs. In this paper, we discuss our contributions on harnessing the latent spaces by proposing steering vectors for control and developing latent space-based model calibrators for trust. Together, our contributions help demystify the latent spaces of language models and offer new insights into how to harness model internals to build more trustworthy language technology.

1 Introduction

Neural network language models (LMs) have evolved from small, unreliable text generators to very large models capable of solving complex reasoning tasks (Peters et al., 2018; Radford et al., 2019; Groeneveld et al., 2024; Yang et al., 2025; Team et al., 2025, inter alia). Despite the vast capability increases, analyzing the internal representations of trillion-parameter models is challenging. Due to this, the NLP community has increasingly treated models as black boxes, neglecting understanding the inner-workings of models. Even though large language models (LLMs) are scaled to millions of users, increasingly interact with external tools (Qu et al., 2024), and make decisions in medium and high-stakes scenarios (Thirunavukarasu et al., 2023), we rely by-and-large on simple behavioral observation (Hendrycks et al., 2021; Srivastava et al., 2023; Liang et al., 2023, inter alia). As a community, we must build fundamental understanding of the inner-workings of models and operationalize the internal representations of LLMs. We need to establish control over model behavior to ensure safety and alignment and establish confidence estimation mechanisms which can accurately adjudicate trust.

We present four threads of research aimed to demystify and harness the latent spaces of language models. To achieve model control, we show that LSTM-based language models can be minimally steered for exact generation (§ 2). We then adapt to transformer-based models in § 3, showing both fine-grained and coarse-grained control via exact and concept-based steering. Shifting to trustworthiness, we build model-internal confidence estimators (mice) to calibrate LLM generations in tool-calling scenarios (§ 4). Lastly, we broaden the framework to new model families and tasks by proposing activation-based confidence, utility, and trust estimators (acute; § 5). Together, our contributions offer actionable recipes to harness model internals to build more controllable, well-calibrated, and trustworthy language technologies.

Refer to caption
Figure 1: Our contributions on harnessing the latent spaces of language models: § 2 and § 3 focus on control, proposing steering vectors for the first time for LSTMs and transformer-based models. § 4 and § 5 focus on trust, building model-internal confidence estimators to assess confidence of language model output generations.

2 Control: Steering Vectors for LSTMs (Subramani et al., 2019)

We focus on control, specifically trying to answer one key question:

Key Question 1 Can LSTM-based language models be steered to generate a desired sequence exactly without updating a single parameter?
Refer to caption
Figure 2: Here we show how the steering vector zs​t​e​e​rz_{steer} can be injected into an LSTM-based language model (left) and a transformer-based one (right). On the left, Z is shown to have a larger dimension than the model dimension. If dim(ZZ) equals the model dimension, K=1K=1, and thus there is just one vector z1z_{1}.

2.1 Prior Work

In 2018, the transformer architecture proposed in Vaswani et al. (2017) had yet to fully permeate the language model landscape and long short-term memory models (LSTMs; Hochreiter and Schmidhuber, 1997) were still the predominant architecture for language modeling. These language models were unreliable text generators. However, they have the potential to learn useful representations, so LMs started to be seen as general-purpose encoders (Dai and Le, 2015; Peters et al., 2018; Devlin et al., 2019, inter alia). Precise control of language model output, on the other hand, remained far out of reach, primarily due to the low quality of the underlying language models of the time.

2.2 Our Contributions

We explore whether language models could be used as general-purpose decoders, something that we now take for granted, but at the time was an unknown. For a pretrained language model to be used a general-purpose decoder, we need (1) to find a continuous-valued sentence representation (a steering vector) that can be fed into the frozen language model, (2) an encoder, likely task-specific, that can convert task inputs into steering vectors, and (3) for those steering vectors to causally generate the desired output. At the time, no work had shown this was possible, but now we take this for granted with advances in prompting and decoder-only LLMs. In our work, we explore the possibilities of this in LSTM-based models, before prompting became popular. Specifically, we ask whether LSTM-based models can be steered to generate a desired sequence exactly while keeping the underlying language model frozen.

Background

To ask this, we first define the sentence space of a recurrent language model. Since the recurrent transition function fθ=ℝd×V→ℝdf_{\theta}=\mathbb{R}^{d}\times V\rightarrow\mathbb{R}^{d} defines a dynamical system based on the observations of tokens in a sequence. As a result, the language model embeds a sequence of length TT as a T+1T+1 step trajectory in a dd-dimensional space, where dd is the dimension of the hidden state of the recurrent LM. Next, we parametrize the sentence space into a flat-vector space 𝒵∈ℝd′\mathcal{Z}\in\mathbb{R}^{d^{\prime}} to better understand the sentence space of the LM.

To map the trajectory of hidden states to a flat vector in 𝒵\mathcal{Z}, we add a bias term zs​t​e​e​rz_{steer} ∈𝒵\in\mathcal{Z} to the previous hidden and cell state at each time step in the model and optimize zs​t​e​e​rz_{steer} to maximize the log-probability of a given sequence. Since we’re adding zs​t​e​e​rz_{steer} to every hidden and cell state as well as at every timestep, information contained in zs​t​e​e​rz_{steer} will not degrade as quickly as if we just intervened at one location at one timestep. Using this formulation, we can go back and forth, from vectors to sequences and vice-versa, and thus design experiments to test whether a frozen model can be steered to generate any sequence of interest.

To map from sequences to steering vectors (forward estimation), we modify the recurrent transition function, see Figure 2 for details:

ht=fθ​(ht−1+zs​t​e​e​r,xt)\displaystyle h_{t}=f_{\theta}(h_{t-1}+z_{steer},x_{t}) (1)

Here, we assume the dimension of zs​t​e​e​rz_{steer} is equal to the model dimension d∗d^{*}. If this is not the case, we can up or down project zs​t​e​e​rz_{steer} without the addition of any parameters. See Subramani et al. (2019) for details. We then optimize zs​t​e​e​rz_{steer} to maximize the log-probability of the sequence as mentioned before via any off-the-shelf gradient-based optimization algorithm (e.g., gradient descent, nonlinear conjugate descent, etc.).

To map steering vectors back to sequences (backward estimation), we intervene on a language model by injecting zs​t​e​e​rz_{steer} and using beam search to decode starting with a <bos> token. We stop when an <eos> token or 100 total tokens is reached. We measure how well the original sequence matches the generated sequence via three string overlap metrics: token-level exact match, BLEU score (Papineni et al., 2002), and longest prefix match.

Refer to caption
Figure 3: Recovery on IWSLT16 for medium (left) and large (center) and on random data for large (right).

Experimental Setup

First, we train our own language models on 50M sentences from the English Gigaword corpus (Graff and Cieri, 2003), with a 879k sentence development set and a 878k sentence test set stratified by article publishing date. We use byte-pair encoding with 20,000 merges for a vocabulary of 20,234 subword tokens (Gage, 1994; Sennrich et al., 2016). Our model is a 2-layer language model with LSTM units of three sizes, small (d=256d=256), medium (d=512d=512), and large (OPENd=1024)d=1024) with shared input and output embeddings (Press and Wolf, 2017), and dropout (Srivastava et al., 2014).

To train the model, we use stochastic gradient descent with Adam with a learning rate of 1e-4 and a batch size of 100 (Kingma and Ba, 2015). To learn steering vectors, we sample 100 randomly selected sentences from the development set as well as 50 sentences from the IWSLT16 En-De translation dataset to measure out-of-distribution generalization (Cettolo et al., 2016). We use nonlinear conjugate gradient (Wright and Nocedal, 1999) for optimization due to the highly non-convex nature of the objective function and use beam search with a width of 5 for backward estimation (Graves, 2012). See Subramani et al. (2019) for more details.

2.3 Takeaways

We can find steering vectors for every sequence that achieve near perfect recoverability (token-level exact match ≥\geq 0.99) on large, offering an avenue for direct causal control. Additionally, we find that larger, better trained models have higher recoverability and longer sequences are harder to recover. One key question is whether our forward estimation procedure operates like a naive compressor without any structure. In other words, does the forward estimation procedure have enough capacity to just encode the entire sequence in the vector without leveraging the language model’s internal representations of language? To test this, we create a random dataset where we sample from the vocabulary with replacement at random where every token has equal probability. We learn steering vectors for these sequences as well as the out-of-domain IWSLT16 data and measure recoverability. In Figure 3, we show that sequences that are lower entropy under the language model (IWSLT data) are much easier to recover at similar sequence lengths as compared to sequences from the random data. However, even for the very high entropy sequences (ones from random), zs​t​e​e​rz_{steer} has the capacity to encode sequences up to a token length of 28 nearly perfectly.

Limitations:

Finding steering vectors was challenging. Optimization via conjugate gradient methods was very slow and could not be easily GPU-accelerated at the time, reducing adoption. Given how good recovery was for random sequences, a natural follow-up would be to understand what steering vectors encode and whether they could be useful beyond single sequence interventions. There is no guarantee that the steering vector space learned here offers any additional utility. This is precisely what we expand upon and explore in the next section.

2.4 Bigger Picture

At the time, being able to intervene on a language model with a single vector and causally force the model to generate any sequence of interest without updating a single parameter was highly surprising. This meant that language models had tremendous potential as universal decoders and steering could open up avenues to move away from task-specific finetuning and replace with inference-time steering. Our work could serve as justification to attempt natural language prompting on better trained, stronger models, which occurred in the years that followed. BERT had just recently come out (Devlin et al., 2019), and while this paper was under review, we learned that BERT rediscovered the classical NLP pipeline (Tenney et al., 2019), hinting that internal structure likely exists in transformer-based language models.

3 Control: Steering Vectors for Transformers (Subramani et al., 2022)

We expand upon control, generalizing steering vectors to transformer-based language models for both exact steering and concept-based steering. We answer the following questions:

Key Question 2 Can transformer-based language models be steered to generate a desired sequence exactly without any parameter updates?
Key Question 3 Can extracted steering vectors act as useful representations with which we perform concept-based steering at inference-time?

3.1 Prior Work

Transformer language models started becoming popular, outperforming and largely replacing recurrent models (Devlin et al., 2019; Radford et al., 2019; Raffel et al., 2020). In § 2, we showed that LSTM-based LMs could be precisely controlled for short sequences with steering vectors, opening up the potential for them to be used as universal or general-purpose decoders. Here, we explore whether higher-quality transformer-based language models could be more easily and efficiently steered, and thus make better universal decoder candidates. This work began prior to the release of GPT3 (Brown et al., 2020), hence the focus on small transformer-based models rather than LLMs.

Injection location Timestep BLEU-4
Embedding all timesteps 33.99
Layer 6 (self attn) all timesteps 100.0
Layer 6 (self attn) first timestep 99.80
Layer 7 (feed fwd) all timesteps 100.0
Layer 7 (feed fwd) first timestep 99.25
All layers (self attn + feed fwd) all timesteps 100.0
All layers (self attn + feed fwd) first timestep 91.72
LM head all timesteps 6.72
Table 1: Sentence recovery for steering vectors when injected into different layers of the transformer model and at multiple timesteps.

3.2 Our Contributions

We coin the term steering vector. A vector zs​t​e​e​rz_{steer} ∈ℝd\in\mathbb{R}^{d} is a steering vector for a sequence xx under a model MM only if MM exactly generates xx via greedy decoding when zs​t​e​e​rz_{steer} is injected into MM.11 1 Note that steering vectors need not correspond to an exact sequence. They are commonly now used to steer towards a desired concept or attribute.

Background

We define a flat-vector space 𝒵∈Rd′\mathcal{Z}\in R^{d^{\prime}} for a transformer language model, similar to the recurrent language model from § 2. To map the trajectory of hidden states for a sequence x1,…,xTx_{1},\ldots,x_{T}, we add a bias term zs​t​e​e​rz_{steer} ∈𝒵\in\mathcal{Z} to the first-timestep at a single layer in the transformer stack after the feed-forward layer, see Figure 2 for details.22 2 zs​t​e​e​rz_{steer} could be added anywhere in the transformer stack and repeated across layers or timesteps, but we found that adding it once at a single layer and first timestep was sufficient. We also optimize zs​t​e​e​rz_{steer} to maximize the log-probability of a given sequence, giving us the ability to map sequences to steering vectors (and vice-versa) and measure recoverability, exactly like in LSTM-based models. This process is more efficient in transformers as compared to LSTMs because zs​t​e​e​rz_{steer} is only added at one layer and one timestep. We can up or down project zs​t​e​e​rz_{steer} if we want to control the capacity of the steering vector.

Experimental Setup

We take the GPT2-117M model and learn steering vectors by sampling sequences from four different genres (movies, books, news, and wikipedia) and stratify them based on length for a total of 256 sequences. Recoverability is measured via BLEU score. We vary where (injection location) and when (injection timestep) to intervene with zs​t​e​e​rz_{steer}. For optimization during forward estimation, we use Adam with a learning rate of 1.0 and use greedy decoding to recover sequences. We measure the extent to which steering vectors can be used as representations and compare them with mean-pooled hidden states.

Lastly, we explore whether concept-based steering is possible. We first extract steering vectors for sequences for positive and negative sentiment respectively via the Yelp sentiment dataset (Shen et al., 2017). We propose difference-of-means (DiffMean) steering, which works as follows. First, we use vector arithmetic to take the mean of the positive sentiment steering vectors zp​o​sz_{pos} and the negative sentiment ones zn​e​gz_{neg}. This is then operationalized at inference-time. For example, to steer towards positive sentiment, you add a steering vector zs​t​e​e​r=λ⁡(zp​o​s−zn​e​g)z_{steer}=\lambda(z_{pos}-z_{neg}) at the timestep and location that those steering vectors were extracted from.33 3 λ∈ℝ\lambda\in\mathbb{R} is a constant known as the steering strength.

Positive Input the taste is excellent!
+0.5∗(zn​e​g−zp​o​s)0.5*(z_{neg}-z_{pos}) the taste is excellent!
+1.0∗(zn​e​g−zp​o​s)1.0*(z_{neg}-z_{pos}) the taste is excellent!
+1.5∗(zn​e​g−zp​o​s)1.5*(z_{neg}-z_{pos}) the taste is bitter and bitter taste is bitter taste is bitter
Negative Input the desserts were very bland.
+0.5∗(zp​o​s−zn​e​g)0.5*(z_{pos}-z_{neg}) the desserts were very bland.
+1.0∗(zp​o​s−zn​e​g)1.0*(z_{pos}-z_{neg}) the desserts were very bland.
+1.5∗(zp​o​s−zn​e​g)1.5*(z_{pos}-z_{neg}) the desserts were very tasty.
+2.0∗(zp​o​s−zn​e​g)2.0*(z_{pos}-z_{neg}) the desserts were very tasty.
Table 2: Concept steering for sentiment for a positive input sentence (top) and negative input sentence (bottom).

3.3 Takeaways

Our experiments show that fine-grained control via the exact steering of transformer-based language models is much easier and more efficient than the exact steering of LSTMs. Table 1 shows that nearly all sequences are perfectly recovered, even when adding zs​t​e​e​rz_{steer} at just the first timestep. As long as zs​t​e​e​rz_{steer} is injected in the transformer stack (after the embedding and before the final layer), recoverability remains nearly perfect. Linearly interpolating between steering vectors gives us a glimpse into what the steering vector space looks like. Decoding from these intermediate points reveals structure: the space seems relatively smooth with large clusters corresponding to each of the sequences being interpolated between and a smooth transition in both syntax and semantics when moving from one sequence to another. Cosine distances between steering vectors at middle layers reflect semantic similarity better than mean-pooled hidden states when measured on the semantic textual similarity benchmark (Cer et al., 2017), indicating that steering vectors may be better representations than the ones learned by the underlying language models.

Steering vectors provide coarse-grained control, too. Our experiments on unsupervised sentiment transfer via DiffMean steering on the Yelp sentiment dataset show that a single direction in latent space learned via these steering vectors can flip sentiment reliably. We show two examples in Table 2. For the first time, we show that concept steering at inference-time is possible.

3.4 Bigger Picture

As language model quality started improving, control became an achievable goal. Two months after starting this project, GPT3 came out showing that large pretrained language models had the ability to, at inference-time, be few-shot prompted to solve different tasks. Our work could serve as further justification for more ambitious inference-time based control such as in-context learning, alignment, and persona-based steering. Over the past 5 years, language models became more performant with higher quality representations and concept-based steering took off, operating on the activation space rather than a reparametrized vector space like our steering vector space for inference-time control (Turner et al., 2023; Li et al., 2023; Dunefsky and Cohan, 2025; Arad et al., 2025; Bigelow et al., 2026; Morgulis and Hewitt, 2026; Wurgaft et al., 2026, inter alia).

4 Trust: mice (Subramani et al., 2025a)

We tackle trust by leveraging model internals to try to answer a key question:

Key Question 4 Can we harness the latent spaces of language models to build better confidence estimators for tool-calling agents?
Refer to caption
Refer to caption
Figure 4: Here we show both the mice (left) and acute (right) systems to calibrate language model generations. The only major differences between the systems are the featurizers and classifiers used.

4.1 Prior Work

A confidence estimator is a model that estimates the probability that a different model’s output is correct. Since language models have an internal confidence for its output already (i.e., the joint probability of the generated sequence), auxiliary confidence estimators are rarely used. However, raw confidences of language models are known to be poorly calibrated (Desai and Durrett, 2020; Jiang et al., 2021; Zhong et al., 2023). To be well-calibrated, a confidence estimator must be correct approximately as often as it thinks it is (Dawid, 1982).44 4 Calibration is commonly measured using expected calibration error (ECE; Murphy and Epstein (1967); Naeini et al. (2015)) and Brier Score (Glenn, 1950).

Confidence estimation in NLP has been studied in tasks such as machine translation (Niculescu-Mizil and Caruana, 2005), semantic parsing (Stengel-Eskin and Van Durme, 2023), and long-form text generation (Band et al., 2024). Calibrating binary classifiers with a single input feature, the raw confidence, is common practice in machine learning with Platt scaling (Platt, 1999), isotonic regression (Barlow et al., 1972), beta calibration (Kull et al., 2017), and histogram regression estimators with adaptive binning (Nobel, 1996).

Our angle, harnessing the latent spaces of models to build confidence estimation mechanisms, is unique. In fact, there are very few studies that even combine mechanistic interpretability with confidence estimation or trust. Beigi et al. (2024) improves trustworthiness by using contrastive learning on activations and Liu et al. (2025) predicts correctness in question-answering tasks using probes learned on activations.

Refer to caption
Figure 5:  mice systems outperform baselines on ETCU on the STE test set, especially at high-risk levels.

4.2 Our Contributions

Motivation:

We explore whether we can leverage model internals to build a class of model internal confidence estimators (mice) to better calibrate tool-calling agents. Taking inspiration from the early-exiting and intermediate layer decoding literature (i.e., the observation that a language model can often be decoded from early layers; Geva et al., 2022; Schuster et al., 2022; Belrose et al., 2023; Elhoushi et al., 2024; Yom Din et al., 2024; Merullo et al., 2024), we theorize that a prediction that is slowly refined through the layers ought to be more trustworthy than one that suddenly appears in the final layer.

mice:

Figure 4 shows mice on the left. For a given input query qq, we pass it through the language model and at each layer ii, we use logit lens to decode from that intermediate layer (nostalgebraist, 2020), resulting in a candidate generation y(i)y^{(i)}. Then, we compare how close that intermediate generation is to the final output generation via BERTscore (BERTscore(y(i),y(f​i​n​a​l)y^{(i)},y^{(final)}); Zhang et al., 2020) using DeBERTa-xlarge-mnli (He et al., 2021), using those as features to train a correctness classifier. The probability that the classifier assigns to <correct> is the new calibrated confidence. In practice, we use both BERTscore features and the raw confidence (i.e., the joint probability that the language model assigns to the output sequence) as features to train the classifier.55 5 We experiment with two classifiers: a random forest and a logistic regressor.

Measuring Trust:

We care about evaluating how good a confidence estimator is. This is commonly measured using ECE. However, ECE suffers from two major drawbacks:

  1. 1.

    Cannot distinguish between an oracle and a base rate estimator (i.e., one that just predicts the base rate regardless of correctness)

  2. 2.

    Invariant to the risk level of the task

These drawbacks hinder decision-making utility and thus calibration is necessary but not sufficient for our purpose. We develop our own metric, expected tool-calling utility (ETCU) to solve these drawbacks, offering a better metric with which to evaluate the quality of a confidence estimator.66 6 See Subramani et al. (2025a) for details. Additionally, we measure calibration error using a recently improved ECE variant called smooth ECE (smECE; Błasiok and Nakkiran (2024)).

Experimental Details:

Our experiments use the simulated trial-and-error (STE) dataset, a synthetically generated tool-calling dataset consisting of English-language queries, which call 50 distinct APIs (Wang et al., 2024). We split the data into demonstration (used to construct few-shot examples), training, validation, and test sets consisting of 4250, 1500, 750, and 750 examples each. We 8-shot prompt three LLMs, Llama-3-8B-Instruct, Llama3.1-8B-Instruct, and Llama3.2-3B-Instruct and decode using greedy decoding to generate candidates (Grattafiori et al., 2024). Our mice classifiers are trained using the training set, hyperparameters are tuned using the validation set, and performance is evaluated on the test set.77 7 A candidate generation is deemed to be correct if and only if it exactly matches the ground-truth answer.

4.3 Takeaways

Our experiments confirm that language models are poorly calibrated. Traditional post-hoc recalibration techniques such as histogram regression (HRE) and Nadaraya-Watson kernel regression (NWKR) tend to be highly conservative (Nadaraya, 1964; Watson, 1964), collapsing to base rate estimators with low calibration error, but low decision-making utility. mice estimators, on the other hand, perform better, maintaining low calibration error, but higher decision-making utility due to a larger spread of probability estimates across examples. In Figure 5, we observe that for tasks with medium or high-risk, both HRE and NWKR remain conservative, always abstaining for all inputs.  mice performs better, correctly increasing its abstention rate as risk-levels increase, while trusting tool-calls some of the time. To quantify how well a confidence estimator does across risk-level settings, we develop an area-under-the-curve (AUC) metric called auc-etcu, following Marcum (1960). 88 8 AUC style metrics are used in many areas of science (Wagner and Ayres, 1977; Geifman et al., 2019; Subramani et al., 2025b, inter alia).

To test out-of-domain generalization for tool-calling, we create a scenario in which new APIs are tested on. We hold out each of the 50 distinct APIs sequentially, resembling 50-fold cross validation and combine predictions across the entire test set. Despite being zero-shot, mice performs comparably to post-hoc calibration baselines across both smECE and ETCU, while having a larger spread of probability estimates across samples.

Limitations:

mice relies on an auxiliary model to calculate BERTscore features, which can be very expensive. Additionally, we experiment with a single model family, Llama, on a single task, tool-calling. Lastly, ETCU makes one strong simplifying assumption, that a confidence estimator should get a reward of 0 for abstaining regardless of whether the candidate generation was correct or not. We address all of these limitations in § 5.

5 Trust: acute (Subramani et al., 2026)

We improve trustworthiness by addressing some limitations of mice by asking:

Key Question 5 Can we harness the latent spaces of language models to efficiently build better confidence estimators for model generations across a variety of tasks including multiple-choice question answering, tool-calling, and scientific document summarization?
MMLU APIGen SCITLDR
(↓\downarrow) auc-euro (↑\uparrow) (↓\downarrow) auc-euro (↑\uparrow) (↓\downarrow) auc-euro (↑\uparrow)
estimator smECE low med high all smECE low med high all smECE low med high all
Raw Conf 0.17 0.90 0.71 0.54 0.72 0.22 0.28 0.53 0.80 0.53 0.15 0.36 0.71 0.92 0.66
HRE 0.11 0.87 0.72 0.71 0.77 0.02 0.82 0.70 0.82 0.78 0.08 0.67 0.71 0.92 0.77
NWKR 0.07 0.90 0.73 0.74 0.79 0.02 0.82 0.69 0.82 0.78 0.08 0.67 0.71 0.92 0.77
acute early act 0.07 0.91 0.73 0.74 0.79 0.05 0.90 0.82 0.87 0.86 0.08 0.69 0.72 0.92 0.78
acute mid act 0.07 0.91 0.76 0.78 0.82 0.06 0.90 0.84 0.88 0.87 0.08 0.70 0.72 0.92 0.78
acute late act 0.07 0.91 0.77 0.80 0.83 0.07 0.90 0.84 0.88 0.87 0.08 0.69 0.72 0.92 0.77
acute cosine 0.09 0.90 0.75 0.78 0.81 0.03 0.87 0.77 0.84 0.83 0.08 0.68 0.71 0.92 0.77
acute pca10 0.08 0.91 0.76 0.78 0.82 0.04 0.90 0.82 0.87 0.87 0.09 0.70 0.72 0.92 0.78
acute pca20 0.08 0.91 0.76 0.77 0.81 0.06 0.91 0.84 0.88 0.88 0.09 0.70 0.72 0.92 0.78
Table 3: Results on the MMLU test set averaged across all 57 subtasks (left), on the APIGen test subset (middle), and on the SCITLDR dev set (right). All results for all tasks are averaged across the 6 LLMs we test. Lower smECE is better, while higher auc-euro is better. Bold, underline indicate the best and second best result respectively.

5.1 Our Contributions

acute:

We introduce an activation-based confidence, utility, and trust estimation protocol (acute) to appropriately assess the confidence of a language model output. In Figure 4, we show how acute works. First, we take the activations and pass them through a featurizer. Those features are fed into a classifier to predict whether the output generation is correct or not, exactly like mice. Finally, the probability that the classifier assigns to correct is the new recalibrated confidence. acute removes the reliance on the auxiliary BERTscore model for input featurization.

euro:

We improve upon the ETCU metric from § 4 by removing the assumption that abstaining should always get 0 reward by introducing a new general and easily interpretable metric called expected utility renormalized by the oracle (euro) that balances decision-making utility with calibration. euro does not make simplifying assumptions, uses a single degree-of-freedom (the normalized net utility of correctly abstaining uc​au_{ca}), and is bounded between an oracle estimator (euro=1) and anti-oracle estimator (euro=0). Calculating euro at different uc​au_{ca} or risk values traces a curve. Measuring the area under that curve provides a score with which to compare confidence estimators called auc-euro.99 9 See Subramani et al. (2026) for further details, including a detailed derivation in the Appendix of that paper.

Experimental Setup

We apply acute to six new LLMs on three new tasks: multiple-choice question answering (MMLU; Hendrycks et al., 2021), tool-calling (APIGen; Liu et al., 2024), and scientific document summarization (SCITLDR; Cachola et al., 2020).1010 10 Results are averaged across LLMs. Performance is measured via smECE and auc-euro.

Refer to caption
Figure 6: Reliability diagrams for the gemma-3-12b-it model for APIGen (top row) and SCITLDR (bottom row) across 3 baseline estimators and two acute  confidence estimators. Darker shading corresponds to higher density of examples in that confidence bin.

5.2 Takeaways

Table 3 and Figure 6 reveals that raw confidences remain poorly calibrated and post-hoc recalibration baselines collapse to base rate estimators as in § 4. acute performs well across all three tasks, outperforming baselines on auc-euro, while maintaining low smECE. Using activations from later layers (acute late act) and PCA-reduction of all layer activations to 20 components (acute pca20) performed best across all tasks.

5.3 Bigger Picture

Without any post-hoc calibration, language models provide terrible confidence estimates. Most post-hoc calibration methods collapse the often large spread of probability estimates into a much narrower range, reducing calibration error without increasing decision-making utility. Despite these drawbacks, probability estimates with or without post-hoc calibration remain the de-facto standard. Our work challenges this by proposing decision-making utility centric confidence estimators that can better adjudicate trust. Further study on this front can help us develop even better confidence estimators and increase the reliability and safety of LLMs. Overall, our work is a small step towards harnessing the latent spaces to appropriately assign trust to LLM outputs.

6 Conclusion

We present four research contributions that demonstrate how we can operationalize the latent spaces of language models for better control and trust. Our work introduces steering vectors for exact and concept-based control on both LSTM- and transformer-based models. Steering vectors provide nearly perfect fine-grained control at inference-time, suggesting that language models could be universal decoders. We propose two methods which harness the latent spaces of models to learn confidence estimators for language model generations to improve trust. Our methods recalibrate model outputs across architectures and tasks effectively, suggesting that the latent spaces contain information to build more reliable and trustworthy language technology, especially in high-stakes scenarios. We hope that our work encourages others to open up the black box and study the latent spaces of language models.

Acknowledgments

We thank the numerous collaborators and authors on each of the individual papers discussed here and both Mona Diab and Nivedita Suresh for feedback on an early version of this work.

References

  • Arad et al. (2025) D. Arad, A. Mueller, and Y. Belinkov SAEs are good for steering – if you select the right features. In EMNLP, External Links: Link Cited by: §3.4.
  • Band et al. (2024) N. Band, X. Li, T. Ma, and T. Hashimoto Linguistic calibration of long-form generations. In ICML, Cited by: §4.1.
  • Barlow et al. (1972) R. E. Barlow, D. J. Bartholomew, J. M. Bremner, and H. D. Brunk Statistical inference under order restrictions: the theory and application of isotonic regression. John Wiley & Sons. Cited by: §4.1.
  • Beigi et al. (2024) M. Beigi, Y. Shen, R. Yang, Z. Lin, Q. Wang, A. Mohan, J. He, M. Jin, C. Lu, and L. Huang InternalInspector I2I^{2}: robust confidence estimation in LLMs through internal states. In EMNLP Findings, External Links: Link Cited by: §4.1.
  • Belrose et al. (2023) N. Belrose, Z. Furman, L. Smith, D. Halawi, I. V. Ostrovsky, L. McKinney, S. Biderman, and J. Steinhardt Eliciting latent predictions from transformers with the tuned lens. arXiv. External Links: Link Cited by: §4.2.
  • Bigelow et al. (2026) E. Bigelow, D. Wurgaft, Y. Wang, N. Goodman, T. Ullman, H. Tanaka, and E. S. Lubana Belief dynamics reveal the dual nature of in-context learning and activation steering. External Links: 2511.00617, Link Cited by: §3.4.
  • Brown et al. (2020) T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. NeurIPS. Cited by: §3.1.
  • Błasiok and Nakkiran (2024) J. Błasiok and P. Nakkiran Smooth ECE: principled reliability diagrams via kernel smoothing. In ICLR, External Links: Link Cited by: §4.2.
  • Cachola et al. (2020) I. Cachola, K. Lo, A. Cohan, and D. Weld TLDR: extreme summarization of scientific documents. In EMNLP Findings, External Links: Link Cited by: §5.1.
  • Cer et al. (2017) D. Cer, M. Diab, E. Agirre, I. Lopez-Gazpio, and L. Specia SemEval-2017 task 1: semantic textual similarity multilingual and crosslingual focused evaluation. In SemEval, External Links: Link Cited by: §3.3.
  • Cettolo et al. (2016) M. Cettolo, J. Niehues, S. Stüker, L. Bentivogli, R. Cattoni, and M. Federico The IWSLT 2016 evaluation campaign. In Proceedings of the 13th International Conference on Spoken Language Translation, External Links: Link Cited by: §2.2.
  • Dai and Le (2015) A. M. Dai and Q. V. Le Semi-supervised sequence learning. NeurIPS. Cited by: §2.1.
  • Dawid (1982) A. P. Dawid The well-calibrated bayesian. Journal of the American statistical Association (379). Cited by: §4.1.
  • Desai and Durrett (2020) S. Desai and G. Durrett Calibration of pre-trained transformers. In EMNLP, External Links: Link Cited by: §4.1.
  • Devlin et al. (2019) J. Devlin, M. Chang, K. Lee, and K. Toutanova BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), External Links: Link Cited by: §2.1, §2.4, §3.1.
  • Dunefsky and Cohan (2025) J. Dunefsky and A. Cohan One-shot optimized steering vectors mediate safety-relevant behaviors in llms. In COLM, External Links: Link Cited by: §3.4.
  • Elhoushi et al. (2024) M. Elhoushi, A. Shrivastava, D. Liskovich, B. Hosmer, B. Wasti, L. Lai, A. Mahmoud, B. Acun, S. Agarwal, A. Roman, A. Aly, B. Chen, and C. Wu LayerSkip: enabling early exit inference and self-speculative decoding. In ACL, External Links: Link Cited by: §4.2.
  • Gage (1994) P. Gage A new algorithm for data compression. The C Users Journal archive. External Links: Link Cited by: §2.2.
  • Geifman et al. (2019) Y. Geifman, G. Uziel, and R. El-Yaniv Bias-reduced uncertainty estimation for deep neural classifiers. In ICLR, Cited by: footnote 8.
  • Geva et al. (2022) M. Geva, A. Caciularu, K. Wang, and Y. Goldberg Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In EMNLP, External Links: Link Cited by: §4.2.
  • Glenn (1950) W. B. Glenn Verification of forecasts expressed in terms of probability. Monthly weather review (1). Cited by: footnote 4.
  • Graff and Cieri (2003) D. Graff and C. Cieri English gigaword corpus. Linguistic Data Consortium. Cited by: §2.2.
  • Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. C. Ferrer, C. Nikolaidis, D. Allonsius, et al. The llama 3 herd of models. External Links: 2407.21783, Link Cited by: §4.2.
  • Graves (2012) A. Graves Sequence transduction with recurrent neural networks. arXiv. Cited by: §2.2.
  • Groeneveld et al. (2024) D. Groeneveld, I. Beltagy, E. Walsh, A. Bhagia, R. Kinney, O. Tafjord, A. Jha, H. Ivison, I. Magnusson, Y. Wang, S. Arora, D. Atkinson, R. Authur, K. Chandu, A. Cohan, J. Dumas, Y. Elazar, Y. Gu, J. Hessel, T. Khot, W. Merrill, J. Morrison, N. Muennighoff, A. Naik, C. Nam, M. Peters, V. Pyatkin, A. Ravichander, D. Schwenk, S. Shah, W. Smith, E. Strubell, N. Subramani, M. Wortsman, P. Dasigi, N. Lambert, K. Richardson, L. Zettlemoyer, J. Dodge, K. Lo, et al. OLMo: accelerating the science of language models. In ACL, External Links: Link Cited by: §1.
  • He et al. (2021) P. He, X. Liu, J. Gao, and W. Chen DeBERTa: decoding-enhanced bert with disentangled attention. In ICLR, Cited by: §4.2.
  • Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. In ICLR, External Links: Link Cited by: §1, §5.1.
  • Hochreiter and Schmidhuber (1997) S. Hochreiter and J. Schmidhuber Long short-term memory. Neural computation (8). Cited by: §2.1.
  • Jiang et al. (2021) Z. Jiang, J. Araki, H. Ding, and G. Neubig How can we know when language models know? on the calibration of language models for question answering. TACL. External Links: Link Cited by: §4.1.
  • Kingma and Ba (2015) D. P. Kingma and J. Ba Adam: a method for stochastic optimization. In ICLR, Cited by: §2.2.
  • Kull et al. (2017) M. Kull, T. S. Filho, and P. Flach Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research. External Links: Link Cited by: §4.1.
  • Li et al. (2023) K. Li, O. Patel, F. Viégas, H. Pfister, and M. Wattenberg Inference-time intervention: eliciting truthful answers from a language model. NeurIPS. Cited by: §3.4.
  • Liang et al. (2023) P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. R’e, D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. Wang, K. Santhanam, L. J. Orr, L. Zheng, M. Yuksekgonul, M. Suzgun, N. S. Kim, N. Guha, N. S. Chatterji, O. Khattab, P. Henderson, Q. Huang, R. Chi, S. M. Xie, S. Santurkar, et al. Holistic evaluation of language models. Annals of the New York Academy of Sciences. Cited by: §1.
  • Liu et al. (2025) J. Liu, J. Jain, M. Diab, and N. Subramani LLM microscope: what model internals reveal about answer correctness and context utilization. arXiv. Cited by: §4.1.
  • Liu et al. (2024) Z. Liu, T. Hoang, J. Zhang, M. Zhu, T. Lan, J. Tan, W. Yao, Z. Liu, Y. Feng, R. RN, et al. Apigen: automated pipeline for generating verifiable and diverse function-calling datasets. NeurIPS. Cited by: §5.1.
  • Marcum (1960) J.I. Marcum A statistical theory of target detection by pulsed radar. IRE Transactions on Information Theory. Cited by: §4.3.
  • Merullo et al. (2024) J. Merullo, C. Eickhoff, and E. Pavlick Language models implement simple Word2Vec-style vector arithmetic. In NAACL, External Links: Link Cited by: §4.2.
  • Morgulis and Hewitt (2026) G. Morgulis and J. Hewitt Subliminal steering: stronger encoding of hidden signals. arXiv. External Links: Link Cited by: §3.4.
  • Murphy and Epstein (1967) A. H. Murphy and E. S. Epstein Verification of probabilistic predictions: a brief review. Journal of Applied Meteorology and Climatology (5). Cited by: footnote 4.
  • Nadaraya (1964) E. A. Nadaraya On estimating regression. Theory of Probability & Its Applications (1). Cited by: §4.3.
  • Naeini et al. (2015) M. P. Naeini, G. Cooper, and M. Hauskrecht Obtaining well calibrated probabilities using bayesian binning. In AAAI, Cited by: footnote 4.
  • Niculescu-Mizil and Caruana (2005) A. Niculescu-Mizil and R. Caruana Predicting good probabilities with supervised learning. In ICML, Cited by: §4.1.
  • Nobel (1996) A. Nobel Histogram regression estimation using data-dependent partitions. The Annals of Statistics (3). Cited by: §4.1.
  • nostalgebraist (2020) nostalgebraist Interpreting GPT: the logit lens. Note: Blogpost External Links: Link Cited by: §4.2.
  • Papineni et al. (2002) K. Papineni, S. Roukos, T. Ward, and W. Zhu Bleu: a method for automatic evaluation of machine translation. In ACL, External Links: Link Cited by: §2.2.
  • Peters et al. (2018) M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), External Links: Link Cited by: §1, §2.1.
  • Platt (1999) J. Platt Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers (3). Cited by: §4.1.
  • Press and Wolf (2017) O. Press and L. Wolf Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, External Links: Link Cited by: §2.2.
  • Qu et al. (2024) C. Qu, S. Dai, X. Wei, H. Cai, S. Wang, D. Yin, J. Xu, and J. Wen Tool learning with large language models: a survey. Frontiers of Computer Science. External Links: Link Cited by: §1.
  • Radford et al. (2019) A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever Language models are unsupervised multitask learners. Cited by: §1, §3.1.
  • Raffel et al. (2020) C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR. External Links: Link Cited by: §3.1.
  • Schuster et al. (2022) T. Schuster, A. Fisch, J. Gupta, M. Dehghani, D. Bahri, V. Tran, Y. Tay, and D. Metzler Confident adaptive language modeling. NeurIPS. Cited by: §4.2.
  • Sennrich et al. (2016) R. Sennrich, B. Haddow, and A. Birch Neural machine translation of rare words with subword units. In ACL, External Links: Link Cited by: §2.2.
  • Shen et al. (2017) T. Shen, T. Lei, R. Barzilay, and T. Jaakkola Style transfer from non-parallel text by cross-alignment. In NIPS, Cited by: §3.2.
  • Srivastava et al. (2023) A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, A. Kluska, A. Lewkowycz, A. Agarwal, A. Power, A. Ray, A. Warstadt, A. W. Kocurek, A. Safaya, A. Tazarv, A. Xiang, A. Parrish, A. Nie, A. Hussain, A. Askell, A. Dsouza, A. Slone, A. Rahane, A. S. Iyer, A. J. Andreassen, A. Madotto, A. Santilli, A. Stuhlmüller, A. M. Dai, A. La, A. K. Lampinen, A. Zou, A. Jiang, A. Chen, A. Vuong, A. Gupta, et al. Beyond the imitation game: quantifying and extrapolating the capabilities of language models. TMLR. Note: Featured Certification External Links: Link Cited by: §1.
  • Srivastava et al. (2014) N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov Dropout: a simple way to prevent neural networks from overfitting. JMLR (56). External Links: Link Cited by: §2.2.
  • Stengel-Eskin and Van Durme (2023) E. Stengel-Eskin and B. Van Durme Calibrated interpretation: confidence estimation in semantic parsing. TACL. External Links: Link Cited by: §4.1.
  • Subramani et al. (2019) N. Subramani, S. Bowman, and K. Cho Can unconditional language models recover arbitrary sentences?. NeurIPS. Cited by: §2, §2.2, §2.2.
  • Subramani et al. (2025a) N. Subramani, J. Eisner, J. Svegliato, B. Van Durme, Y. Su, and S. Thomson MICE for CATs: model-internal confidence estimation for calibrating agents with tools. In NAACL, External Links: Link Cited by: §4, footnote 6.
  • Subramani et al. (2025b) N. Subramani, A. Gomez, and M. T. Diab SimBA: simplifying benchmark analysis using performance matrices alone. In EMNLP Findings, External Links: Link Cited by: footnote 8.
  • Subramani et al. (2026) N. Subramani, P. Goyal, Y. Song, M. Malek, Y. Xue, T. Pfister, and H. Palangi The acute protocol: operationalizing language model activations for better calibration, utility, and trust. In ICML, Cited by: §5, footnote 9.
  • Subramani et al. (2022) N. Subramani, N. Suresh, and M. Peters Extracting latent steering vectors from pretrained language models. In ACL Findings, External Links: Link Cited by: §3.
  • Team et al. (2025) G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, et al. Gemma 3 technical report. arXiv. Cited by: §1.
  • Tenney et al. (2019) I. Tenney, D. Das, and E. Pavlick BERT rediscovers the classical NLP pipeline. In ACL, External Links: Link Cited by: §2.4.
  • Thirunavukarasu et al. (2023) A. J. Thirunavukarasu, D. S. J. Ting, K. Elangovan, L. Gutierrez, T. F. Tan, and D. S. W. Ting Large language models in medicine. Nature Medicine. External Links: Link Cited by: §1.
  • Turner et al. (2023) A. M. Turner, L. Thiergart, G. Leech, D. Udell, J. J. Vazquez, U. Mini, and M. MacDiarmid Steering language models with activation engineering. arXiv. Cited by: §3.4.
  • Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin Attention is all you need. In NeurIPS, External Links: Link Cited by: §2.1.
  • Wagner and Ayres (1977) J. G. Wagner and J. W. Ayres Bioavailability assessment: methods to estimate total area (auc 0-∞\infty) and total amount excreted (ae∞\infty) and importance of blood and urine sampling scheme with application to digoxin. Journal of Pharmacokinetics and Biopharmaceutics. External Links: Link Cited by: footnote 8.
  • Wang et al. (2024) B. Wang, H. Fang, J. Eisner, B. Van Durme, and Y. Su LLMs in the imaginarium: tool learning through simulated trial and error. In ACL, External Links: Link Cited by: §4.2.
  • Watson (1964) G. S. Watson Smooth regression analysis. Sankhyā: The Indian Journal of Statistics, Series A. Cited by: §4.3.
  • Wright and Nocedal (1999) S. Wright and J. Nocedal Numerical optimization. Springer Science (67-68). Cited by: §2.2.
  • Wurgaft et al. (2026) D. Wurgaft, C. Rager, M. Kowal, V. Shyam, S. Feucht, U. Bhalla, T. Haklay, E. Bigelow, R. Sarfati, T. McGrath, O. Lewis, J. Merullo, N. Goodman, T. Fel, A. Geiger, and E. S. Lubana Manifold steering reveals the shared geometry of neural network representation and behavior. External Links: 2605.05115, Link Cited by: §3.4.
  • Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv. Cited by: §1.
  • Yom Din et al. (2024) A. Yom Din, T. Karidi, L. Choshen, and M. Geva Jump to conclusions: short-cutting transformers with linear transformations. In COLING, External Links: Link Cited by: §4.2.
  • Zhang et al. (2020) T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi Bertscore: evaluating text generation with bert. In ICLR, Cited by: §4.2.
  • Zhong et al. (2023) R. Zhong, C. Snell, D. Klein, and J. Eisner Non-programmers can label programs indirectly via active examples: a case study with text-to-SQL. In EMNLP, External Links: Link Cited by: §4.1.