Towards a Reliable and Practical Eval Pipeline

Abstract

LLM-based software systems increasingly require effective “evals” as quality gates in the development lifecycle. However, existing work typically addresses individual aspects of eval reliability rather than the full set of practical requirements. We present an end-to-end eval pipeline that combines eval checklist creation, with learned aggregation for checklist responses, to improve agreement across LLM judges and accuracy against human judgments. The framework additionally provides self-consistency, explanations, and prediction uncertainty, and we empirically demonstrate its effectiveness.

1 Introduction

Over the past few years Large Language Models (LLM) have become a common ingredient in software, where they assist users in a variety of tasks such as drafting emails, writing and summarizing documents. This has engendered novel challenges in the software development process, where such product features are required to be released only after passing through rigorous quality tests. Here, the traditional practice of writing purely programmatic tests no longer suffices, and this step needs to utilize LLMs as well. For example, if a particular LLM-driven feature produces summaries, a corresponding test that checks for its coherence or fluency also typically needs to use an LLM.

This creates an interesting problem: the fundamental behaviors we want our tests to check, e.g., effects of LLM non-determinism, are also possessed by the tests themselves. How do we then ensure that our tests precisely measure what we intend them to, and also provide guarantees similar to a traditional quality assurance framework?

Effective LLM-as-judge evaluations (“evals” henceforth) are a topic of active research, e.g., G-Eval (Liu et al., 2023), CheckEval (Lee et al., 2025), Prometheus-2 (Kim et al., 2024), FActScore (Min et al., 2023), dynamic rubrics based on information gain (Xu et al., 2026), hybrid judges for image caption evaluation (Matsuda et al., 2025), GraphJudge for evaluating knowledge graphs (Huang et al., 2025). However, most research focuses on specific aspects of evals; usually certain forms of alignment with human judges. We believe that this state of the current literature leaves open many questions wrt practical deployment .

These gaps exist both in form of (a) not examining different aspects of alignment, and (b) not covering the full gamut of practical use-cases. In a sense, recent research leans towards addressing challenges that may be deemed necessary, but are not sufficient for a practical setup. Our focus is the latter.

Our primary contributions are: (a) first, we present the desiderata of a practical eval framework, outlining risks and benefits, and (b) then, we propose such a framework, and empirically show that it effectively satisfies those needs. Rather than a single novel idea, this work offers an assembly of strategies to create an end-to-end eval pipeline that can augment a standard quality assurance pipeline.

The paper is organized as follows: we begin by listing the desiderata for a practical eval framework in §2, and then we present an overview of our framework in §3. Sections 4 - 7 present our empirical analyses. Our final section, §8, discusses limitations of the current study, and our conclusions.

2 A Practical Eval Pipeline

We believe that a practical eval system must at least ensure the following properties:

• Inter-LLM Agreement (or agreement in short): An eval output must exhibit minimal variance across LLMs. The lack of this property is a business risk. LLM A might clear all tests today, but when evals are switched to use LLM B (maybe due to an organization’s policy) that’s a harsher critic, we might end up with high failure rates.

• Accuracy: Eval outcomes must closely match human judgments. This is in contrast to most empirical analyses where some form of correlation, e.g., Kendall’s $\tau$ or Spearman correlation coefficient $\rho$, is measured, e.g., Lee et al. (2025); Liu et al. (2023). Product quality gates are almost always hard thresholds, e.g., “the coherence score for this summary must be $\geq 4$”. For such scenarios, there is a budgeting risk in productizing research with reported success in correlation-type metrics.

• Self-consistency: An eval output should exhibit minimal variation across multiple executions against the same LLM. This directly impacts how many times a test needs to be run for it to certify that a feature works. Thus, this impacts development cost and time-to-ship.

• Explainability: In traditional software testing, it is easy to trace failures to pinpoint a cause, by looking at a stack trace or logs. This is difficult when evals are complex and are written as large text prompts. What’s the equivalent of a stack trace in this new world?

• Confidence scores: Since LLM-based evals are probabilistic, we need to provide some representation of confidence for eval outcomes. We need to move away from hard predictions, e.g., “the eval score is 5” to soft estimates, e.g., “the eval score has a 90% prediction interval of 3-5”.

P1-P3 characterize forms of alignment, while P4 and P5 represent additional critical use-cases.

Figure 1: Our eval pipeline. A checklist of YES/NO eval questions are sent to an LLM, along with inputs for the eval. In this example, the eval assesses the quality of summarization, so a document and its corresponding summary constitute our additional inputs. The outputs from the LLM are sent to a tabular model, e.g., Gradient Boosted Decision Trees, to produce the final judgment score. An explanation, and some form of confidence score, are also produced. See §3 for details.

3 Methodology

Our eval pipeline is shown in Figure 1. All evals are represented as a checklist of $d$ atomic YES/NO questions. The checklist, along with eval inputs, are sent to an LLM, which produces a binary vector of responses $x\in\{0,1\}^{d}$. A model $f:\{0,1\}^{d}\mapsto\mathbb{R}$ learned earlier is used to “aggregate” these responses to predict a judgment $y=f(x)$. We produce two additional outputs: (1) explanations using SHAP, which help a user trace back a judgment to specific questions, and (2) a representation of confidence, either in the form of a calibrated probability (Platt, 1999; Guo et al., 2017; Niculescu-Mizil and Caruana, 2005) or a high-confidence interval via conformal prediction (Shafer and Vovk, 2008).

The checklist and the model are created in a prior eval-authoring step. This requires human inputs in the form of: (a) an eval prompt - we’ll refer to this as the seed prompt, and (b) sample inputs and outputs for their eval. The step produces, as output, (1) an eval checklist, and (2) a model $f$.

The seed prompt is decomposed into multiple simpler YES/NO questions. The idea is that (a) having targeted atomic questions leaves little room for ambiguity, and (b) having multiple of them acts as an “error buffer”: even if the LLM misinterprets one question, it is unlikely that it will misinterpret every related question, which leads to a stable fraction of YES responses. This strategy leads to both improved agreement and self-consistency (empirically shown in §5).

We are motivated to follow the above strategy due to its success in a variety of eval settings (Min et al., 2023; Lee et al., 2025; Liu et al., 2024; Li et al., 2025; Xu et al., 2026). We specifically use CheckEval (Lee et al., 2025) because of its good results wrt inter-LLM agreement.

Since our goal also is accuracy wrt human judgment, we use the eval input-output examples to construct our model $f$. The technique of using a model in the final step has successful precedents (Hashemi et al., 2024; Liu et al., 2024). We use Gradient Boosted Decision Trees (GBDT) (Friedman, 2001; Ke et al., 2017) in this work, which gives us positive results (see §5).

In addition to being a powerful model family, GBDTs offer other benefits: (a) they may be easily learned for a specific output type, e.g., $y$ may be boolean, ordinal, categorical or continuous; (b) we use SHAP (Lundberg and Lee, 2017) to generate prediction explanations, which permits exact explanations for tree-based models such as GBDTs.

4 Experiment Setup

In our experiments, we rigorously measure agreement, self-consistency and accuracy, and present persuasive results for confidence scoring and explanations (these rely on standard techniques).

Dataset and LLMs: The SummEval (Fabbri et al., 2021) dataset (MIT License) is used for experiments. It consists of documents, their corresponding summaries (more than one summary per document), and human-provided scores per summary that assess their quality across different axes. These axes are coherence, consistency, fluency and relevance, and we have one eval corresponding to each axis. Each summary, per quality axis, is rated by multiple human raters, with the provided scores being in $\{1,2,3,4,5\}$. The summaries are our measurement units, and for summary $s_{i}$ and an eval axis, we average the human ratings to obtain the ground truth (GT) label $y_{i}\in[1,5]$.

We generate LLM scores for two additional configuration parameters :

1. LLMs: We use $L=4$ different LLMs to measure agreement. The LLMs used are Opus 4.8 (abbreviated to opus), Sonnet 4.6 (sonnet), GPT 5.6 Sol (sol), Grok 4.6 (grok). For sonnet the temperature was set to $0$; the others don’t support this parameter.

2. Number of trials: Each eval execution is repeated $T$ times to measure statistical significance. In our experiments $T=5$.

These additional configurations will be denoted by the indices $l$ and $t$ respectively; the predicted scores are denoted as $\hat{y}_{ilt}$ for summary $s_{i}$ for a particular eval axis.

Assuming a checklist size of $d$ questions for an eval, a response vector generated for various combinations is denoted by $x_{ilt}\in\{0,1\}^{d}$ . We train one GBDT model per eval, on tuples of the form $(x_{ilt},y_{i})$ to minimize $RMSE(y_{i},\hat{y}_{ilt})$ on a held-out set. Thus, for an eval, each GT label is paired with $L\times T$ inputs. Essentially, the model is trained to map from various noisy realizations of $x$ to a GT label. We use a train-conformal-test split of $50:10:40$ (the conformal split is used in §7), and to prevent data leakage, we ensure that data related to a specific $s_{i}$ is in exactly one split. For additional details, please see §A.

Metrics: As mentioned above, accuracy is measured using the RMSE score. Lower is better.

For measuring agreement and self-consistency (note, these scores don’t use GT, only $\hat{y}_{ilt}$ ), we do not use the popular Krippendorff’s-$\alpha$ score because: (a) we have multiple trials per rater/LLM, which $\alpha$ doesn’t naturally accommodate, and (b) $\alpha$ might be negative (Artstein and Poesio, 2008), and we prefer a range of $[0,1]$ for easy interpretation.

Instead we map our scores from a range of $[1,5]$ to $[0,1]$, and define the disagreement between two scores $a$ and $b$ as $D=|a-b|$. For an eval, this is averaged over trials for a measurement unit (here, a summary) to produce $\bar{D}_{i}$. For agreement, the averaging for two given LLMs is over their $T^{2}$ possible trial combinations, and for self-consistency the $T(T-1)/2$ unique combinations for an LLM are used. Finally we report $1-\bar{D}_{i}$ as our score (agreement or self-consistency, depending on the averaging) for this unit. Further averaging is called out in our analyses in §5.

Of course, since these scores are not chance-adjusted, they should be interpreted jointly with accuracy.

Hardware: The LLMs were accessed through an Amazon Bedrock API. All experiments were run on a M3 Max Apple Macbook Pro laptop.

5 Results

This section presents results from our setup as compared to a baseline. The latter uses seed prompts from prior literature - see §A.1.

Agreement: The average agreement for a pair of LLMs is shown as $4\times 4$ heatmap in Figure 2. A particular cell further averages $1-\bar{D}_{i}$ over all summaries and evals in a common held-out set. Figure 2(a) shows baseline results, and both (b) and (c) come from our setup.

In Figure 2(b), the aggregator model $f$ has seen data from all LLMs during training, and a cell indexed by $[A,B]$ compares the agreement between LLMs $A$ and $B$ on a held-out set. We notice that (b) produces much higher scores than (a), and the overall average (in the plot title) improves: $0.84\rightarrow 0.96$.

Perhaps more interesting is Figure 2(c). Here an LLM is held-out as well, and we cycle through all the LLMs, denoted by rows. For example, the first row indicates that the model $f$ only saw data from opus, sol and sonnet in the training data, but the agreement is computed between each of these and grok, on a held-out set. Even here, we note high scores for all pairs.

(a) Baseline agreements.

(b) Agreements - all LLMs in train/test.

(c) Agreements - with one held-out LLM.

Figure 2: LLM agreements on held-out test data (higher is better). (a) shows baseline agreements, (b) and (c) are from our setup. In (b) data from all LLMs were present both in train and test splits. In (c), for a row, data from the LLM indexing the row was absent in train, but agreements are reported with the other LLMs on a held-out set. See text for details.

Self-consistency: The self-consistency scores on the baseline prompts, and then on the output from our framework, are shown in Table 1. The score for an LLM are averaged over summaries and evals. Here, the scores look good to start with, although Sol and Grok seem to benefit most from our approach, and the overall average slightly improves.

	grok	opus	sol	sonnet	Average
Baseline	0.94	0.97	0.96	0.99	0.97
Our setup	0.97	0.99	0.98	0.99	0.98

Table 1: Self-consistency averaged across all evals, per LLM. Higher is better.

Accuracy: Accuracy scores, per eval, are shown as grouped bar plots in Figure 3. Each group shows a bar for the baseline RMSE (labeled Baseline), aggregation via just reporting the fraction of YES answers (labeled CheckEval), and the output from the entire pipeline (labeled CheckEval+ML).

We show the CheckEval bars to emphasize that reducing variability - which CheckEval is effective at when the fraction of YESes are reported as the final score (as reported in the original work Lee et al. (2025)) is insufficient to obtain good accuracy. In fact, for the coherence eval, CheckEval is worse than Baseline. In all cases, CheckEval+ML produces the lowest RMSE score.

Figure 3: Eval accuracy measured using RMSE (lower is better).

6 Explanations

It is often not enough to know why a test produced a certain result; the ability to trace an outcome back to causes provides an actionable path to improving a product feature. For a typical LLM eval driven by one prompt, since both the prompt and input might be long, pinpointing an exact fragment in the input, and the corresponding violated rubric in the eval prompt, is much harder. The simple strategy of asking the LLM to explain its judgment has been shown to be unreliable: LLMs tend to favor plausibility over faithfulness (Turpin et al., 2023; Madsen et al., 2024; Chen et al., 2025).

Our framework is able to sidestep this issue:

• Because our eval is distilled into a checklist, there are specific contributing questions to point back to.

• Using a tabular model as the aggregator allows us to use a popular technique like SHAP to identify questions most influential to a specific outcome. Additionally for GBDTs, TreeSHAP (Lundberg et al., 2019) provides exact SHAP attribution values. Figure 4 shows a waterfall plot of question influences, produced by SHAP.

Figure 4: A standard SHAP waterfall chart for a particular input - for the consistency eval, on opus - showing influences of various questions in a checklist, in decreasing order from top-to-bottom.

7 Prediction Confidence

We discuss how we quantify uncertainty, which arises due to various factors such as variability across LLMs, non-determinism, and eval inputs that are poorly represented in the training data.

While this is convenient with a model like Gaussian Process (Rasmussen and Williams, 2005), as mentioned earlier, we prefer a tree-based model due to high-fidelity SHAP explanations. We use conformal predictions (CP) here to report prediction intervals that are likely to contain the true outcome $90\%$ of the time. Being model-agnostic, CP also makes this step invariant to future model changes. We use the residual-normalized version (Lei et al., 2018; Cordier et al., 2023) to make the intervals input-adaptive.

To validate, we generate synthetic data from valid $x_{ilt}\in\{0,1\}^{d}$ vectors by flipping $b\in\{0.2,0.4,0.6\}$ fraction of bits, and visualize the distribution of the interval widths; shown in Figure 5, for the (a) consistency, and (b) coherence evals. The “inliers” KDE plot is on the test data, and is expected to have relatively small widths; the legend shows its empirical coverage. In both cases, we observe that higher values $b$ lead to wider intervals (expected behavior). See §A.4 for more plots.

(a) consistency eval

(b) coherence eval

Figure 5: Distribution of interval widths on actual (inliers) vs synthetic noisy data.

8 Limitations and Conclusion

In this work we presented a reliable eval pipeline that meets multiple practical requirements, listed in §2. We think this work is important in bringing academic research into the industry; even if the exact pipeline isn’t used, the general architecture and specific components may be built upon. An obvious limitation of this work is that it is validated on a single dataset for the task of summarization, against proprietary LLMs; something that the authors are working to address.

9 Acknowledgements

This work was possible with the support of the Q3 Salescloud team in Salesforce, specifically Ronnie Fong, Nitin Jain and William Hackett. We are grateful to Harsha Medikonda, Prasad Maderamitla, Hinita Patel, Aaron Quan and Soumya Mittapalli, for pointing us to multiple real-world challenges. We also would like to thank Pratik Gupte, Kimberley Lee and Daniel May for various technical suggestions.

Appendix A Additional Experiment Details

A.1 Baseline Prompts

We use the G-Eval [Liu et al., 2023] prompts without the autogenerated chain-of-thoughts evaluation steps as the baseline prompts in our experiment.

A.2 Generated Checklists

Dimension	# Sub-dimensions	# Seed Questions	# Final Questions
Coherence	3	3	20
Consistency	3	3	16
Fluency	4	4	24
Relevance	5	5	21

Table 2: Checklist size per SummEval dimension. The sub-dimensions and seed questions were taken from CheckEval [Lee et al., 2025]. We used Opus 4.8 to generate the final questions.

A.3 GBDT Hyperparameters

The GBDT in §4 was learned using cross-validation of hyperparameters from the following search space:

    "n_estimators": [20, 50, 100, 200]
    "learning_rate": [0.001, 0.01, 0.1, 1.0]
    "max_depth": [3, 5, 10]
    "min_child_samples": [5, 10, 20, 40]
    "reg_lambda": [0.0, 1.0, 5.0]
  

A.4 Additional Conformal Prediction Results

Continuing from §7, we present the plots for all evals in Figure 6.

(a) consistency eval

(b) coherence eval

(c) fluency eval

(d) relevance eval

Figure 6: Distribution of interval widths for points in the data or “inliers” and synthetic noisy data, which is constructed by flipping bit for different fractions of bits $b\in\{0.2,0.4,0.6\}$. The empirical coverage of the inliers is shown in the legend, which may be observed to be quite close to intended coverage of $90\%$.