Toward Workflow-Aware Benchmarking for Healthcare NLP Agents

Healthcare is one of the clearest domains in which the distinction between a fluent language model and a useful language agent becomes operationally important. Clinical and care workflows are multi-step, stateful, and high stakes. A system may need to summarize a chart, retrieve guideline evidence, maintain dialogue context, defer when information is missing, and support a human handoff when uncertainty increases. In such settings, strong one-turn performance is not sufficient evidence of practical usefulness.

This gap matters because the healthcare AI literature is moving quickly toward agentic framing. Recent work discusses planning, tool use, retrieval, memory, and multi-step interaction as central features of LLM systems in medicine and care [1, 2, 3, 4]. At the same time, several surveys and perspective papers warn that generic benchmark gains can be mistaken for clinical readiness [5, 6, 7, 8, 9]. The main failure is not always incorrect text in isolation. Often it is a failure to preserve state, retrieve the right evidence, surface uncertainty, or stay within workflow scope.

This short paper takes a narrow position: a useful healthcare-agent benchmark needs an operational layer between static prompt-response tests and prospective clinical workflow studies. This concern is consistent with broader agent-evaluation arguments that apparent system performance can depend heavily on hidden execution assumptions rather than architecture alone [10]. Static benchmarks remain useful, but they should be treated as one layer in a broader evidence stack.

Our contribution is threefold. First, we distinguish claims supported by evidence at the model, agent, and simulated-workflow levels. Second, we specify a reproducible episode protocol, including reference decisions, annotation units, and aggregation. Third, we provide four task templates and a cost-sensitive escalation score. The contribution is a design and measurement protocol, not a released dataset or clinical validation study.

Healthcare work is sequential rather than atomic. Documentation, triage, patient follow-up, discharge communication, and evidence-assisted decision support all unfold over time. The user goal may change mid-task. New information may arrive after an initial output. A system may need to ask a clarifying question rather than answer immediately. These features make workflow quality at least as important as local answer quality.

This point is especially visible for agents. A standalone model can appear strong if it gives a fluent answer to a static question. A healthcare agent, by contrast, may fail even when its language is polished. It can omit a contraindication, carry forward stale chart information, or retrieve irrelevant evidence while sounding persuasive. In other words, healthcare failures are often state or process failures rather than pure generation failures. This is why recent work on agent memory, retrieval-grounded generation, and dynamic medical evaluation is directly relevant to healthcare AI assessment [11, 12, 13, 14, 15].

The practical implication is straightforward: evaluation should track whether an agent preserves relevant context, uses evidence appropriately, recovers from interruption, and escalates when scope boundaries are reached. Recent work on realistic static and live evaluation of agent routing further motivates treating routing as an explicit behavior rather than an incidental text output [16]. A benchmark that ignores these behaviors may be useful for capability screening while still being weak as evidence for deployment.

Static medical QA benchmarks remain useful for testing factual recall, explanation, multilingual coverage, and specialty breadth [17, 18, 19]. Hallucination benchmarks similarly help characterize unsupported claims and safety-sensitive failure modes [20, 21]. However, these tasks freeze interaction into a narrow slice. They rarely test whether the model should have deferred, queried for missing context, or tracked prior state across turns.

Clinical phenotype concept recognition is another useful capability target [22], but strong performance on a prompting-based extraction task likewise does not establish that an agent can manage state changes, evidence constraints, or escalation within a workflow.

This limitation becomes more serious once systems are framed as assistants embedded in care workflows. Recent benchmark directions begin to address the problem. MedAgentBench evaluates agentic tasks in EHR-like environments rather than only text prompts [23]. PhysicianBench and standardized-patient-style evaluations move further toward long-horizon clinical behavior [24, 25, 26]. EHR-grounded and consultation-centered benchmarks also improve realism by anchoring evaluation in records, physician chats, and evolving patient trajectories [27, 28, 29, 30]. Medical summarization and RAG-focused evaluations add another important axis by testing provenance retention, evidence selection, and grounded generation [31, 15, 32].

The field is therefore already moving in the right direction. The remaining challenge is conceptual integration. Different benchmarks capture different layers of useful behavior, but they are often reported side by side without a common evaluation logic. This can obscure what kind of claim a strong score actually supports and makes reusable benchmark design harder.

We organize healthcare NLP agent assessment according to three evidence levels. The distinction is not a new claim that evaluation is layered; its added value is a reporting rule: every result must identify its evidence level and must not be promoted to a stronger deployment claim.

Table I summarizes the framework in a compact form. The table is intentionally practical: each level is tied to a deployment question, a typical metric family, and the main blind spot if that level is used alone.

Documentation support, evidence retrieval, patient messaging, and triage do not fail in the same way. A benchmark package that is adequate for one task may be poorly matched to another. Table II restores this task-sensitive view: a universal healthcare-agent leaderboard can obscure the changing balance between correctness, grounding, escalation, and continuity.

For example, note transformation can be locally accurate yet still be weak for triage, where the dominant risk shifts from factuality to escalation and continuity. Likewise, evidence retrieval does not establish safety for patient-facing communication without testing boundaries and harmful-advice mitigation.

The evaluation unit is a short, artifact-grounded workflow episode rather than a single prompt. Each episode has five required fields: (i) an initial context packet, (ii) a state-changing event or interruption, (iii) an allowed action space, (iv) a reference decision profile, and (v) an adjudication rubric. The context uses de-identified or synthetic artifacts; the state change identifies facts that must be retained, revised, or discarded. The action space makes clear whether the agent may answer, retrieve, revise, ask a question, abstain, or escalate. This fixed schema makes episode construction auditable and prevents post-hoc scoring criteria.

Two qualified annotators independently produce reference decisions using an episode guide. They label required facts, prohibited claims, acceptable evidence sources, the appropriate action set, and handoff fields. A third reviewer adjudicates disagreements, and a benchmark release should retain both the final label and disagreement type. This supports reproducibility without claiming that a single reference response is uniquely correct.

For every episode, scores are calculated from atomic annotations rather than global impressions. State continuity is scored as the fraction of required facts retained or correctly revised, minus contradictions and stale facts. Evidence traceability is scored as the proportion of substantive claims linked to an admissible source, with penalties for unsupported or mismatched citations. Handoff completeness is scored as the fraction of required routing fields that are present and correct. All score components and weights are reported by task class.

Escalation is evaluated as a decision, not as a generic safety preference. Let $E$ denote the reference escalation label and $\hat{E}$ the agent action. We report sensitivity, specificity, and a configurable utility, $U=u_{TP}TP+u_{TN}TN-c_{FN}FN-c_{FP}FP$, where $c_{FN}>c_{FP}$ when missed escalation is judged more harmful than unnecessary escalation. The cost matrix must be declared before evaluation and reviewed by the task’s clinical governance group. This makes the trade-off visible instead of silently rewarding either excessive deferral or unsafe completion.

Consider a patient message reporting worsening shortness of breath after a recent medication change. A static benchmark may reward a correct explanation or summary, but it can miss the clinically salient failure: producing a polished reply instead of initiating escalation. The episode protocol instead asks whether warning signals are recognized, prior context is retained, and the handoff contains the required information. In this setting, a brief acknowledgment plus escalation can be preferable to a detailed answer.

The protocol operationalizes four requirements and sets a clear boundary on its claims.

Healthcare NLP agents should be evaluated as workflow-bound systems rather than as isolated response generators. Static medical benchmarks remain useful, but they do not justify claims about deployable assistance by themselves. We provide an episode-level protocol that separates evidence levels and makes state continuity, traceability, escalation, and handoff behavior reproducibly scorable.

The immediate next step is a small, clinically governed pilot that releases episode guides, adjudication decisions, score distributions, and failure analyses. Until then, this paper should be read as a testable measurement protocol rather than evidence that any agent improves healthcare work.