Measurement-Driven Sub-Network Selection for
On-Premise Retrieval-Augmented Factory Agents

LLM inference cost is reduced by structural pruning with recovery training [6, 7], knowledge distillation [8, 18], low-rank adaptation [11], and quantization [22]. Weight-shared super-networks train many sub-networks jointly under the sandwich rule [5, 17] and mature into train-once, specialize-per-device deployment [16]; hardware-aware benchmarks [19] and on-device measurements [20] bring these ideas to LLM scale. Train-once pipelines, however, select by pre-adaptation accuracy predictors [16]; this paper shows selection must follow application adaptation (Sec. III-D).

Retrieval-augmented generation grounds an LLM’s answer in retrieved document passages rather than parametric memory alone [9]. Retrieval-aware finetuning (RAFT) [10] trains with distractor passages mixed into the context; Sec. IV adopts this inside a distillation objective.

In [3], intent-based agentic automation for manufacturing is proposed and [4] extends agentic orchestration to network infrastructure and services, while in [2] an assistant framework is presented, built around a task-orchestrator LLM delegating to external agents, explicitly including computer-vision models. These frameworks generally assume large, typically cloud-hosted models, leaving a gap in how such capabilities can be realized through compact models inside resource-constrained industrial environments. This paper addresses that gap.

Supernetworks are built with an identical recipe at two scales, Llama-3.2-3B-Instruct and Llama-3.2-1B-Instruct [13], matching the deployment tiers of Sec. VIII; the 3B instance is described here. The transformer layers are made elastic along two dimensions, network depth and per-layer MLP intermediate size, which leave the shared residual dimensionality and attention geometry intact (the costliest to recover after pruning [6, 7]) and trade quality against size smoothly. Calibration collapses this space into an 11-point candidate grid spanning 2.20-3.21 billion unique (as-deployed) parameters on the 3B base and 0.90-1.24 billion on the 1B; the largest candidate is the unpruned base model itself. A candidate is activated in place by slicing the shared weight tensors, so no separate student model is instantiated.

The supernetwork is finetuned on general instruction data (Alpaca-GPT4 [12]) before any sub-network is committed to. Following slimmable networks [5] and single-stage supernets [17], each optimization step trains the full supernetwork together with $M{=}3$ sampled sub-networks, using the full network’s detached output distribution as an in-place teacher. Sampling is importance-calibrated, in that channels are pre-permuted by importance and the $M$ sub-networks are drawn uniformly from the 11-point calibrated grid of Sec. III-C rather than the raw space. The per-step objective is

where $z_{\text{full}}$ and $z_{i}$ are the logits of the full network and the $i$-th sub-network on the same batch, $y$ are the ground-truth tokens, $p^{\tau}$ is the softmax softened at temperature $\tau$, $\mathrm{KL}$ is the forward Kullback-Leibler (KL) divergence from the full network’s detached distribution, and $\alpha,\beta$ balance the terms ($\alpha{=}0.8$, $\beta{=}0.2$, $\tau{=}1.0$ in production). Gradients from all forwards accumulate into the shared weights, transferring large-capacity behavior to smaller candidates without a separate teacher pass. The ablation in Sec. VII isolates what calibrated sampling contributes.

After supernetwork training, sub-networks are materialized as standalone checkpoints. Candidates come from an 11-point grid over the elastic dimensions, built by binning the parameter range and keeping the best WikiText-perplexity configuration per bin plus the smallest and the full one. Extraction itself is adopted from the sub-network selection literature [1].

Committing to one grid rank per target is itself a selection problem, and cheap proxies are insufficient for it. The natural quality proxy (KL divergence to the full-width supernetwork on production-shaped prompts) is perfectly rank-correlated with parameter count (Spearman $\rho{=}{-}1.0$); no monotone proxy can express the inversions of measured fronts. Selection rests on measurement, in three stages: (i) proxy metrics for orientation, (ii) real evaluation, in which every rank is adapted (Sec. IV), exported, and measured for routing accuracy, judged RAG quality, and on-device throughput, latency, memory, and energy (Sec. VIII), and (iii) selection, in which the deployed rank is

where $Q_{r}$ is judged RAG quality, $T_{r}$ measured on-device throughput, tildes denote min-max normalization over the Pareto front, $w{=}0.5$, ties resolve toward higher throughput, and $\mathcal{F}=\{r:C_{r}\geq\phi\,C_{\text{full}},\;M_{r}\leq M_{\max}\}$ imposes an ARC-Easy capability floor ($\phi{=}0.8$) and a memory ceiling. The floor $\phi$ is a deployment-policy parameter, yet it changes the outcome, disqualifying candidates retaining only $67$-$78\%$ of the unpruned ARC-Easy score, among them the unconstrained blend’s picks on both grids. A three-anchor throughput predictor ($\text{tok/s}=a/n_{\text{params}}$), fitted per device on three ranks spanning the parameter range, predicts the eight held-out ranks with $2.5$-$3.1\%$ mean absolute error, and the fronts computed from the predicted throughput coincide with the fully measured fronts on both grids. Commissioning a new device, therefore, requires benchmarking only the three anchors ($6.0$ hours instead of $28.8$ for the full grid).

Table I separates what pruning costs from what distillation recovers. Under an identical retrieval stack and a single judge pass over all $633$ held-out questions, extraction at the deployed rank 6 drops judged answer quality by $13.7\%$ relative to the base; the retrieval-grounded second stage returns the model to within $4.6\%$ of the unpruned model’s judged quality, recovering two thirds of the loss (RQ1). The gap is estimated over $633$ paired questions; its bootstrap $95\%$ confidence interval is $[-5.9\%,-3.3\%]$ of the base score. General capability and adapted task quality also decouple. ARC-Easy falls almost linearly with parameter count ($R^{2}{=}0.93$); judged RAG quality is only loosely size-ordered ($R^{2}{=}0.76$), with rank 6 matching the two largest candidates. Selection exploits this decoupling, and the capability floor guards against it.

Table II isolates the Stage-2 ingredients ($n{=}291$, one judge pass): context-free supervised finetuning degrades the extracted model ($0.654$ vs. $0.701$), grounded task loss reaches $0.727$, the softened distillation term adds three points, and RAFT distractors complete the deployed recipe at $0.765$.

Fig. 3 shows the decoupling and the measured selection plane; Table III compares selection rules on the RevPi grid. Every simpler rule forfeits something: size- and speed-driven picks retain only $69$-$74\%$ of base capability and quality-only ignores throughput, while the constrained blend keeps all three (RQ2). On the 1B grid the floor likewise removes the unconstrained pick (rank 10) and selects rank 5. The floor sweep (Table III, bottom) shows the picks move only at extreme values. With the floor in place, the blend weight barely matters, selecting rank 6 at every $w\in[0.15,0.85]$ on both 3B devices and rank 5 on the UNO Q (rank 8 at speed-leaning $w$).

The deployed cores’ scores appear in Table IV. The Jetson and RevPi cores (rank 6) reach $0.773$ faithfulness, $0.845$ answer relevancy, and $0.811$ context utilization ($n{=}633$, the session of Table I); the 1B core, judged in its own session, scores $0.591/0.749/0.682$: the native 1B trails the compressed 3B on all three metrics, so the 3B grid is used wherever memory allows (the grids were judged separately, so the gap is indicative). Tool routing reaches $40/40$ for every fine-tuned core once $37$ routing demonstrations join the training mix, costing $1.3$-$2.5$ points across the RAG metrics.

We ablate the supernetwork stage along both axes (calibrated sampling, supernetwork training), judging all arms in one interleaved session. At full width the arms are indistinguishable ($0.77$-$0.80$); mid-grid, where deployment candidates live, calibrated sampling without supernetwork training collapses to $0.44$-$0.50$ (ranks 4-5) while the trained supernetwork holds ${\sim}0.72$; supernetwork distillation, not sampling calibration, carries mid-grid quality. With training, the recipes split only at the extremes: calibrated sampling is better at the largest ranks, including the deployed rank 6 ($0.783$ vs. $0.738$, paired $t$-test, $n{=}30$ per rank, $t{=}2.3$), and worse below rank 8 ($+0.13$-$0.15$ for random sampling, $t{=}3.9$-$5.3$); ARC-Easy reproduces the crossover. Calibrated sampling buys quality where selection operates, at the cost of a small-rank tail it never deploys.

Retrieval is not evaluated: the retriever is fixed everywhere, so every RAG score is conditioned on its unmeasured recall. Judged quality comes from a single LLM judge; interleaving removes between-arm drift but not the judge itself, and the 3B and 1B grids were judged separately, so cross-grid comparisons are indicative only. The $80\%$ ARC-Easy capability floor is defensible but arbitrary; a different floor can change the picks, though on our grids they move only at extreme values. Finally, the compact cores inherit small-model failure modes. Multi-step reasoning, long tool chains, and counting-style vision queries remain unreliable, and a retrieval miss cannot be repaired downstream.

We deploy on a RevPi Connect 5 (4 GB industrial DIN-rail computer on the Raspberry Pi CM5; llama.cpp, Q4_K_M, 3B rank 6), an NVIDIA Jetson Orin Nano (8 GB unified memory; ONNX Runtime GenAI, W8A16, 3B rank 6; measured in the default 15-W mode), and an Arduino UNO Q (2 GB; llama.cpp, Q4_K_M), whose memory ceiling admits no 3B candidate even at 4-bit, so its core is the 1B grid’s rank 5, selected by the same procedure. Document index, prompts, routing grammar, and queries are identical across platforms; only model scale, quantization, and runtime differ. Fig. 4 shows the deployed assistant answering through both tool paths on the Jetson tier of the demo setup.

Each platform is evaluated along three axes. (i) RAG quality: the three RAG metrics of Sec. VI, scored by a frontier-LLM judge with all systems’ answers interleaved and shuffled per question in one session, guarding against judge drift and documented position bias [21]. Retrieval is identical everywhere, so any cross-platform difference in the three scores comes from the reasoning core. (ii) Latency: time-to-first-token (TTFT), decode tokens per second (TPS), and end-to-end (E2E) latency per query composed from measured per-call latencies. (iii) Resources: peak memory (process RSS; unified high-water on the Orin), per-inference joules, and standby watts (60-s idle, model resident). Latency, memory, and energy use a $4\times 4$ grid of prompt and generation lengths, eight repetitions per cell after warm-up, end-of-sequence ignored.

A full query (router, retrieval, synthesis at production lengths) takes about $20$ s on the Jetson versus one to over two minutes on the CPU tiers, where prefill dominates (Table IV); per-inference energy is similar across the three tiers ($117$-$170$ J) while standby varies by $4\times$ ($1.3$-$5.0$ W), so at sparse duty cycles standby dominates total energy. The delta rows compare each deployed model against the unpruned rank on the same device; all three tiers gain on latency, energy, and memory, except for an insignificant 2% memory increase on the UNO Q (RQ3).