SurgSkill-Bench: A Benchmark for Multimodal Surgical Skill Assessment

Abstract

Objective assessment of surgical technical skill is important for surgical training and structured feedback, but current workflows remain dependent on labor-intensive expert review. Existing automated approaches primarily focus on visual inputs and provide limited support for jointly studying operative performance, structured skill scores, and evaluator feedback. We introduce SurgSkill-Bench, an initial video-score-text benchmark-style dataset containing 214 surgical training simulation videos, six-dimensional OSATS scores, and expert free-text comments. We define two evaluation settings: video-only OSATS prediction for automated assessment and post hoc expert-comment-assisted prediction, where evaluator comments are available as auxiliary information. We provide controlled baseline experiments using representative frozen visual backbones, content-adaptive key-frame sampling, and a simple video-text co-attention fusion module. Under internal video-level validation, content-adaptive sampling improves video-only performance in this dataset, while evaluator comments provide additional score-related signal in the assisted setting. The best mean AUROC reaches 0.88 under dataset-specific median dichotomization. We further discuss evaluation constraints related to dataset scale, metadata completeness, and the interpretation of comment-assisted prediction. Code will be released publicly at a later date.

1 Introduction

Objective assessment of surgical technical skill is central to surgical training, credentialing, and quality assurance. The Objective Structured Assessment of Technical Skill (OSATS) is a widely adopted standard in which expert raters score procedures across multiple dimensions using structured rubrics [8, 9, 2]. Despite its clinical relevance, OSATS requires manual expert review, making it labor-intensive, difficult to scale, and susceptible to inter-rater variability and subjective bias [12, 5]. Although operative videos are increasingly used for education and assessment, recent surveys highlight the lack of standardized datasets and evaluation protocols for automated surgical skill assessment [4, 6]. Importantly, expert assessment in clinical practice often includes qualitative feedback explaining assigned scores, yet this semantic information is less commonly modeled together with operative video and structured skill scores [10, 3].

Advances in computer vision have enabled analysis of surgical video, including phase recognition, tool usage, and outcome prediction [7]. Beyond surgical video analysis, generative learning has been applied to standard-plane synthesis and missing-slice imputation in cardiac MRI, joint restoration of multiple image degradations, and diffusion-based brain-tumor inpainting [16, 17, 15, 14]. However, transferring general-purpose video representations to surgical skill assessment remains challenging. Surgical training videos contain long low-information intervals, repeated movements, and subtle instrument–tissue interactions, which can dilute skill-relevant cues under uniform sampling [10, 1]. In addition, technical skill assessment is not based solely on visual appearance: expert raters often combine observed motion, tissue handling, instrument use, procedural flow, and qualitative reasoning when assigning OSATS scores. Existing automated studies have made progress in video-based assessment, but few benchmark settings explicitly connect operative video, structured multi-dimensional skill scores, and evaluator free-text comments within a unified evaluation protocol.

As an initial step toward studying these issues, we introduce SurgSkill-Bench, a pilot video-score-text benchmark for surgical skill assessment. The benchmark contains surgical training videos, six-dimensional OSATS ratings, and expert free-text comments. The primary goal of this work is to establish a reproducible benchmark setting and baseline protocol, rather than to claim a new state-of-the-art architecture. Our contributions are fourfold: (1) We introduce an initial video-score-text dataset of 214 surgical training simulation clips with six-dimensional OSATS scores and evaluator free-text comments. (2) We define two evaluation settings: video-only OSATS prediction and post hoc expert-comment-assisted OSATS prediction. (3) We provide a controlled baseline suite using representative frozen visual backbones, shared regression heads, and content-adaptive sampling. (4) We provide an explicit evaluation analysis covering thresholding, comment-assisted prediction, metadata constraints, and future participant-level validation.

2 Dataset & Benchmark

2.1 Data Collection and Composition

SurgSkill-Bench contains 214 curated surgical training simulation video clips provided by physicians. Each clip is approximately one minute long and records the operative field during task execution. The current release is designed for postoperative surgical skill assessment rather than intraoperative decision support. Each clip is linked to six-dimensional OSATS rating events and a set of available evaluator free-text comments. Available scores are aggregated by dimension to construct video-level consensus targets. Input frames are resized to $224\times 224$ during preprocessing. Because expert annotation of surgical skill is costly, the current release remains modest in scale. We use a video-level split to prevent direct clip overlap between training, validation, and test partitions. Specifically, 10% of videos are held out as an independent test set, and the remaining 90% are used for five-fold cross-validation. In each fold, all frames, scores, and comments derived from the same video are assigned to the same partition. Model selection is performed using the validation fold only, and the held-out test set is not used for hyperparameter selection. The reported mean and standard deviation are computed across the five video-level folds unless otherwise stated. Because participant- and session-level metadata are incomplete, the current results should be interpreted as internal video-level benchmark performance rather than participant-level generalization.

2.2 Annotations: OSATS and Expert Comments

Each video was assessed by two or more expert expert raters using six OSATS dimensions: Respect for Tissue, Time & Motion, Instrument Handling, Flow of Operation, Knowledge of Instruments, and Overall Score. Each dimension was scored on a 1–5 ordinal scale. The regression target for each dimension is the mean rater score, treated as a continuous consensus label. In addition to numerical OSATS scores, raters provided free-text comments describing technical performance, including tissue handling, instrument use, flow, motion efficiency, and overall technical quality. Since the same evaluators provided both comments and scores, comment-assisted experiments may contain score-related information and are interpreted as post hoc assisted prediction rather than autonomous video-only assessment.

2.3 Benchmark Tasks

Models are trained to regress continuous consensus scores on a 1–5 scale across the six OSATS dimensions. We report MAE and MSE as the primary regression metrics because the prediction target is the averaged rater score for each dimension. For supplementary discrimination analysis, we compute AUROC after median-based binarization for each dimension: scores greater than the median are treated as high-score samples, and scores at or below the median are treated as low-score samples. These thresholds are dataset-specific and should not be interpreted as clinically validated competency cutoffs.

3 Methodology

To evaluate baseline algorithms on SurgSkill-Bench, we provide a modular framework for video-only and expert-comment-assisted OSATS prediction. As illustrated in Fig. 1, the full assisted framework consists of three components: (1) a video representation module for extracting spatiotemporal features; (2) a text encoding module for representing evaluator comments; and (3) a fusion and regression head that combines visual and comment-derived features to predict six OSATS scores. For video-only experiments, the text branch and co-attention module are removed, and predictions are made from the video representation using the same type of regression heads.

Figure 1: Overview of the expert-comment-assisted baseline framework. Frozen video encoder backbones and a frozen GPT-2 text encoder are used to extract visual and comment-derived features. A bidirectional co-attention module fuses the two representations, followed by regression heads to predict six OSATS scores. In the video-only setting, the text branch and fusion module are removed.

3.1 Content-Adaptive Key-Frame Extraction

To reduce visual redundancy in surgical videos, we evaluate Content-Adaptive Key-Frame Extraction (CA-Frame) as a simple sampling baseline. A pre-trained InceptionV3 model extracts a 2048-dimensional feature vector for each frame [13]. Starting from the first frame as the reference, a new frame is selected as a key-frame when its cosine distance from the current reference exceeds a threshold $\tau=0.05$, after which the reference is updated. If too few key-frames are detected for the required input length, the data loader falls back to uniform sampling from the original clip. Otherwise, the selected key-frame pool is sampled in chronological order to form the fixed-length input sequence required by each backbone. Standard uniform sampling and CA-Frame use the same final input length, so CA-Frame changes only the temporal locations of selected frames. In the current dataset, CA-Frame selects an average of 80 frames from over 3000 original frames per clip. Since it uses ImageNet-pretrained InceptionV3 features, CA-Frame may miss subtle skill cues or fine-grained motion continuity.

Figure 2: Content-Adaptive Key-Frame Extraction (CA-Frame) pipeline. InceptionV3 frame features and cosine distance to a reference frame are used to select key-frames, with uniform sampling applied when key-frames are sparse.

3.2 Expert-Comment-Assisted Fusion

Expert comments are encoded using an off-the-shelf GPT-2 text encoder [11]. Token-level features are pooled into comment embeddings, and rater comments are aggregated with multi-head attention to obtain a text representation $f_{t}$. Missing comments are represented by a neutral placeholder. The GPT-2 text encoder is kept frozen in all benchmark experiments. Only the text projection layer, comment aggregation module, fusion block, and regression heads are trained.

The co-attention module is used as a controlled fusion baseline rather than a novel architectural contribution. Let $F_{v}\in\mathbb{R}^{T\times d}$ denote the projected video feature sequence and let $F_{t}\in\mathbb{R}^{M\times d}$ denote the projected comment feature sequence, where $T$ is the number of visual tokens and $M$ is the number of available comment embeddings for a video. The bidirectional attention is computed as:

$$A_{v\rightarrow t} =\mathrm{MHA}(F_{v}W_{v},F_{t}W_{t},F_{t}W_{t}),$$ (1)

$$A_{t\rightarrow v} =\mathrm{MHA}(F_{t}W_{t},F_{v}W_{v},F_{v}W_{v}).$$ (2)

The attended features are combined with residual connections, layer normalization, temporal pooling, and a feed-forward fusion block. In video-only experiments, the model is trained and evaluated without comments. Therefore, comment-assisted results are reported separately from video-only results.

3.3 Multi-Task Regression and Optimization

The video-only or fused representation is passed to six independent regression heads corresponding to the OSATS dimensions. The model is trained with the average MSE loss over the six consensus OSATS targets:

$$\mathcal{L}_{total}=\frac{1}{6}\sum_{k=1}^{6}||\hat{y}_{k}-y_{k}||^{2}_{2}.$$ (3)

The visual backbone is frozen in benchmark runs, while the trainable projection layers, fusion module, and regression heads are optimized for six-dimensional OSATS prediction. Unless otherwise stated, both visual and textual pretrained encoders are frozen. We use regression rather than ordinal classification because the training targets are averaged rater scores, which produce continuous consensus labels. Ordinal modeling is left for future benchmark extensions.

4 Experiments

Implementation Details. All experiments are implemented in PyTorch on an NVIDIA A100 GPU. Input frames are resized to $224\times 224$. Standard sampling uniformly samples frames from each clip, whereas CA-Frame first selects candidate key-frames and then samples them in chronological order to match the same final input length. The final input length is 16 frames for all video backbones; for DINOv3, 16 frames are encoded independently and temporally pooled. All pretrained visual and textual encoders are frozen, and only projection layers, fusion modules, and six regression heads are trained.

Models are optimized with AdamW using a learning rate of $1\times 10^{-4}$, weight decay $1\times 10^{-4}$, batch size 8, and gradient clipping with maximum norm 1.0. Training is performed for 200 epochs with model selection based on validation MAE. Mixed precision is used consistently across experiments. The projection and fusion hidden dimension is 256, and the co-attention module uses 4 attention heads. Results are reported across five video-level folds with fixed split seeds. The held-out test set is not used for hyperparameter selection.

Baselines. We evaluate representative pretrained visual backbones, including ViViT, VideoMAE, DINOv3, V-JEPA 2, X-CLIP, and a Surgical SSL baseline. To ensure a controlled comparison, all pretrained visual backbones are used as frozen feature extractors unless otherwise stated, and all models are evaluated with comparable projection layers and six OSATS regression heads. For image-level encoders such as DINOv3, frame-level features are temporally pooled to obtain video-level representations. For video backbones such as ViViT, VideoMAE, and V-JEPA 2, backbone outputs are projected into a shared hidden space before regression or multimodal fusion. Although X-CLIP is originally a video-language model, we evaluate it as a frozen video feature extractor under the same downstream OSATS regression protocol to ensure comparability with the other backbones. The Surgical SSL baseline uses a surgical-domain encoder pretrained on over 840 hours of publicly available surgical videos from SurgVU [18].The SSL encoder was pretrained with 16-frame clips resized to $224\times 224$ for 200 epochs using AdamW and a masked video modeling objective. No SurgSkill-Bench videos or labels were used during SSL pretraining. In downstream experiments, the SSL encoder is frozen and evaluated under the same projection-head and OSATS regression protocol as the other backbones.

Evaluation Metrics. Because OSATS scores are ordinal scores on a 1–5 scale, MAE and MSE are used as the primary regression metrics. They are computed for each OSATS dimension and then macro-averaged across the six dimensions. We additionally report quadratic weighted Cohen’s kappa between rounded model predictions and rounded consensus labels as a secondary ordinal agreement statistic. For kappa calculation, continuous predictions are rounded to the nearest integer and clipped to the valid OSATS range of 1–5 before comparison with expert-derived target scores. Kappa is computed per dimension and then macro-averaged. Because the reference labels are averaged rater scores, this statistic reflects agreement with rounded consensus labels rather than agreement with individual expert raters, and should not be interpreted as human inter-rater reliability. For supplementary discrimination analysis, AUROC is computed after median-based binarization as described in Section 2. Since the median thresholds are dataset-specific and not clinically validated competency cutoffs, AUROC is interpreted as a secondary benchmark statistic. We also include a mean-score predictor as a naive baseline to contextualize MAE and MSE. This baseline predicts the training-set mean score for each OSATS dimension.

Quantitative Results. Table 1 reports video-only performance under standard frame extraction and CA-Frame extraction, and Table 2 reports the corresponding post hoc expert-comment-assisted setting. CA-Frame is associated with improved video-only performance for most backbones under the current internal video-level protocol; for example, VideoMAE improves from 0.57 to 0.86 AUROC, and ViViT improves from 0.55 to 0.85 AUROC. Evaluator comments improve several models in the post hoc assisted setting, particularly under standard frame extraction, but the benefit is not uniform across all backbones and metrics. Under key-frame inputs, ViViT and V-JEPA 2 reach the best mean AUROC of 0.88 under dataset-specific median dichotomization. These results support the assisted-prediction setting but should not be conflated with autonomous video-only assessment.

Table 1: Performance of video-only models under Standard Frame Extraction and Key-Frame Extraction (Mean $\pm$ STD).

Model	Standard Frame Extraction				Key-Frame Extraction
	MAE $\downarrow$	MSE $\downarrow$	Kappa $\uparrow$	AUROC $\uparrow$	MAE $\downarrow$	MSE $\downarrow$	Kappa $\uparrow$	AUROC $\uparrow$
VideoMAE	$0.51\pm 0.03$	$0.41\pm 0.02$	$0.56\pm 0.06$	$0.57\pm 0.03$	$0.16\pm 0.01$	$0.04\pm 0.01$	$0.85\pm 0.07$	$0.86\pm 0.06$
ViViT	$0.47\pm 0.02$	$0.33\pm 0.02$	$0.61\pm 0.08$	$0.55\pm 0.04$	$0.20\pm 0.02$	$0.06\pm 0.01$	$0.81\pm 0.06$	$0.85\pm 0.07$
X-CLIP	$0.67\pm 0.11$	$0.64\pm 0.19$	$0.45\pm 0.04$	$0.58\pm 0.04$	$0.36\pm 0.07$	$0.26\pm 0.09$	$0.83\pm 0.05$	$0.83\pm 0.06$
DINOv3	$0.78\pm 0.04$	$0.83\pm 0.06$	$0.33\pm 0.06$	$0.61\pm 0.05$	$0.60\pm 0.07$	$0.52\pm 0.09$	$0.63\pm 0.06$	$0.79\pm 0.03$
V-JEPA 2	$0.48\pm 0.00$	$0.40\pm 0.00$	$0.49\pm 0.07$	$0.56\pm 0.06$	$0.15\pm 0.01$	$0.05\pm 0.01$	$0.87\pm 0.05$	$0.85\pm 0.04$
SSL	$0.72\pm 0.06$	$0.74\pm 0.09$	$0.34\pm 0.08$	$0.66\pm 0.05$	$0.34\pm 0.00$	$0.30\pm 0.00$	$0.84\pm 0.07$	$0.83\pm 0.04$

Table 2: Performance of post hoc expert-comment-assisted models under Standard Frame Extraction and Key-Frame Extraction (Mean $\pm$ STD). Expert comments are evaluator-provided feedback and may contain score-related information.

Model	Standard Frame Extraction				Key-Frame Extraction
	MAE $\downarrow$	MSE $\downarrow$	Kappa $\uparrow$	AUROC $\uparrow$	MAE $\downarrow$	MSE $\downarrow$	Kappa $\uparrow$	AUROC $\uparrow$
VideoMAE	$0.76\pm 0.01$	$0.80\pm 0.01$	$0.36\pm 0.05$	$0.68\pm 0.05$	$0.14\pm 0.02$	$0.03\pm 0.01$	$0.84\pm 0.07$	$0.87\pm 0.05$
ViViT	$0.34\pm 0.03$	$0.18\pm 0.03$	$0.83\pm 0.05$	$0.81\pm 0.04$	$0.13\pm 0.02$	$0.03\pm 0.01$	$0.85\pm 0.08$	$0.88\pm 0.04$
X-CLIP	$0.44\pm 0.03$	$0.29\pm 0.05$	$0.75\pm 0.08$	$0.78\pm 0.03$	$0.18\pm 0.01$	$0.06\pm 0.01$	$0.86\pm 0.07$	$0.86\pm 0.04$
DINOv3	$0.59\pm 0.02$	$0.49\pm 0.02$	$0.34\pm 0.06$	$0.75\pm 0.02$	$0.27\pm 0.02$	$0.12\pm 0.02$	$0.79\pm 0.09$	$0.84\pm 0.02$
V-JEPA 2	$0.56\pm 0.06$	$0.50\pm 0.07$	$0.60\pm 0.08$	$0.75\pm 0.03$	$0.13\pm 0.02$	$0.03\pm 0.01$	$0.85\pm 0.08$	$0.88\pm 0.04$
SSL	$0.73\pm 0.03$	$0.74\pm 0.04$	$0.35\pm 0.07$	$0.70\pm 0.02$	$0.16\pm 0.02$	$0.04\pm 0.01$	$0.85\pm 0.07$	$0.87\pm 0.03$

Figure 3: Qualitative attention visualization. The first row shows original surgical frames with manual annotations of instrument, hand, and tissue interaction regions. The second row shows attention maps from the CA-Frame-guided surgical SSL encoder, and the third row shows attention maps from a generic visual baseline.

Qualitative Visualization. Fig. 3 shows self-attention heatmaps from validation samples. In these selected examples, the CA-Frame-guided SSL encoder visually appears to produce more concentrated responses around regions containing instruments, hands, and tissue interactions, whereas the generic visual baseline appears more diffuse. These visualizations are qualitative examples only and should not be interpreted as quantitative evidence of interpretability or causal proof of clinically meaningful reasoning.

5 Conclusion

We introduced SurgSkill-Bench, an initial video-score-text benchmark for surgical skill assessment that combines surgical training videos, six-dimensional OSATS scores, and evaluator free-text comments. We defined separate video-only and post hoc expert-comment-assisted settings and provided controlled baselines using frozen visual backbones, content-adaptive sampling, and video-text fusion. Under internal video-level validation, CA-Frame was associated with improved video-only performance, while evaluator comments provided additional score-related signal in the assisted setting. These findings establish a benchmark baseline, while remaining constrained by dataset scale, incomplete participant metadata, dataset-specific AUROC thresholds, and the assisted nature of comment-based prediction. Future work will expand the dataset, add participant-level validation, include text-leakage controls, and analyze human agreement.

Disclosure of Interests

The authors have no competing interests to declare.