Standardizing foundation model training and evaluation in robot-assisted surgery
Standardizing foundation model training and evaluation in robot-assisted surgery.
Few foundation models exist for robot-assisted surgery, partly because large robotic-surgery video corpora are difficult to assemble and existing models are evaluated mostly on laparoscopic benchmarks. We present OmniRAS, a family of 1B- and 2B-parameter V-JEPA-2.1 encoders for robot-assisted surgery, and detail their training.
First, we release two densely annotated robotic-cholecystectomy datasets: OmniRAS-PR and a multi-label YT-Chole tool–verb–target task, the first triplet-style annotation for robotic cholecystectomy, together with splits, probe protocols, and an inter-rater study validating the shared phase ontology. Second, we document continued pretraining at up to 256 compute nodes with global batch 6,144 over 19 sources totaling approximately 2,650 hours of surgical video, 51% robotic, and analyze compute and data composition. Third, we evaluate against raw V-JEPA-2.1 and specialized surgical models on six tasks spanning triplet, phase, and step recognition, action segmentation, and detection, under frozen-encoder and final-four-block fine-tuning regimes.
Across three seeds, this yields 254 downstream runs, including 109 with partial backbone fine-tuning. The best OmniRAS models achieve the strongest adapted results across all task families, while frozen differences are smaller.
Public surgical corpora cover a narrow band of procedures, and for cholecystectomy in particular offer no multi-label interaction annotation on robotic video. These two datasets span complementary temporal scales: phase labels capture procedural state under a shared coarse ontology, while triplets capture short interactions at a finer granularity.
51 procedures under a unified 11-class procedural-phase ontology, combining 41 private procedures with ten publicly available SurgeNet procedures. The public portion holds out two complete procedures for evaluation, preventing clip-level leakage.
A re-annotation of ten SurgeNet batches as a multi-label triplet prediction problem with three independent axes. Each axis carries an independent sigmoid head, since several labels can be active in one clip. To our knowledge, the first triplet-style action annotation defined on robotic cholecystectomy video.
A 10% sample of YT-Chole was independently re-annotated for phase by two raters against the existing reference. Mean pairwise Cohen's κ over all eleven classes is 0.664 at zero boundary tolerance — substantial on the Landis–Koch scale, and reached independently by every one of the three pairs — rising to 0.741 at ±2 s and 0.807 at ±4 s. The residual disagreement is structural rather than interpretive: 82.1% of it involves the five-phase Calot's-triangle cluster, whose sub-activities are performed concurrently rather than in sequence. Pooling that cluster raises the mean to 0.833, and the coarser seven-class view is released alongside the full ontology.
The triplet ontology was assessed separately along its three axes. At ±1 s tolerance, Krippendorff's α reaches 0.917 for instrument, 0.666 for verb, 0.693 for target, and 0.722 for the complete triplet.
Every encoder is evaluated under the same probe protocol: matched task definitions, matched training protocols, and a task head whose design is fixed across backbones, so that a difference in score reflects the representation rather than the readout. Each panel below reports the paper's representative configuration for that benchmark against the strongest published or baseline system on the same task.
Bars without a whisker are single published numbers with no seed spread reported by their authors: the SAR-RARP50 reference (84.10, not shown above), TAPIS, and the best SARAS-ESAD challenge submission.
The panels above compress each benchmark to one representative metric. Below is every result figure in full — each encoder configuration, each metric, and the three-seed standard deviation — replotted from the paper's own values so a metric can be selected and the field narrowed to the comparison at hand. Gold marks the best bar in a panel, solid blue a fine-tuned configuration, grey a frozen probe or external baseline. Each figure links to the original PDF from arXiv v2 (September 9, 2026).
One difference from the printed plates: every axis here starts at zero, so bar length stays proportional to the value. Several paper panels crop the axis to spread a tight field out, which reads well on the page but makes a bar twice as long as another that is a few points behind it. Every mean and standard deviation is the paper's own.
Adaptation matters more than substitution. OmniRAS benefits are most consistent once part of the backbone is allowed to adapt. Frozen comparisons are mixed, but partial fine-tuning improves over the corresponding raw initialization on both YT-Chole triplet scales, reverses the frozen ordering on GraSP, and produces the strongest OmniRAS-PR phase and SARAS-ESAD models. The two GraSP optima also occur under different adaptation regimes: full fine-tuning is best for phase recognition (85.34 ± 0.33 mAP), whereas last-four-block fine-tuning is best for step recognition (57.85 ± 1.29 mAP). The strongest transfer does not come from applying one downstream recipe everywhere.
Continued pretraining resumes the released V-JEPA-2.1 objective on the surgical catalog at 384² resolution and 16 frames. Production runs use 256 compute nodes with a global batch of 6,144 and a fixed 50 iterations per epoch, so epoch counts are directly comparable across arms of the same run.
| Production run | Epochs | Node-hours | YT-Chole IVT mAP | GraSP phase macro-F₁ |
|---|---|---|---|---|
| 9.22 M samples | 30 | 438 | 33.3 | 73.5 |
| 18.43 M samples | 60 | 872 | 32.4 | 75.0 |
| 36.86 M samples | 200 | 1,728 | 35.7 | 84.0 |
The longest production run yields the strongest transfer, but its catalog also changed, so the gain cannot be attributed to compute alone. The one matched-budget comparison — 9.22 M to 18.43 M samples under the same catalog and recipe — was approximately probe-neutral. We report the campaign as practical guidance rather than a scaling law.
| Run | Epoch | Node-hours | Training loss | YT-Chole IVT mAP | GraSP phase macro-F₁ |
|---|
Select probes with different temporal and semantic requirements in advance and evaluate them at fixed checkpoints. YT-Chole IVT captures short-horizon, spatially grounded activity; GraSP phase recognition probes longer-horizon procedural context. Their joint trajectory reveals both checkpoint sensitivity and whether continued self-supervised optimization is actually producing increasingly transferable representations.
Clear transfer gains emerged only beyond inexpensive screening budgets. A null result at small scale should not be read as strong evidence that an intervention is ineffective. Once the recipe and evaluation harness are validated, extend a strong configuration to production scale before distributing the same compute across many shallow variants.
Checkpoint indexing, epoch definitions, data manifests, and probe locations must be fixed as part of the training specification — ambiguities are difficult to reconstruct after long distributed runs. The evaluation suite should also include at least one diagnostic outside the target surgical benchmarks.
Surgical continued pretraining is specialization, not free improvement. With Kinetics-400 rehearsal during pretraining, OmniRAS reaches 53.82 top-1 and 44.70 macro-F₁ on Something-Something-v2, against 60.27 and 52.58 for the raw Meta 2B initialization — 6.45 and 7.88 points of general-action capability traded for surgical transfer. That trade-off is invisible from the surgical benchmark suite alone, which is why the campaign carries a general-video probe.
Every downstream number depends to some extent on the task-specific readout. We therefore replicate the V-JEPA-2.1 masking objective as an evaluation — no task head anywhere in the loop — and ask whether the surgical checkpoint predicts masked targets in its own latent space more accurately than the checkpoint it was initialized from.
Across five corpora — OmniRAS-PR, GraSP under both the phase and step sampling protocols, SAR-RARP50, and SARAS-ESAD — the surgically pretrained checkpoint achieves higher matched cosine similarity, higher cosine lift, and lower L₁ prediction error in every evaluation setting. At the independent source-unit level the direction favours it in all 60 of 60 paired comparisons. The frozen probes therefore do not indicate that surgical CPT left the representation unchanged; the label-free result identifies the readout as the cause of the frozen-probe ties.
To support reproducibility and further research, we will release all OmniRAS checkpoints together with the two datasets introduced in this work. The two datasets were constructed under two separate IRB-approved protocols, and their release will follow the corresponding institutional, privacy, and data-use requirements.
@article{borgioli2026omniras,
title = {OmniRAS: Standardizing Foundation Model Training and
Evaluation in Robot-Assisted Surgery},
author = {Borgioli, Leonardo and Getty, Neil and Xiu, Wenli and
Cassiani, Jessica and Ducas, Alvaro and Orda, Carlos Agustin and
Waris, Hira and Xia, Fangfang and Stevens, Rick and
Giulianotti, Pier Cristoforo and {\v{Z}}efran, Milo{\v{s}}},
journal = {arXiv preprint arXiv:2608.31048},
eprint = {2608.31048},
archivePrefix = {arXiv},
primaryClass = {eess.IV},
note = {Manuscript under review},
year = {2026}
}