A lightweight two-stage pipeline that combines a VLM (Moondream2 1.86B, Florence-2 base 231M / large 770M) with MobileSAM for fast referring expression segmentation on edge GPUs.
| Pipeline | Mean IoU (mIoU) | FPS | GPU Memory |
|---|---|---|---|
| Florence-2-base + MobileSAM (LoRA) | 73.64% | 22.8 | 2.2 GB |
| Florence-2-large + MobileSAM (LoRA) | 72.82% | 13.0 | 3.38 GB |
| Moondream2 + MobileSAM (LoRA) | 63.22% | 16.5 | 3.9 GB |
| GroundingDINO-L + SAM (baseline) | 59.8% | 1.8 | 14+ GB |
Numbers are from paper/vespa_paper.pdf. P@0.5 and overall IoU are also reported in the paper.
Text prompt + image
|
v
VLM (Moondream2 or Florence-2) -> bounding box
|
v
MobileSAM (LoRA) -> segmentation mask
These are the non-empty checkpoints currently present under checkpoints/ in this workspace:
| Path | Description | Size |
|---|---|---|
checkpoints/mobilesam_lora_best.pt |
MobileSAM LoRA (best) | ~39.5 MB |
checkpoints/mobilesam_lora_latest.pt |
MobileSAM LoRA (latest) | ~40.9 MB |
checkpoints/florence/florence_lora_best.pt |
Florence-2 LoRA (large) | ~1.47 GB |
checkpoints/florence_base/florence_lora_step500.pt |
Florence-2 base LoRA (step 500) | ~472 MB |
checkpoints/ is gitignored. If a .pt file is 0 bytes, the weights were not saved/exported.
VespaSeg/
|-- checkpoints/ # Fine-tuned weights (gitignored)
|-- data/ # Datasets and sample images (gitignored)
|-- eval_results/ # Saved eval logs/plots
|-- logs/
|-- models/ # Base models (gitignored)
| |-- mobilesam/ # MobileSAM repo and weights
|-- outputs/ # Generated outputs
|-- paper/ # Paper sources and PDF
|-- results/ # Metrics and logs
|-- scripts/
| |-- demos/ # Interactive demos
| |-- eval/ # Evaluation scripts
| |-- training/ # Fine-tuning scripts
|-- FINETUNED_MODELS.md
|-- README.md
|-- LICENSE
python scripts/demos/moondream2_pipeline.py --images data/images_480p --weights models/mobilesam/weights/mobile_sam.ptThis demo uses the base Moondream2 weights from HuggingFace and requires CUDA.
python scripts/eval/eval_florence2_box.py \
--sam-base models/mobilesam/weights/mobile_sam.pt \
--mobilesam-checkpoint checkpoints/mobilesam_lora_best.pt \
--florence-checkpoint checkpoints/florence_base/florence_lora_step500.ptThe evaluation script expects COCO images under ~/huggingface/datasets/mscoco and loads RefCOCO via HuggingFace datasets.
python scripts/training/finetune_mobilesam.py
python scripts/training/finetune_moondream.pypip install torch torchvision transformers datasets pillow pycocotools opencv-python tqdm peft bitsandbytes
pip install git+https://github.com/ChaoningZhang/MobileSAM.git- Training: NVIDIA RTX 6000 Ada (48 GB VRAM)
- Inference: 2.2-3.9 GB VRAM for the paper results (RefCOCO val)
@article{wickramasinghe2026vespaseg,
title={VespaSeg: Efficient VLM-Guided MobileSAM for Referring Expression Segmentation},
author={Wickramasinghe, Savindu Dilshan},
year={2026}
}