runai_streamer: S3 loading 6x slower (800 MiB/s) than local disk (4.9 GiB/s)

I'm testing model loading from S3 using runai_streamer and encountering performance issues. Any insights would be greatly appreciated!

Environment

• vLLM: 0.11.0 with runai_streamer
• S3 Backend: SeaweedFS standalone server running on the same node as vLLM
• Storage: Local NVMe SSD (6800 MB/s sequential read capability) via Kubernetes CSI volume
• Model: Llama-3-8B-Instruct (~16 GB)
• Deployment: All tests use localhost (same-pod) or pod IP (same-node) - no cross-node network traffic

Deployment Topology

┌─────────────────────────────────────────────────────────────┐
│  Physical Node:  GPU Node (NVMe 6800 MB/s)                  │
│                                                             │
│  Test Setup 1: Same Pod (localhost)                         │
│  ┌────────────────────────────────────────────────────────┐ │
│  │  Pod: all-in-one                                       │ │
│  │  ├─ Container: model-store (SeaweedFS)                 │ │
│  │  │  └─ S3 API: 0.0.0.0:9000                            │ │
│  │  │                                                     │ │
│  │  └─ Container: model-service (vLLM)                    │ │
│  │     └─ Load model from: http://localhost:9000          │ │
│  │        Result: ~800 MiB/s @ concurrency=1              │ │
│  └────────────────────────────────────────────────────────┘ │
│                                                             │
│  Test Setup 2: Same Node, Different Pods                    │
│  ┌─────────────────┐          ┌──────��───────────────────┐  │
│  │ Pod: agent      │          │ Pod: vllm-service        │  │
│  │ model-store     │◄─────────│ model-service (vLLM)     │  │
│  │ (SeaweedFS)     │ pod IP   │ Load from: pod-ip:9000   │  │
│  │ :9000           │          │ Result: ~787 MiB/s       │  │
│  └─────────────────┘          └──────────────────────────┘  │
│         │                                                   │
│         └─────────► Same NVMe disk (CSI local volume)       │
└─────────────────────────────────────────────────────────────┘

Configuration

# RunAI Streamer settings (all tests)
RUNAI_STREAMER_CHUNK_BYTESIZE=67108864  # 64 MB
RUNAI_STREAMER_MEMORY_LIMIT=21474836480  # 20 GiB
RUNAI_STREAMER_S3_USE_VIRTUAL_ADDRESSING=0
S3_ENDPOINT_URL=http://localhost:9000

# Variable parameter
RUNAI_STREAMER_CONCURRENCY=1  # Tested: 1, 2, 4, 8, 16

# SeaweedFS configuration
weed server -s3 -dir=/data/seaweedfs \
  -volume.readMode=local \
  -volume.index=memory \
  -volume.fileSizeLimitMB=8192 \
  -master.defaultReplication=000

Observations

1. S3 loading throughput far below disk capability

• Measured throughput: 787 MiB/s
• Disk capability: 6800 MB/s sequential read
• Model load time: 19.5 seconds
• Utilization: Only 11.5% of disk bandwidth

2. Higher concurrency reduces throughput

Each doubling of concurrency reduces throughput by ~25-35%.

3. Network topology has minimal impact

• Same-pod (localhost): ~800 MiB/s
• Same-node different pods (pod IP): ~787 MiB/s
• Performance is nearly identical across deployment topologies

4. Container resources appear unconstrained

• CPU utilization: Well below limits during loading
• Memory utilization: Well below limits during loading
• No visible resource saturation in metrics

5. Significant performance difference between load methods

• Loading from local filesystem (same disk): ~4.9 GiB/s
• Loading from S3 localhost: 787 MiB/s
• Performance ratio: 6.2x difference

Questions

I'm trying to understand the performance characteristics and optimize for production use. Could you help with:

1. Is the observed throughput (787 MiB/s from local S3) expected? What throughput should I expect when loading from a local S3 endpoint backed by fast NVMe storage?

2. What causes the inverse concurrency scaling? I expected higher concurrency to improve throughput, but I'm seeing the opposite. Is this:

   • Expected behavior with certain S3 backends?
   • A configuration issue on my end?
   • Something worth investigating further?

3. What diagnostic/profiling steps do you recommend?

What I've tested

• ✅ Different chunk sizes (16 MB, 64 MB, 128 MB) - 16 to 64 helps a bit, 64 to 128 minimal impact
• ✅ Different memory limits (10 GiB, 20 GiB, -1 unlimited) - minimal impact
• ✅ Localhost vs pod IP - identical performance
• ✅ Container resource limits - no CPU/memory saturation
• ✅ Concurrency values 1-16 - inverse scaling observed

Any guidance on:

• Expected performance characteristics
• Diagnostic approaches
• Configuration best practices
• Known limitations or trade-offs

...would be greatly appreciated!