Best pattern for scalable file uploads in Spring Boot?

[simoz404]

Select Topic Area

Question

Body

I’m building a Java Spring Boot API that needs to handle file uploads, and I want to design it in a way that scales well.

What’s the recommended way to structure the upload flow if I plan to process files asynchronously and store them in something like S3 later?

Are there common patterns to avoid blocking threads on large uploads (e.g. streaming, queues, async controllers, etc.)?

I’d appreciate any architecture tips, code-level patterns, or example setups.

[TanishqCh07]

1. High-level flow

For a scalable “upload now, process later, store in S3” setup, a common pattern is:

Client → API: send file.

API:

Accepts the file (ideally in a streaming fashion).

Stores it temporarily (local disk / temp S3 bucket / object storage).

Enqueues a message with metadata (fileLocation, userId, …).

Returns 202 Accepted + uploadId immediately.

Worker / Consumer:

Listens to the queue (RabbitMQ/Kafka/SQS, etc.).

Downloads the temp file / reads from storage.

Does CPU / I/O intensive work.

Uploads the processed result to S3.

Updates status in DB.

This keeps the HTTP request short-lived and lets you scale processing independently.

2. Avoid blocking threads during upload

You have a few options, depending on how far you want to go:

a) Simple (blocking) but good enough for many cases

Use classic Spring MVC with MultipartFile, write to disk or stream to S3 using InputStream. For moderate traffic this is fine:

@RestController
@RequestMapping("/uploads")
public class UploadController {

private final UploadService uploadService;

public UploadController(UploadService uploadService) {
    this.uploadService = uploadService;
}

@PostMapping(consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
public ResponseEntity<UploadResponse> upload(@RequestParam("file") MultipartFile file) {
    String uploadId = uploadService.acceptUpload(file);
    return ResponseEntity.accepted().body(new UploadResponse(uploadId, "ACCEPTED"));
}

}

@service
public class UploadService {

private final UploadQueueProducer queueProducer;
private final Path tempDir = Paths.get("/tmp/uploads");

public String acceptUpload(MultipartFile file) {
    String uploadId = UUID.randomUUID().toString();
    Path target = tempDir.resolve(uploadId);

    // Blocking write, but quick: just park it on disk
    try (InputStream in = file.getInputStream()) {
        Files.copy(in, target);
    } catch (IOException e) {
        throw new RuntimeException(e);
    }

    // Enqueue a message for async processing
    queueProducer.send(new UploadMessage(uploadId, target.toString()));

    return uploadId;
}

}

This does block a servlet thread while the file is being uploaded, but it’s only busy writing to disk or S3 and then returns quickly. With reasonable timeouts, this is often acceptable.

b) Streaming / non-blocking with WebFlux

If you expect very large files or lots of concurrent uploads, you can build the API with Spring WebFlux and reactive types (Flux). That lets the server reuse fewer threads and apply backpressure:

@RestController
@RequestMapping("/uploads")
public class ReactiveUploadController {

private final ReactiveUploadService uploadService;

public ReactiveUploadController(ReactiveUploadService uploadService) {
    this.uploadService = uploadService;
}

@PostMapping(consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
public Mono<ResponseEntity<UploadResponse>> upload(@RequestPart("file") Flux<DataBuffer> fileFlux) {
    return uploadService.acceptUpload(fileFlux)
            .map(uploadId -> ResponseEntity.accepted()
                    .body(new UploadResponse(uploadId, "ACCEPTED")));
}

}

Service sketch:

@service
public class ReactiveUploadService {

private final UploadQueueProducer queueProducer;

public Mono<String> acceptUpload(Flux<DataBuffer> fileFlux) {
    String uploadId = UUID.randomUUID().toString();
    Path target = Paths.get("/tmp/uploads").resolve(uploadId);

    return DataBufferUtils.write(fileFlux, target)
            .then(Mono.fromCallable(() -> {
                queueProducer.send(new UploadMessage(uploadId, target.toString()));
                return uploadId;
            }));
}

}

This way you never load the whole file into memory, and you don’t block servlet threads.

If you want to go further, you can combine this with the AWS async S3 SDK and stream directly to S3 instead of disk.

3. Asynchronous processing

For the “do work later” part, queues are the usual pattern:

Use Kafka / RabbitMQ / SQS / Redis streams, etc.

Message contains: uploadId, tempPathOrUrl, userId, timestamp, etc.

A separate Spring Boot worker (could be Spring Batch or just a plain @scheduled or listener) does:

@KafkaListener(topics = "uploads")
public void handleUpload(UploadMessage message) {
// 1. Read from temp storage
// 2. Process / transform
// 3. Upload to S3
// 4. Update DB: status = COMPLETED / FAILED
}

Your main API can expose endpoints like:

GET /uploads/{uploadId}/status

so clients can poll or you can push via WebSocket / SSE.

4. Async controllers vs queues

@async or CompletableFuture in controllers does not solve the main problem: the HTTP connection is still open until you finish. Good for CPU offload but not for long, multi-minute jobs.

A queue + short HTTP request is the standard approach for long-running / heavy processing.

If you’re fine with blocking during the upload itself, you can keep controllers simple and move all heavy work behind the queue.

5. Direct-to-S3 pattern (if you want to offload uploads entirely)

For very high scale, you can skip uploading through your Spring Boot app:

API endpoint returns a pre-signed S3 URL and an uploadId.

Client uploads the file directly to S3.

S3 → event (S3 event / SNS / SQS / Lambda) → your worker / queue.

Your processing worker runs, then updates DB.

This removes large file traffic from your app entirely; it only handles metadata and orchestration.

6. Summary of recommended patterns

Use short-lived upload endpoints that:

Store file (disk, S3, object store) as a stream.

Publish a message to a queue.

Return 202 Accepted with an uploadId.

Implement workers (separate Spring Boot apps or modules) that:

Consume the queue.

Process files.

Upload to S3.

Update status in DB.

For heavy traffic / huge files:

Prefer WebFlux + streaming or direct-to-S3 with pre-signed URLs.

Avoid relying only on @async in controllers for long jobs; use queues instead.

You can pick the level of complexity depending on your scale: start with blocking MVC + queue + temp storage, and evolve to WebFlux / direct-to-S3 as traffic grows.