I kept hitting the same problem in my data pipeline: how to efficiently encrypt compressed streams without buffering entire files or large chunks, while keeping the ability to append data later without corrupting existing content. The existing solutions either:
-
Required loading everything into memory (
crypto/aes+bytes.Buffer), or -
Broke streaming capabilities (I mean true streaming with tiny buffers and the ability to append at any time)
So I built rusjoan/streamcrypt – a minimal Go library that:
✅ Wraps any io.Reader/io.Writer with AES-GCM encryption
✅ Preserves streaming (constant memory usage)
✅ Works seamlessly with gzip/zstd
✅ Supports appending data to existing payloads without corruption
✅ Adds only 32 bytes per chunk (~13% overhead for JSON+GZIP)
Why this matters:
-
Process TB-scale data with KB-scale RAM
-
Encrypt before/after compression without temp files
-
Zero dependencies (pure Go stdlib)
Basic usage:
var enc, _ = streamcrypt.NewEncryptor(secret) // Encrypting gzip stream gzip.NewWriter(enc.Seal(file)) // Decrypting gzip.NewReader(enc.Open(file))
Benchmarks:
// allocations
goos: darwin
goarch: arm64
pkg: github.com/rusjoan/streamcrypt
cpu: Apple M1 Pro
BenchmarkTee
BenchmarkTee/rnd->encryptor->discard
BenchmarkTee/rnd->encryptor->discard-10 765747 1525 ns/op 0 B/op 0 allocs/op
PASS
// heap grow
=== RUN TestMemoryOverhead
streamcrypt_test.go:218: Size: 16.0 KiB, Memory delta: 704 B
streamcrypt_test.go:218: Size: 1.0 MiB, Memory delta: 576 B
streamcrypt_test.go:218: Size: 32.0 MiB, Memory delta: 576 B
streamcrypt_test.go:218: Size: 1.0 GiB, Memory delta: 576 B
--- PASS: TestMemoryOverhead (2.42s)
PASSNext version plans:
-
Allow custom encryption methods beyond built-in AES-GCM
Would love community feedback on:
-
Real-world use cases I haven't considered
-
Any similar solutions I might have missed (I did thorough research)
-
Ways to further reduce the overhead
This is my first open-source library after a long time of being read-only, so I'd really appreciate your support!
|
