Programmatic usage of v0.5.0: replication issues (restore errors, missing LTX files, compaction failure)

[jaovitubr]

I implemented the following code to handle replication using Litestream, but I am not sure if this is the correct way to do it.

I’ve noticed some issues:

1. Sometimes there are gaps of several minutes where no LTX files are sent to my ReplicaClient, even though writes are happening in the database. After some time, replication resumes again (sometimes it doesn’t).
2. Occasionally the database restore fails on startup.
3. On startup, I also see a compaction error in the logs.

Log excerpt:

2025/08/29 16:15:46 replica.go:27: INFO restore database failed db=/var/lib/daemon/containers/1/data/database/sqlite.db error="decode database: decode page 48: copy page 56 header: nonsequential page numbers in snapshot transaction: 54,56"
2025/08/29 16:15:46 replica.go:42: INFO replication started db=/var/lib/daemon/containers/1/data/database/sqlite.db
2025/08/29 16:15:46 store.go:139: INFO starting compaction monitor level=1 interval=5m0s
2025/08/29 16:15:46 store.go:139: INFO starting compaction monitor level=2 interval=1h0m0s
2025/08/29 16:15:46 store.go:139: INFO starting compaction monitor level=9 interval=24h0m0s
2025/08/29 16:15:46 store.go:165: ERROR compaction failed level=9 error="page size not initialized yet"
2025/08/29 16:15:47 main.go:188: Container 1 started!
2025/08/29 16:15:48 db.go:1370: INFO deleting ltx file db=sqlite.db level=0 minTXID=000000000000041c maxTXID=000000000000041c
2025/08/29 16:15:48 db.go:1370: INFO deleting ltx file db=sqlite.db level=0 minTXID=000000000000041d maxTXID=000000000000041d
...
2025/08/29 16:15:48 store.go:222: INFO compaction complete level=1 txid.min=000000000000041d txid.max=0000000000000515 size=450621

After this log, no LTX files are sent for several minutes, even though the database continues to receive writes. Eventually replication resumes, but sometimes it does not.

Code:

package replica

import (
	"context"
	"fmt"
	"log/slog"
	"os"
	"time"

	"github.com/benbjohnson/litestream"
)

type Replication struct {
	DatabasePath string
	Prefix       string
}

func (rep *Replication) Start(ctx context.Context) error {
	db := litestream.NewDB(rep.DatabasePath)
	db.Replica = litestream.NewReplica(db)
	db.Replica.Client = NewReplicaClient(rep.Prefix)

	if err := restoreDatabase(ctx, db.Replica); err != nil {
		slog.Info("restore database failed", "db", rep.DatabasePath, "error", err)
	}

	levels := litestream.CompactionLevels{
		{Level: 0},
		{Level: 1, Interval: 5 * time.Minute},
		{Level: 2, Interval: 1 * time.Hour},
	}

	store := litestream.NewStore([]*litestream.DB{db}, levels)

	if err := store.Open(ctx); err != nil {
		return fmt.Errorf("failed to open database: %w", err)
	}

	slog.Info("replication started", "db", rep.DatabasePath)

	go func() {
		defer store.Close(context.Background())
		<-ctx.Done()
		slog.Info("replication stopped", "db", rep.DatabasePath)
	}()

	return nil
}

func restoreDatabase(ctx context.Context, replica *litestream.Replica) error {
	opt := litestream.NewRestoreOptions()
	opt.OutputPath = replica.DB().Path()

	if _, err := os.Stat(opt.OutputPath); err == nil {
		slog.Info("local database already exists, skipping restore", "path", opt.OutputPath)
		return nil
	} else if !os.IsNotExist(err) {
		return fmt.Errorf("stat failed: %w", err)
	}

	updatedAt, _ := replica.CalcRestoreTarget(ctx, opt)

	if updatedAt.IsZero() {
		fmt.Printf("no restore targets found, creating new database\n")
		return nil
	}

	return replica.Restore(ctx, opt)
}

Questions:

• Is this the correct way to configure replication with a ReplicaClient?
• What could cause the restore errors and "nonsequential page numbers" message?
• Why does replication sometimes stop sending LTX files for several minutes even while the DB is receiving writes?
• Is the "page size not initialized yet" error something I should worry about, or is it expected during startup?

[jaovitubr]

My ReplicaClient is an HTTP client that writes to a Cloudflare Worker, which then stores data in R2.

[youngpm]

I've seen similar things while using recent commits; I wonder if this is related to some of the newer development in litestream around using ltx? #645

[jaovitubr]

v0.5.0 LTX

[jaovitubr]

I found that I need to manually call the Sync() function before opening the store, otherwise I get the error "page size not initialized yet". The solution is to add db.Sync(ctx) before store.Open(ctx):

store := litestream.NewStore([]*litestream.DB{db}, levels)

db.Sync(ctx)

if err := store.Open(ctx); err != nil {
    return fmt.Errorf("failed to open database: %w", err)
}

Without the manual Sync() call, the database page size isn't properly initialized, causing the initialization error.