EvBricks is an Azure Event Hubs to Databricks Zerobus bridge.

It keeps the proven EvSnow source-side shape: Event Hub receive, batching, message metadata, and checkpoint-after-success semantics. It replaces the Snowflake destination with Databricks Zerobus writes to Unity Catalog managed Delta tables.

uv sync
cp config/evbricks.example.toml config/evbricks.toml
cp .env.example .env

Local files with deployment names and secrets must stay out of Git:

git check-ignore -v .env config/evbricks.toml

Shape-only validation does not open Event Hub, Databricks, or Postgres clients:

uv run evbricks validate-config --config-file config/evbricks.example.toml
uv run evbricks run --config-file config/evbricks.example.toml --dry-run

For a live run, validation must include Databricks service-principal settings and a Postgres checkpoint backend. The recommended Databricks-hosted control backend is Lakebase Postgres with a native Postgres password role:

uv run evbricks validate-config \
  --config-file config/evbricks.toml \
  --env-file .env \
  --require-runtime-secrets

Zerobus requires a pre-existing Unity Catalog managed Delta table in supported non-default managed storage. Start from sql/create_eventhub_events.sql, adjust the catalog and schema to the configured target, then grant the Databricks service principal:

GRANT USE CATALOG ON CATALOG <catalog> TO `<service-principal-uuid>`;
GRANT USE SCHEMA ON SCHEMA <catalog>.<schema> TO `<service-principal-uuid>`;
GRANT MODIFY, SELECT ON TABLE <catalog>.<schema>.eventhub_events TO `<service-principal-uuid>`;

Configure the runtime values in .env and keep them untracked:

DATABRICKS_WORKSPACE_URL=https://<workspace-host>
ZEROBUS_SERVER_ENDPOINT=https://<workspace-id>.zerobus.<region>.azuredatabricks.net

Use a managed Delta table in Unity Catalog backed by explicit ADLS Gen2 managed storage:

catalog: <catalog>
storage credential: <storage-credential>
external location: <external-location>

Before live smoke, confirm the target table is managed, in the same supported region as the workspace, and not backed by unsupported default/private-endpoint storage.

Use Databricks Lakebase Postgres for the EvBricks checkpoint/control tables when you want the control DB inside Databricks. Do not run an ad hoc Postgres server on Databricks compute for durable control state.

The working Lakebase control configuration is:

[control]
backend = "postgres"
target_db = "databricks_postgres"
target_schema = "evbricks"
target_table = "ingestion_status"

CONTROL_PG_HOST=<lakebase-direct-read-write-endpoint-host>
CONTROL_PG_PORT=5432
CONTROL_PG_USER=evbricks_runtime
CONTROL_PG_PASSWORD=<lakebase-native-postgres-password>
CONTROL_PG_SSLMODE=require
CONTROL_PG_AUTH_MODE=password

See Lakebase control DB runbook.

Acceptance for a live smoke run:

• EvBricks opens Zerobus before Event Hub messages are sent.
• The EvSnow sender emits a small batch with a unique run ID.
• Databricks returns the same number of matching rows for that run ID.
• Payload sequence IDs match the sent range.
• Event Hub metadata matched namespace, hub, and $Default consumer group.
• Duplicate identity rows were 0.
• Lakebase checkpoint state advances.

• Published documentation
• Zensical documentation home
• What EvBricks is
• Run a live smoke test
• Configuration reference
• Migration plan
• Local smoke runbook
• Lakebase control DB runbook

Build the docs site with Zensical:

uv run --group docs zensical build --strict --clean

Preview locally:

uv run --group docs zensical serve

The repository includes a GitHub Actions workflow at .github/workflows/docs.yml that builds the Zensical site and deploys the generated site/ directory to GitHub Pages. In GitHub repository settings, configure Pages to use GitHub Actions as the publishing source.