Skip to content

Add an optional native parser backend with pure-Python fallback - #694

Open
lowmiaq-gmail wants to merge 3 commits into
theskumar:mainfrom
lowmiaq-gmail:codex/native-backend-adapter-20260827
Open

Add an optional native parser backend with pure-Python fallback#694
lowmiaq-gmail wants to merge 3 commits into
theskumar:mainfrom
lowmiaq-gmail:codex/native-backend-adapter-20260827

Conversation

@lowmiaq-gmail

Copy link
Copy Markdown

Problem

Some parse-heavy consumers of python-dotenv can spend measurable time in the parser, while the default package must remain pure Python and portable.

Proposal

Add a small optional adapter at the parser boundary:

  • pip install python-dotenv remains the current pure-Python path.
  • pip install "python-dotenv[native]" opts into fast-dotenv-rs-backend on CPython.
  • PyPy and environments without the backend continue using the existing parser.
  • The adapter returns the existing upstream Binding and Original objects.
  • Backend contract failures are visible; they are not silently converted into a false native pass.

The backend-only distribution does not install dotenv or a console script, so it can coexist with this package. The Rust implementation and wheels are maintained separately in fast-dotenv-rs.

Compatibility

The patch is based on upstream main at a00cb2eed0704cd6d2071b2004c37e95ccc86ee5 and only changes the parser boundary plus the optional extra. Existing Binding, Original, parse_stream, CLI, and fallback behavior remain the public contract.

Local validation:

  • Upstream suite: 263 passed, 1 skipped in an isolated CPython 3.14 environment; one unrelated macOS printenv --version fixture differs from Linux behavior.
  • Native adapter tests pass.
  • Differential corpus: 1,088 records, zero mismatches across Python/native paths.
  • Property-style corpus: 500 valid + 500 malformed inputs, zero mismatches.
  • Backend CI: 10/10 jobs passed across Linux x86_64/arm64, macOS arm64/Intel, and Windows x86_64 on CPython 3.10 and 3.14.

Performance evidence

In the real-consumer benchmark, 21/30 p50 rows improved, 8 were neutral, and 1 small cold-start CLI row regressed. Representative absolute savings were approximately 0.13 ms for a small dotenv_values workload, 0.75 ms for medium input, and 4.94 ms for large input. The adapter is intentionally optional because small cold-start workloads are not universally faster.

Maintenance and rollback

  • No Rust compiler is required for the default install.
  • The native dependency is restricted to CPython; PyPy remains pure Python.
  • No compatible backend means the existing parser remains active.
  • The backend wheel matrix and release workflow are maintained in the separate backend repository.

This is the concrete implementation for #693; the backend-only package release is being completed independently so that the upstream package does not need to own a Rust build pipeline.

@lowmiaq-gmail

Copy link
Copy Markdown
Author

Update: the backend-only distribution is now publicly available as fast-dotenv-rs-backend 0.1.1.

The release workflow built and audited 5 platform wheels plus an sdist; the publish job succeeded, and fresh public PyPI install/coexistence/native-call checks pass locally. Evidence run: https://github.com/lowmiaq-gmail/fast-dotenv-rs/actions/runs/33084678870. The PR remains default-off: without the optional backend, python-dotenv stays on the existing pure-Python parser.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant