Skip to content

fix: handle non-UTF-8 packed refs - #2216

Closed
Sohel2309 wants to merge 1 commit into
gitpython-developers:mainfrom
Sohel2309:fix/2064-packed-refs-encoding
Closed

fix: handle non-UTF-8 packed refs#2216
Sohel2309 wants to merge 1 commit into
gitpython-developers:mainfrom
Sohel2309:fix/2064-packed-refs-encoding

Conversation

@Sohel2309

@Sohel2309 Sohel2309 commented Aug 14, 2026

Copy link
Copy Markdown

Fixes #2064.

Bug

GitPython raises UnicodeDecodeError when iterating .git/packed-refs if a packed ref name contains bytes that are not valid UTF-8.

Git ref names are byte strings and are not guaranteed to contain valid UTF-8. A single non-UTF-8 ref should not make the entire packed-refs file unreadable.

Root Cause

SymbolicReference._iter_packed_refs() opened packed-refs in text mode with strict UTF-8 decoding:

open(..., "rt", encoding="UTF-8")

This caused UnicodeDecodeError when a packed ref contained non-UTF-8 bytes.

Fix

Read packed-refs in binary mode and decode each line using surrogateescape.

This follows the existing lenient byte-to-string handling used elsewhere in GitPython and preserves the original bytes during round-tripping instead of raising an exception.

Regression Test

Added a regression test covering a packed ref containing a non-UTF-8 byte and verifying that iterating packed refs does not raise UnicodeDecodeError.

Validation

  • Targeted regression test: passed
  • test/test_refs.py: regression test passed
  • ruff check: passed
  • ruff format --check: passed
  • git diff --check: passed
@Byron

Byron commented Aug 30, 2026

Copy link
Copy Markdown
Member

I don't think that making this pass is the right choice here.
Instead, GitPython shouldn't assume encodings at all, or at most ascii, and get its typing straight.
A lot of work will be needed to fix this, and I plan to remove all the pure python implementations in favor of using the Git binary instead.
Thanks for your understanding.

@Byron Byron closed this Aug 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants