fix: decode numeric character references in text

[Kjubikstronk]

Marked version: 18.0.11 (53cb13f)

Markdown flavor: CommonMark

Description

CommonMark treats a numeric character reference as equivalent to the character it names, everywhere except code spans and code blocks. marked passes them through unchanged, so they reach the output exactly as written.

escapeTestNoEncode deliberately declines to escape anything shaped like an entity:

/[<>"']|&(?!(#\d{1,7}|#[Xx][a-fA-F0-9]{1,6}|\w+);)/

which leaves it to the browser to resolve. That works for display but does not match the spec, and it is visible to anything reading the output as text rather than rendering it.

Expectation

# Ӓ   ->  # Ӓ
" ಫ  ->  " ಫ
foo

bar -> foo, blank line, bar
	foo         ->  a tab, then foo

Result

Each one is emitted as written: #, ", 
, 	.

What was attempted

Decoding numeric references in inlineText, so the token carries the character and the existing escaping handles the rest. Zero, out of range, and surrogate values become the replacement character, per the spec. Raw HTML blocks keep whatever they were written with.

This deliberately does not touch named references. © and friends need the HTML5 named table, which is the dependency question raised in #4050. The numeric half needs no table at all, only digit parsing, which is why it is separate.

Measured over the whole spec with the spec's own test/normalize.py: 28 failures before, 24 after. The four that move are 26, 27, 39 and 40, and nothing else changes, in both the CommonMark and GFM runs.

Worth noting why this was not caught: # and # are equivalent once parsed as HTML, so html-differ reports them equal. All four of these already report as passing and carry no shouldFail. Same blind spot as #4073, #4074 and #4075, in a different corner.

Contributor

• Test(s) exist to ensure functionality and minimize regression (if no tests added, list tests covering this PR); or,
• no tests required for this PR.
• If submitting new feature, it has been documented in the appropriate places.

Five tests in test/unit/marked.test.js under numeric character references. Three fail without the change; the other two are controls, an invalid reference that must stay literal and a code span that must not decode. Full spec suite (1789) and unit suite (196) pass.

Committer

In most cases, this should be a different person than the contributor.

• CI is green (no forced merge required).
• Squash and Merge PR following conventional commit guidelines.

[vercel]

@Kjubikstronk is attempting to deploy a commit to the MarkedJS Team on Vercel.

A member of the Team first needs to authorize it.