Skip to content

fix: fold Latin diacritics so accented comments match unaccented keywords - #58

Open
dchaudhari7177 wants to merge 1 commit into
CharanMN7:masterfrom
dchaudhari7177:fix/latin-diacritic-folding
Open

fix: fold Latin diacritics so accented comments match unaccented keywords#58
dchaudhari7177 wants to merge 1 commit into
CharanMN7:masterfrom
dchaudhari7177:fix/latin-diacritic-folding

Conversation

@dchaudhari7177

Copy link
Copy Markdown

Closes #20

Reproduced first on master:

MISS   "preco"  in "qual o PREÇO?"
MATCH  "preço"  in "qual o preço"

The change

foldLatinDiacritics applied at the top of normalizeCommentText. As you noted, that function already normalises both the keyword and the comment, so one change covers both sides.

text.normalize('NFD').replace(/(?<=[A-Za-z])\p{Diacritic}/gu, '').normalize('NFC')

The scope you asked for falls out of NFD directly: decomposition places the base character immediately before its combining mark, so "only drop a mark whose base is Basic Latin" is a single lookbehind. No script table to keep in sync.

I added the trailing NFC so the function's output stays canonical rather than half-decomposed. Matching would work without it — both sides run through the same normaliser — but it means every untouched script comes out byte-identical to its input, which is a much easier property to assert than "equivalent under some normalisation form".

Acceptance criteria

criterion result
preco matches PREÇO, preço, PRECO ✅ all three
cafe matches café; uber matches über
й is not folded to и normalizeCommentText('й') !== normalizeCommentText('и')
is not folded to ✅ same assertion
Devanagari and Thai survive unchanged ✅ byte-identical, along with Arabic, Korean and Cyrillic

Two extra cases worth having: preço also matches an unaccented preco comment, since the fold is symmetric; and İstanbul (U+0130, a precomposed capital with dot above) folds correctly, because NFD decomposes it before the lowercase step.

Tests

A Latin diacritic folding block in scripts/test-unit.ts: the eight matching cases as a table, a dedicated test each for the Cyrillic breve and the Japanese dakuten asserting both non-equality and non-matching, a loop asserting six non-Latin strings survive normalisation byte-for-byte, and one guarding that folding widens what counts as equal without widening what counts as a boundarycafe still does not match cafeteria.

Reverting only src/match.ts while keeping the tests fails 7 of the 42.

Verification

npm test — 42 pass, 0 fail. npm run typecheck — clean. No new dependency.

Note this branches from master independently of #56 (issue #19); both touch src/match.ts but in different functions, so whichever lands first, the second is a clean rebase.

…ords

A comment reading 'qual o PREÇO?' did not match a keyword of 'preco', so
a creator had to guess every accent variation their audience might type.

Fold in normalizeCommentText, which already runs over both the keyword
and the comment, so one change covers both sides.

Scoped to Latin: a blanket strip of every combining mark turns Cyrillic
й into и and Japanese が into か -- different letters, not accented ones
-- and in Devanagari, Thai and Arabic the marks are vowels rather than
decoration. NFD puts the base character immediately before its mark, so
the scope is one lookbehind on [A-Za-z]. Recomposed with NFC afterwards,
so untouched scripts come out byte-identical.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant