Skip to content

Keyword matching silently fails for every non-Latin script #19

Description

@CharanMN7

The bug

Keywords written in Cyrillic, Arabic, Chinese, Japanese, Korean, Devanagari, Thai — any non-Latin script — never match anything. The rule saves fine, the comment arrives fine, and the DM silently never sends.

Confirmed against the real matcher:

  MATCH  Latin control       "guide"  in "guide please"
  MISS   Cyrillic            "гид"    in "нужен гид пожалуйста"
  MISS   Chinese             "指南"    in "请发指南给我"
  MISS   Arabic              "الدليل"  in "أريد الدليل"
  MISS   Japanese            "ガイド"   in "ガイドください"

Why it happens

keywordMatches in src/match.ts builds the pattern \b<keyword>\b:

const re = new RegExp(`\\b${escapeRegex(k)}\\b`, 'u');

JavaScript defines \b as a boundary between [A-Za-z0-9_] and anything else — even with the u flag. For a Cyrillic keyword, both sides of the boundary are non-ASCII, so \b never asserts and the match always fails.

Note that normalizeCommentText is already Unicode-correct — it uses \p{P} and \p{S} and does not strip non-Latin letters. The bug is only in the boundary assertion.

Suggested approach

Replace the ASCII boundary with a Unicode-aware lookaround built from \p{L} and \p{N}:

const B = '[\\p{L}\\p{N}]';
const re = new RegExp(`(?<!${B})${escapeRegex(k)}(?!${B})`, 'u');

Lookbehind is supported in workerd and in Node 20, so both the Worker and the test runner are fine.

Worth checking while you are in there: scripts without word separators (Chinese, Japanese, Thai) have no meaningful notion of a word boundary at all. The lookaround above does the right thing for them — it matches anywhere the keyword is not glued to another letter or digit — but do add test cases so the intent is recorded.

Acceptance criteria

  • All five cases in the table above match
  • "guide" still does not match "guides" or "guidebook" (the existing behaviour that \b was there for)
  • Tests added to scripts/test-unit.ts covering Latin, Cyrillic, Arabic, CJK, and the negative Latin cases
  • npm test and npm run typecheck pass

Where to look

  • src/match.tskeywordMatches, and normalizeCommentText for context
  • scripts/test-unit.ts — the existing keywordMatches and findMatchingRule suites

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: matchingKeyword matching and text normalisationbugSomething isn't workingeffort: smallA focused change, roughly an eveninggood first issueGood for newcomers

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions