test: add config validity, rule snapshot, and option tests - #681
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a Bun-based test suite for the createConfig() ESLint flat-config generator, and wires those tests into CI/release so configuration validity and effective rule sets are continuously verified.
Changes:
- Add new tests for config validity (via
Linter), option behavior, and rule-set snapshots (via Bun snapshots). - Add shared test helpers for building configs and resolving effective rules with ESLint.
- Run
bun testinanalyze.ymlandrelease.yml, and includetests/intsconfig.jsonso type-checking/linting cover the test suite.
Reviewed changes
Copilot reviewed 8 out of 9 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tsconfig.json | Include tests/ so TypeScript checks and linting cover the new test suite. |
| tests/support.ts | Shared helpers to build configs and resolve effective rules/severities for assertions and snapshots. |
| tests/rule-snapshot.test.ts | Snapshot tests that lock in the enabled rule set per platform/style. |
| tests/create-config.test.ts | Behavioral tests for key createConfig() options (useImport/files/overrides/ignores/platform). |
| tests/config-validity.test.ts | Validates generated configs don’t throw under ESLint Linter and includes negative controls. |
| tests/snapshots/rule-snapshot.test.ts.snap | Baseline snapshots for rule-set lock-in tests. |
| CLAUDE.md | Update contributor docs to describe the new Bun test suite and its purpose/limits. |
| .github/workflows/release.yml | Run bun test as part of the release workflow. |
| .github/workflows/analyze.yml | Run bun test in CI analysis before building. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+21
to
+24
| test(`${platform} / ${style} / useImport=${String(useImport)}`, () => { | ||
| const config = build({ platform, style, useImport }); | ||
| const filePath = platform === "react" ? "src/sample.tsx" : "src/sample.ts"; | ||
|
|
Runs on Bun's built-in test runner, so no new dependency. Per request there is no `test` script in package.json — analyze.yml and release.yml call `bun test` directly. tests/config-validity.test.ts Builds all 8 platform/style/useImport combinations plus the files-scoped and overrides/ignores variants, and runs each through ESLint's Linter with configType "flat". ESLint validates rule ids and option schemas on first use and throws, so this catches the class of bug tsc cannot see through the config's `as unknown as ...` casts. It ends with three negative controls (unknown rule id, unknown rule option, invalid severity) asserting ESLint does throw. Without them the suite would pass against any config at all. tests/rule-snapshot.test.ts Snapshots the effective enabled rule set per platform/style. Explicitly-off rules are excluded so a diff shows only rules turning on, off, or changing severity. A plugin bump that silently changes what consumers get is then a reviewable diff instead of a surprise — eslint-plugin-unicorn 68 -> 73 added 29 error-level rules to `recommended` with no other signal. tests/create-config.test.ts Option behaviour: useImport toggling, files scoping, overrides precedence, ignores placement, platform-specific plugins. tsconfig.json now includes `tests`, so build:check and lint cover them. tsconfig.build.json is unchanged, so they stay out of dist/. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tests/file-extensions.test.ts Every platform/style against every supported extension (32 cases). Regression test for the glob mismatch fixed in 6769c57 — before it, the node platform threw on .mjs/.cjs/.jsx/.tsx/.mts/.cts and react on .mts/.cts, taking down the whole lint run rather than one file. tests/type-aware.test.ts Addresses the review point on config-validity.test.ts. Those tests assert a config *validates*; these assert it actually *runs*, by requiring a type-aware rule (@typescript-eslint/await-thenable) to fire — which is only possible if parser, TypeScript program and rule all wired up. This needs ESLint#lintText with filePath pointing at a file the tsconfig really includes; a path outside the program is silently skipped and reports nothing, which would read as a passing test. No fixture files are needed. config-validity.test.ts keeps its sample paths and now documents why they are valid: config validation runs before parsing, and an unincluded path yields a fatal message rather than an exception. The negative controls already prove validation fires on exactly those paths. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Viper-Bit
force-pushed
the
test/add-config-test-suite
branch
from
August 22, 2026 17:37
5d892c8 to
5203c53
Compare
The type-aware test passed locally and failed in CI: Expected to contain: "@typescript-eslint/await-thenable" Received: [ "@stylistic/lines-around-comment", "@stylistic/lines-around-comment" ] It used lintText with filePath pointing at src/globs.ts while passing different source as the text. Whether type-aware rules then fire depends on whether an earlier test already cached a TypeScript program for this tsconfig — the file is in the program, but with its on-disk contents, so the supplied text has no type information behind it. The non-type-aware rules still reported, which is why the run looked healthy but the type-aware assertion failed. The failing call took 447ms against 56ms for the one after it, i.e. no program was built for it. The violation now lives in tests/fixtures/type-aware.ts and is linted from disk with lintFiles, so the AST and the program always agree. The fixture is inside tsconfig.json's `include` (so it is part of the program) and inside eslint.config.ts's `ignores` (so the repo's own lint run skips it). Verified: passes in CI's file order and with type-aware.test.ts running last, and reverting the fixture to non-violating code fails both platform tests, so the assertion still has teeth. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
|
🎉 This PR is included in version 5.0.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runs on Bun's built-in test runner, so no new dependency. Per request there is no
testscript in package.json — analyze.yml and release.yml callbun testdirectly.tests/config-validity.test.ts
Builds all 8 platform/style/useImport combinations plus the files-scoped
and overrides/ignores variants, and runs each through ESLint's Linter with
configType "flat". ESLint validates rule ids and option schemas on first
use and throws, so this catches the class of bug tsc cannot see through
the config's
as unknown as ...casts.It ends with three negative controls (unknown rule id, unknown rule
option, invalid severity) asserting ESLint does throw. Without them the
suite would pass against any config at all.
tests/rule-snapshot.test.ts
Snapshots the effective enabled rule set per platform/style. Explicitly-off
rules are excluded so a diff shows only rules turning on, off, or changing
severity. A plugin bump that silently changes what consumers get is then a
reviewable diff instead of a surprise — eslint-plugin-unicorn 68 -> 73 added
29 error-level rules to
recommendedwith no other signal.tests/create-config.test.ts
Option behaviour: useImport toggling, files scoping, overrides precedence,
ignores placement, platform-specific plugins.
tsconfig.json now includes
tests, so build:check and lint cover them. tsconfig.build.json is unchanged, so they stay out of dist/.