fix(perf): stop loading hard-link-directory on every bit invocation; dual-metric self-diagnosing fs-read guard

[luvkapur]

Why

e2e_test_bbit (nightly, 32 nodes) has been red since 2026-07-23 — 19 of the last 21 failures are one single test out of ~2,900: the bit status filesystem-read guard (MAX_FILES_READ_STATUS = 1515, actual 1532 → 1541 and creeping).

Root cause

Diffing BIT_DEBUG_READ_FILE output of bvm-installed 2.0.33 (last green) vs 2.0.35 (first red) — only two commits apart — shows zero new logical modules (1623 → 1623). Instead, #10515's dependency change reshaped the released bundle's pnpm hoisted layout: the hard-link-directory cluster (plus a private fs-extra tree) moved from a small nested copy under one package to a larger nested copy under the compiler package (net +21 physical reads, duplicate copies 212 → 233). The nightly +1–3 creep is the same mechanism — every nightly bundle re-resolves latest transitive deps, and hoisting shifts occasionally add nested duplicates.

Changes

1. compiler.task.ts: lazy-import hard-link-directory — it is used only inside the build task, yet its top-level import pulled ~36 files into every bootstrap/bit status in the bundle layout. Verified the compiled dist defers the require until the task executes.
2. The bit status guard now tracks two metrics instead of one raw count:
   • Unique logical modules (threshold 1700): paths normalized past the last node_modules/, so all copies of the same module count once. Catches bit loading more code, is immune to the hoisting-drift noise class above, and converges across install flavors — one snapshot (files-snapshot-status.txt) serves repo and bundle runs alike. Measured baselines: 1364 (repo), 1623–1645 (bundles).
   • Duplicate copies (ceiling 350): physical files minus logical modules. Catches bit loading redundant copies/versions of packages — a class the logical metric alone is blind to, and a core motivation for this test. Measured baselines: ~99 (repo), 212–233 (bundles); headroom accommodates ordinary hoisting drift.
3. Failures are self-diagnosing, greens are self-measuring — a logical-modules red prints which modules are new vs the snapshot plus the full current list; a duplication red names the most-duplicated packages and the locations their copies load from; every run prints a total reads / physical files / logical modules / duplicate copies report line, accumulating true per-environment baselines in CI logs for tightening both thresholds from measurements.

Verification

• npm run lint green (oxlint + tsc).
• Snapshot-diff logic exercised against real captured outputs of 2.0.33 / 2.0.35 / 2.0.74; root cause re-reproduced independently from fresh bvm installs of 2.0.33/2.0.35.
• The bit status e2e passes on the repo flavor (total reads 1616, physical 1463, logical 1364, duplicates 99); the duplication failure path verified by artificially lowering the ceiling.
• bit compile teambit.compilation/compiler — dist verified lazy: await Promise.resolve().then(() => require('@teambit/toolbox.fs.hard-link-directory')).

[qodo-free-for-open-source-projects]

PR Summary by Qodo

Reduce status filesystem reads and add dual-metric diagnostics

🐞 Bug fix 🧪 Tests ✨ Enhancement 🕐 20-40 Minutes

AI Description

• Defers build-only hard-link loading to avoid unnecessary startup filesystem reads.
• Separates logical-module growth from dependency-layout duplication in the status guard.
• Adds actionable failure diagnostics, continuous metrics, and a shared logical-module baseline.

Diagram

graph TD
  A["Bit invocation"] -->|loads| B["Compiler task"] -->|build only| C["Lazy hard-link import"]
  A -->|traces reads| D["Read trace"] --> E["Metric analyzer"] --> F["Logical guard"] --> H["CI diagnostics"]
  E --> G["Duplication guard"] --> H

Loading

High-Level Assessment

The combined approach is appropriate: defer the build-only dependency at its sole use site, then measure logical code growth separately from physical duplication. Simply raising the raw-read threshold or maintaining install-layout-specific snapshots would preserve hoisting noise and provide weaker diagnostics, while removing duplication checks would miss real redundant loading.

Files changed (3) +1814 / -18

Bug fix (1) +5 / -1

compiler.task.tsLazy-load hard-link support during compiler builds +5/-1

Lazy-load hard-link support during compiler builds

• Moves hard-link-directory from a top-level import into the artifact-linking build step. This prevents its dependency tree from loading during unrelated Bit invocations such as bit status.

scopes/compilation/compiler/compiler.task.ts

Tests (2) +1809 / -17

files-snapshot-status.txtAdd status logical-module baseline snapshot +1645/-0

Add status logical-module baseline snapshot

• Adds a 1,645-entry, install-layout-independent baseline of logical modules loaded by bit status. The guard uses it to identify newly loaded modules without treating pnpm hoisting paths as code growth.

e2e/performance/files-snapshot-status.txt

filesystem-read.e2e.tsReplace raw status read limit with dual performance guards +164/-17

Replace raw status read limit with dual performance guards

• Normalizes traced paths into logical module identifiers and independently limits logical growth and duplicate physical copies. It now logs all metrics on every run and emits focused module-diff or package-location diagnostics on failure.

e2e/performance/filesystem-read.e2e.ts

[qodo-free-for-open-source-projects]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Status snapshot seems mismatched ✓ Resolved 🐞 Bug ☼ Reliability

Description

files-snapshot-status.txt contains 1,865 entries, while the test comments describe the released
bundle as ~1,540 reads and the guard threshold is 1,650; this strongly suggests the snapshot was
generated from a different flavor/run than what the test documents. If/when the bit status guard
fails, getNewlyLoadedFiles() diagnostics can become misleading or miss the actual newly-loaded
files you expect it to highlight for that environment.

Code

e2e/performance/files-snapshot-status.txt[R1862-1865]

+lodash.compact/index.js
+@teambit/graph/dist/component-id-graph.js
+@teambit/status/dist/status-formatter.js
+@teambit/cli/dist/pager.js

Evidence

The test explicitly documents an expected released-bundle read count (~1,540) and sets a threshold
of 1,650, but the newly-added status snapshot file itself has 1,865 entries, indicating the snapshot
likely doesn’t represent the documented baseline/flavor and can therefore mislead diagnostics.

e2e/performance/filesystem-read.e2e.ts[14-35]
e2e/performance/filesystem-read.e2e.ts[85-99]
e2e/performance/files-snapshot-status.txt[1851-1865]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`files-snapshot-status.txt` appears inconsistent with the documented expected `bit status` read counts and the configured threshold, which can reduce the usefulness of the self-diagnosing diff when the guard trips.
### Issue Context
- The test’s `MAX_FILES_READ_STATUS` is set to 1650 and comments state the released/hoisted bundle is ~1540 reads.
- The committed `files-snapshot-status.txt` currently contains 1865 lines (entries).
- `getNewlyLoadedFiles(output, 'files-snapshot-status.txt')` uses this file only for diagnostics, so an overinclusive/mismatched snapshot makes failure output noisy or misleading.
### Fix Focus Areas
- e2e/performance/filesystem-read.e2e.ts[14-99]
- e2e/performance/files-snapshot-status.txt[1-1865]
### Suggested fix
1. Regenerate `files-snapshot-status.txt` using the same environment/flavor the comments describe (released bundle hoisted layout vs repo `.pnpm` layout), and ensure it corresponds to the expected baseline.
2. Align the inline documentation (expected counts) with the regenerated snapshot.
3. Optionally add a lightweight sanity check (comment or assertion) to prevent committing a status snapshot whose size is wildly inconsistent with the expected baseline/threshold.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Tip of the day

💡 Did you know, you can start a comment with 'qodo' or '@qodo' to chat about any finding

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[davidfirst]

Added a second guard in 4874ced: the logical-module metric alone is blind to duplication — if bit starts loading two copies/versions of the same package (real extra I/O and memory), logical modules stay flat. Since duplication was a core motivation for this test, the guard now also fails when duplicate copies (physical files minus logical modules) exceed a ceiling (350; measured baselines: ~99 repo flavor, 211–233 released bundles). The failure output names the most-duplicated packages and the versioned locations their copies load from, e.g.:

fs-extra: 52 duplicate file read(s) from 3 locations:
  - .pnpm/fs-extra@10.0.0/
  - .pnpm/fs-extra@10.1.0/
  - .pnpm/fs-extra@11.4.0/

Also merged latest master into the branch. Verified: lint green; the bit-status e2e passes on the repo flavor (report line: total reads 1616, physical 1463, logical 1364, duplicates 99), and the duplication failure path renders correctly when the ceiling is artificially lowered.

[qodo-free-for-open-source-projects]

Code review by qodo was updated up to the latest commit 4b980ad