Merge pull request #22 from Lolo1222/main

add passnet skills

Author: hanhainebula

skills/task-oriented/Passnet/skills/passnet-feedback/SKILL.md

@@ -0,0 +1,145 @@
+---
+name: passnet-feedback
+description: >
+  Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification
+  WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs,
+  and eval-log parsing into per-variant status + estimated score + failure classification.
+  Use BEFORE every GPU evaluation (check_pattern), at the START of a sample
+  (analyze_graph), and AFTER every evaluation (parse_eval_log).
+---
+
+GPU evaluations cost minutes and are rate-limited; these tools answer in seconds. The
+iteration loop is: `analyze_graph` once → author passes → `check_pattern` until green →
+GPU evaluate → `parse_eval_log` → fix the top issue → repeat.
+
+All scripts live in the `scripts/` directory next to this `SKILL.md`. In this repo, prefer
+`SCRIPTS=<repo>/.claude/skills/passnet-feedback/scripts`. Invoke
+`python3 $SCRIPTS/<name>.py ...` from anywhere. They need: the PassNet repo importable
+(auto-detected from the sample dir's `entry.sh` symlink, or set
+`PYTHONPATH`/`PASSNET_ROOT`), torch, and for smoke/bench steps a GPU
+(`CUDA_VISIBLE_DEVICES` respected; they degrade to CPU structure-checks without one).
+
+## 1. `analyze_graph.py` — where is the bottleneck, what is matchable
+
+```bash
+python3 $SCRIPTS/analyze_graph.py --sample-dir <sample_root> [--bench] [--max-variants N]
+# default --max-variants 3, dtype-spread
+```
+- Captures the REAL dynamo graph of each variant (exactly what passes must match).
+- Per node: op kind, target, written args/kwargs form, output shape/dtype, and callable
+  pattern **matchability** (`method`/`C-bound` = mirror exactly; `PY-sig positional` =
+  matchable; `PY-sig kwargs/partial` = normal callable pattern will normalize differently).
+  A high-value kwargs-form Python functional region may still be recoverable with an exact
+  manual FX `GraphModule` pattern; use passnet-pattern-fusion and confirm with
+  `check_pattern.py`.
+- `--bench` (GPU): per-node eager µs via an instrumented FX interpreter + whole-forward
+  eager e2e estimate → tells you the absorbable time and a ROUGH speedup ceiling. The
+  ceiling assumes a ~70 µs fixed tax — real tax grows with FX node count (150–230 µs on
+  13-node graphs); treat the printed ceiling as optimistic and recalibrate after your
+  first eval.
+- Use the region table at the bottom as the starting plan: it lists maximal runs of
+  matchable nodes with their absorbed-µs totals.
+
+## 2. `check_pattern.py` — pre-flight a pass_dir in seconds (run before EVERY eval)
+
+```bash
+python3 $SCRIPTS/check_pattern.py --sample-dir <sample_root> [--pass-dir <dir>] [--smoke [--bench]]
+# --max-variants N (default 8, dtype-spread; 0 = all). --bench implies --smoke.
+# --smoke/--bench run on ONE variant per dtype; matching runs on all chosen variants.
+```
+Replicates the harness pipeline faithfully and reports:
+1. JSON manifest sanity (names ↔ files).
+2. AST validation per pass (the real `validate_pass_source` when repo importable).
+3. `replacement_func()` stability + **distinct-function count** (must be 1 when >1 pass —
+   otherwise passes WILL be silently dropped by the limit).
+4. Pattern trace → real `SubgraphMatcher` against each variant's dynamo graph:
+   match count per pass per variant, single-output check, containment check, and on
+   mismatch a node-by-node nearest-miss diff (pattern form vs graph form).
+5. Verdict per variant: would the run match (≥1 pass), or early-exit at 0.1.
+6. `--smoke` (GPU): actually applies the passes (real `PassMgrBackend`) and runs one
+   poisoned warmup call + numeric comparison vs eager at the dtype's baseline tolerance —
+   catches Unauthorized-Operator, dtype mismatch, and gross numeric bugs pre-eval.
+7. `--bench` (GPU): micro-benchmarks compiled-vs-eager e2e (200 calls) per variant for a
+   speedup preview (no dynamo guards, so real eval ≈ a few % worse).
+
+CPU-only output can validate structure and matchability triage, but it does not prove poisoned-wrapper legality, dtype behavior, numeric correctness, or real speed. GPU smoke/bench or a completed evaluation is still required for those claims.
+
+`--smoke` is a one-shot semantic check, not a full harness proof. Be extra cautious when a
+replacement absorbs `inplace=True` nodes or other side-effectful behavior: a one-shot compare
+can pass while repeated benchmark calls expose mutation/aliasing differences. For those
+regions, either leave the in-place node outside the pattern or confirm repeated-call behavior
+with a harness-style check/completed evaluation.
+
+Green output = safe to spend a GPU evaluation. Any red line tells you which skill to open:
+match issues → passnet-pattern-fusion §7; poison/API issues → passnet-pattern-fusion §6;
+numeric issues → passnet-triton-opt §3.
+
+## 3. `parse_eval_log.py` — turn an eval into decisions
+
+```bash
+python3 $SCRIPTS/parse_eval_log.py <validation.log | evaluate_response.json>
+# also reads the "stdout"/"stderr" fields of a saved /evaluate JSON response
+```
+Prints:
+- Per-variant table: dtype | status | e2e/gpu speedup | eager/compiled medians | max_diff |
+  passes applied/failed.
+- Failure classification with the fix pointer:
+  - localhost connection failure from a Codex managed sandbox → retry normal bounded curl once, then retry through the approved escalation path if available before diagnosing service downtime.
+  - empty, interrupted, HTTP000-style, status-corrupted, or non-JSON `/evaluate` response → no completed eval; leave score, speed, correctness, and pass-matched metrics null, retry service access/upload state if appropriate, and do not interpret any performance or correctness result.
+  - `no pass matched` + diagnostic lines → pattern form problem (which pass, which node).
+  - `AssertionError` in `_replace_pattern` → multi-output pattern (rule 1).
+  - `Unauthorized Operator (aten.xxx)` → illegal torch op in wrapper (poison).
+  - `Detected hacking behavior` → AST validation rejected a file (it was NOT loaded).
+  - `Loaded N passes` < listed → replacement_func limit dropped passes (shared dispatch!).
+  - dtype mismatch / accuracy with max_diff vs that dtype's baseline tolerance.
+  - replacement crash mentioning a returning node with no users → the pattern likely matched
+    userless/dead duplicate work; re-anchor the region on an observable value or use a proven
+    graph-rewrite approach instead of a normal output replacement.
+  - evaluator/environment fluctuation messages, timing-stability errors, or rerun requests
+    with otherwise clean matching and successful-variant correctness → classify separately
+    from numeric, no-match, unauthorized-operator, timeout, and OOM. If the pass pre-flights
+    cleanly and successful variants are correct, an unchanged rerun can be a valid use of
+    remaining evaluation budget.
+  - timeout/OOM hints.
+- **Estimated sample score** using the real ES(t) weights, plus per-variant rectified
+  speedups — so you know the score impact of fixing each failing variant before re-running.
+
+For round or multi-worker aggregation, keep family labels separate:
+
+- `triage_family`: the static family assigned before implementation.
+- `worker_confirmed_family`: the family the worker confirmed after reading the graph and
+  pre-flight results.
+- `actual_winning_region_family`: the family of the best completed-evaluation state.
+
+Do not aggregate a win under the original triage family if the best completed state belongs
+to a different region family. Round summaries should report, per family: number of completed
+workers, score-above-eager count, correct-but-slow count, numeric-preflight-blocked count,
+correctness-regressed-on-larger-region count, larger-region-attempted count, and evaluator
+instability count. Keep evaluator/environment fluctuation separate from numeric, no-match,
+unauthorized-operator, timeout, and OOM.
+
+## 4. Reading raw logs yourself (when needed)
+
+Key markers in `stdout`/`validation.log`:
+`[PassMgrBackend] Loaded/Applied/failed to match/Diagnostic`,
+`[Result] status: success|failed`, `[Speedup][e2e]/[gpu]`,
+`[Performance][eager|compiled]` (medians), `[Correctness][max_diff|mean_diff|equal]`,
+`[Datatype][eager|compiled]`, `debug-model-execution <ExcType>` (crash),
+`Has Any pass matched? [True|False]`, `aggregated_speedup=...`.
+The service strips `Trial`/`[Profiling]`/`all_close` lines; everything above survives.
+
+## 5. Bottleneck decision guide (after analyze/eval)
+
+| signal | diagnosis | action |
+|---|---|---|
+| eager e2e < 150 µs, ≤2 matchable cheap nodes | overhead-bound, ceiling < 1 | floor pass, move on |
+| big gap between eager e2e and Σ(node µs) | per-call fixed costs dominate | absorb more nodes per launch; nothing else helps |
+| one node ≥ 60% of eager time, matchable | real kernel target | fuse it + its neighbors; tune via passnet-triton-opt |
+| a `conv` dominates AND `stride == kernel_size`, `kernel_size > 1`, `padding == 0` (non-overlapping) | disjoint windows ⇒ exactly a dense patch matmul; cuDNN pays for a general im2col path | reformulate as a `tl.dot` patch-gather kernel + fused tail (kernel-templates §13), AFTER banking a tail floor. Don't dismiss as "cuDNN, leave alone" or "unmatchable" |
+| a `conv`/`matmul` dominates in any OTHER form (1×1, depthwise/grouped, overlapping conv, general dense matmul) | vendor sweet spot | default: leave it in aten; fuse its cheap tail (bias/BN/act/residual). Rewriting usually loses — try only if parameters look off-regime and a completed eval beats the floor |
+| one node ≥ 60%, callable-unmatchable kwargs form | try exact manual FX if the region is single-output and valuable; otherwise fuse what's left | expectation depends on whether the exact pattern pre-flight matches |
+| compiled gpu ≈ e2e, both < 1 | stream gaps (launch-bound) | fewer launches/allocs |
+| compiled gpu > 1, e2e < 1 | host overhead | drop autotune churn, simplify wrapper, fewer passes |
+| some dtype variants fail only | numeric fidelity | passnet-triton-opt §3 recipes |
+| timeout (600 s) | too many variants × compile/tune cost | remove autotune, single config, fewer passes |
+| evaluator reports timing/environment fluctuation while successful variants match and pass correctness | evaluator instability, not proven kernel bug | rerun unchanged if budget remains; record separately from numeric/no-match/unauthorized |