APP-560: Dag viewer

[eucliss]

Add Resource Graph (DAG) Visualization System

This PR introduces the Resource Graph system for visualizing project dependencies across Sources → Models → Metrics Views → Dashboards. The system is built on SvelteFlow with Dagre layout and supports deep-linking, grouping, overlays, and multiple embedding modes.

Key Additions

• ResourceGraphOverlay
  Branded modal for “View Dependencies” from any resource. Handles anchor resource, loading, and errors.
• ResourceGraph
  Full graph component with seed filtering, grouping, and optional URL sync via expanded query params.
• ResourceGraphCanvas
  Low-level visualization wrapper around SvelteFlow. Exposes layout primitives and flow control.
• GraphOverlay
  Generic expansion component supporting inline, modal, and fullscreen modes for embedded graphs.
• URL Deep-Linking
  /graph?seed=<kind>:<name> loads targeted graphs. Supports multiple seeds, kind aliases, and kind expansion.

Functionality Overview

• Launch dependency graphs directly from any resource menu with minimal integration.
• Render graphs inline in dashboards, widgets, or sidebars with or without URL synchronization.
• Use partitionResourcesBySeeds to generate multiple independent graph groups on a single page.
• Graphs support panning, zooming, grouping by resource kind, and configurable UI elements (showSummary, maxGroups, etc.).
• Hidden resources are automatically filtered out; only supported resource types appear.

Integrations Included

• Source, Model, and Metrics View navigation menus now include “View Dependencies.”
• Standardized props across ResourceGraph, ResourceGraphCanvas, and ResourceGraphOverlay.
• Added examples and JSDoc within components for implementation clarity.

[ericpgreen2]

File Organization Feedback

First off, this is excellent work! The graph visualization is well-architected with solid algorithms, comprehensive tests, and thoughtful error handling. However, I'd like to discuss the file organization before merging.

Current Structure

The resource-graph/ directory currently has 21 files in a flat structure:

resource-graph/
├── build-resource-graph.ts
├── cache-manager.ts
├── graph-config.ts
├── graph-error-handling.ts
├── graph-traversal.ts
├── GraphErrorBoundary.svelte
├── GraphOverlay.svelte
├── navigation-utils.ts
├── resource-graph-quick-view-store.ts
├── resource-id.ts
├── ResourceGraph.svelte
├── ResourceGraphCanvas.svelte
├── ResourceGraphContainer.svelte
├── ResourceGraphOverlay.svelte
├── ResourceGraphQuickView.svelte
├── ResourceNode.svelte
├── seed-utils.ts
├── SummaryCountNode.svelte
├── SummaryCountsGraph.svelte
├── types.ts
└── use-graph-url-sync.ts

Inconsistency with Codebase Patterns

This is inconsistent with how we organize other features. For comparison:

features/models/: Only 3 files at root, organized into subdirectories:

models/
├── error-utils.ts
├── index.ts
├── selectors.ts
├── incremental/
├── inspector/
├── navigation/
├── partitions/
├── utils/
└── workspace/

features/chat/: Organized by semantic features:

chat/
├── core/
│   ├── messages/
│   ├── input/
│   └── [core logic]
└── layouts/
    └── sidebar/

features/dashboards/: Organized with clear subdirectories:

dashboards/
├── config.ts
├── pivot/
├── stores/
├── time-controls/
├── filters/
└── dimension-table/

Proposed Reorganization

I recommend organizing by semantic sub-features (user flows) rather than file types:

resource-graph/
├── quick-view/                 # "View Dependencies" menu action
│   ├── QuickView.svelte
│   ├── quick-view-store.ts
│   └── README.md
│
├── graph-canvas/               # Core visualization engine
│   ├── GraphCanvas.svelte
│   ├── ResourceNode.svelte
│   ├── graph-builder.ts
│   ├── graph-builder.spec.ts
│   └── README.md
│
├── navigation/                 # URL deep-linking (/graph?seed=...)
│   ├── GraphContainer.svelte
│   ├── url-sync.ts
│   ├── url-sync.spec.ts
│   ├── seed-parser.ts
│   ├── seed-parser.spec.ts
│   └── README.md
│
├── embedding/                  # Embeddable graph widgets
│   ├── ResourceGraph.svelte
│   ├── GraphOverlay.svelte
│   ├── ResourceGraphOverlay.svelte
│   └── README.md
│
├── summary/                    # Simplified aggregate views
│   ├── SummaryGraph.svelte
│   ├── SummaryNode.svelte
│   └── README.md
│
└── shared/                     # Cross-cutting concerns
    ├── cache/
    │   └── position-cache.ts
    ├── traversal/
    │   ├── graph-traversal.ts
    │   └── graph-traversal.spec.ts
    ├── partitioning/
    │   ├── partition-by-seeds.ts
    │   └── partition-by-metrics.ts
    ├── errors/
    │   ├── error-handling.ts
    │   └── ErrorBoundary.svelte
    └── config.ts

Result: 3-6 files per directory (vs. 21 in a flat list)

Benefits

1. Consistency: Matches patterns in models/, chat/, dashboards/
2. Discoverability: Directory names explain purpose ("What does this do?")
3. Co-location: Everything for a feature lives together
4. Documentation: README per sub-feature vs. one giant doc
5. Maintainability: Clear ownership boundaries

Additional Cleanup

I also noticed resource-id.ts (298 lines) appears to be an unused alternative implementation. The codebase uses the simpler resource-utils.ts functions instead. Consider removing:

• resource-id.ts
• resource-id.spec.ts

Thoughts?

[ericpgreen2]

Miller, overall this looks good! I did a first pass through the code, and the first thing I noticed was that the resource-graph directory could be better structured. As a rough heuristic, we should try to avoid frontend feature directories that have more than ~8 top-level files. See the comment above (which I generated via a discussion with Claude Code) for more detail.

[ericpgreen2]

URL API Feedback: Ambiguity & Terminology

Developed in collaboration with Claude Code

I've reviewed the URL API design and identified two issues that should be addressed before merging:

Problem 1: Ambiguity with Kind Token Expansion

The current implementation checks if a bare seed matches a kind token ("metrics", "sources", "models", "dashboards") before treating it as a resource name. This creates an unavoidable collision if users name their resources with these common terms.

Example scenario:

# metrics.yaml - a MetricsView literally named "metrics"
type: metricsview
table: orders

Current behavior:

• ?seed=metrics → Expands to ALL MetricsView resources (not the one named "metrics")
• ?seed=metricsview:metrics → Works, but non-intuitive workaround
• No clean way to link to the specific resource named "metrics"

The same issue affects any resource named sources, models, or dashboards - all plausible names for metrics views (e.g., a "sources" dashboard showing source health, or a "models" metrics view tracking model performance).

Code reference: web-common/src/features/resource-graph/navigation/seed-parser.ts:217-242

Problem 2: "Seed" is Developer Jargon

The term "seed" is computer science terminology (graph seeding, traversal start points) that won't be intuitive to business users who see these URLs in their browser or shared links.

Current URLs:

• ?seed=orders - What's a "seed"?
• ?seed=metrics - Is this seeding metrics?

User-facing touchpoints:

• Shared URLs between team members
• Browser address bar when clicking "View Dependencies"
• Bookmarks and documentation
• Error messages referencing "invalid seed"

Proposed Solution

Replace the seed parameter with two explicit parameters that eliminate ambiguity:

/graph                          # All graphs (default overview)
/graph?kind=metrics             # All MetricsView graphs
/graph?kind=models              # All Model graphs
/graph?kind=sources             # All Source graphs
/graph?resource=orders          # Specific graph for "orders" resource

Benefits:

1. ✅ No ambiguity - kind and resource are mutually exclusive, semantically distinct parameters
2. ✅ Self-documenting - URLs clearly express intent (kind= for filtering, resource= for specific)
3. ✅ User-friendly - Non-technical terminology that business users understand
4. ✅ Prevents collisions - Resources can be named anything, including "metrics" or "sources"
5. ✅ Simple validation - Can't specify both kind and resource (error state is clear)

Implementation notes:

• Update seed-parser.ts to read from new parameters
• Add validation to reject ?kind=X&resource=Y (mutually exclusive)
• Update URL builders in navigation components
• Minimal changes to core graph logic (just parameter parsing)

Migration path:
Since this is a new feature (PR #8326), there are no existing URLs to migrate. This is the ideal time to get the API right.

[ericpgreen2]

localStorage Key Convention

Developed in collaboration with Claude Code

The current implementation uses a versioned key with dot notation:

CACHE_NAMESPACE = `rill.resourceGraph.v${CACHE_VERSION}` // "rill.resourceGraph.v2"

Issues with Current Approach

1. Unnecessary versioning: The version is meant to invalidate cached positions when layout algorithms change, but there's no compelling reason users would want to revert to an old version. If layout issues occur, users can manually clear the cache via the exposed debug utilities (window.__RESOURCE_GRAPH_CACHE.clearAll()).

2. Inconsistent with existing patterns: The codebase lacks a consistent localStorage convention, but the most similar namespaced key is rill:theme which uses colon separators.

3. Dot notation + camelCase: Uses Java-style dot notation and camelCase, while most other keys use kebab-case (e.g., sidebar-width, connector-explorer-state).

Proposed Convention

Use a simple namespaced key without versioning:

CACHE_NAMESPACE = "rill:resource-graph"

Benefits:

• ✅ Simpler: No version management overhead
• ✅ Consistent: Matches rill:theme pattern (namespaced with colon)
• ✅ Kebab-case: Follows majority pattern in codebase
• ✅ Namespace protection: Essential since Rill Developer runs on localhost:9009, which users may reuse for other projects

Why the rill: prefix matters:

Since Rill Developer runs on localhost:9009, the localStorage namespace is shared with:

• Other projects users run on the same port
• Browser extensions and developer tools
• Multiple Rill projects over time

Without the prefix, keys like resource-graph could collide with other applications. The rill: prefix provides clear namespacing and prevents conflicts.

Versioning rationale:

If cache invalidation is needed due to layout changes, users have clear paths:

• Debug tool: window.__RESOURCE_GRAPH_CACHE.clearAll()
• Settings UI: Could add "Clear Graph Cache" button if needed
• Automatic migration is unnecessary for a feature where stale cache causes minor visual inconsistency, not data corruption

Establishing a Convention

While the codebase currently lacks a consistent localStorage key convention, this PR is an opportunity to establish one. I propose:

Standard pattern for new features:

rill:{feature-name}

Examples:

• rill:resource-graph (this feature)
• rill:theme (existing)
• Future features: rill:user-preferences, rill:workspace-layout, etc.

This creates clear namespacing under the rill: prefix while using human-readable kebab-case names.

[ericpgreen2]

Thanks for the code re-org. Looks much better!

Now:

1. See the two comments above about the URL and LocalStorage APIs.
2. See my UXQA findings from playing with the feature.

UXQA:

1. I don't see the "View dependency graph" on the menu items for Explores or Canvases. Can we add it?
2. Can we remove the animation that kicks-in when a DAG is first rendered? IMO it's distracting.
3. When I click on each node of the master DAG, I'd expect it to filter the below individual DAGs to match the number presented in the master node. So, in the picture below, because there are 0 sources in my project, I wouldn't expect to see any sub-DAGs.
   
4. When I "expand" a DAG on the master page, I'd expect it to be more of a "selection" and it'd be the only DAG I see on the screen. Here I've expanded the "Adbids" DAG and it spills out of my viewport.
   

[ericpgreen2]

Given this is empty, can we delete?

[ericpgreen2]

Minor: Unused Cache Field

Developed in collaboration with Claude Code

The cache includes a refs field that appears to be written but never read:

Writing refs:

// web-common/src/features/resource-graph/graph-canvas/graph-builder.ts:252
graphCache.addRef(dependentId, sourceId);

Reading refs:
I don't see any graphCache.getRefs() calls in the codebase.

Recommendation:

Since the cache is persisted to localStorage and counts toward the 4MB quota, consider either:

1. Remove it if it's vestigial from an earlier iteration
2. Document its purpose if it's planned for future features (e.g., historical dependency tracking)
3. Use it if there's an intended use case

This keeps the cache lean and prevents storing unused data in users' browsers.

[ericpgreen2]

Code is looking great!

From my end, here are a couple last UXQA items:

1. There still seems to be something wrong with the kind filtering. In the project-level DAG, I see 3 metrics views, but when I filter for metrics view DAGs, I see 6.
   
2. I'm still not a fan of how the "expand" action works. When I click it, the target DAG still spills off-screen. IMO, rather than considering it an expansion, I'd consider it a selection where the URL is /graph?name=<selected-resource> and the entire workspace is dedicated to the DAG (i.e. we don't additionally show the project-level DAG).
   

I've also tagged @Di7design for a final review from the Product team.

[Di7design]

UXQA from Design:

1. Functionally, the main thing I noticed is the order in the DAG overview. I think it should be: resources → models → metrics → dashboards.

2. As noted in Eric G’s second point, keeping the DAG overview at the top causes an expanded chart to display only partially and hide the remaining DAGs. After syncing with Eric, we propose using the same overlay pattern for opening a single chart in the project graph page instead of expanding the window.

3. Clicking a box and having the entire chart highlight is confusing. Can we highlight only the clicked box instead of the entire chart? Also, the link icon for opening the source is hard to notice especially when the whole chart is highlighted, we can use an icon+text button and add “open” next to the icon to make the action clearer.

[Di7design]

Functionally looks good to me! Just a small bug : the close button on popup modal window is not working when I open it from side bar - dropdown menu - view dependency graph.

[ericpgreen2]

Looks great. Let's merge!