automation script to pull models.yml#9635
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
This PR is large and would use a significant portion of your monthly review quota. Comment |
Bundle ReportChanges will increase total bundle size by 17.01kB (0.07%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: marimo-esmAssets Changed:
Files in
|
Contributor
|
This PR is large and would use a significant portion of your monthly review quota. Comment |
Contributor
|
This PR is large and would use a significant portion of your monthly review quota. Comment |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a manual sync workflow for packages/llm-info/data/models.yml from the public models.dev catalog, restructures the model catalog to be provider-keyed, and updates codegen + frontend consumption to match the new schema (capabilities, modalities, pricing, release dates).
Changes:
- Add a
pnpm sync-modelsscript + implementation to fetchmodels.dev/api.jsonand append/replace entries inmodels.ymlwhile preserving curated entries/comments. - Change the
llm-infomodel schema and data layout to a top-level provider map, enriching entries with capabilities, modalities, release dates, and cost. - Update codegen/tests and frontend model registry + UI “thinking” indicator to use the new structure.
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/llm-info/src/sync-models.ts | CLI entrypoint + YAML-preserving append/replace writer for models.yml. |
| packages/llm-info/src/cli.ts | Parses sync-models CLI flags (mode, providers filter, per-provider cap). |
| packages/llm-info/src/sources/models-dev.ts | Fetches/parses models.dev API response with Zod validation and warnings. |
| packages/llm-info/src/sources/merge.ts | Merges models.dev data into local provider buckets with trimming/sorting. |
| packages/llm-info/src/index.ts | Updates exported types (capabilities, modalities, provider-keyed structure). |
| packages/llm-info/src/generate.ts | Updates codegen validation + JSON structure to provider-keyed models. |
| packages/llm-info/src/tests/sync-models.test.ts | Adds comprehensive tests for merge + sync behavior and YAML formatting preservation. |
| packages/llm-info/src/tests/schema.test.ts | Updates schema tests for new model entry shape + provider-keyed YAML. |
| packages/llm-info/src/tests/json-structure.test.ts | Updates JSON shape expectations (models is a provider-keyed map). |
| packages/llm-info/data/models.yml | Converts catalog to provider-keyed sections and adds enriched fields. |
| packages/llm-info/package.json | Adds sync-models script. |
| packages/llm-info/README.md | Documents pnpm sync-models usage examples. |
| packages/llm-info/skills/SKILL.md | Adds a documented workflow for backfilling empty descriptions (Cursor skill). |
| .cursor/skills/fill-model-descriptions/SKILL.md | Same skill doc mirrored under .cursor/skills. |
| frontend/src/core/ai/model-registry.ts | Updates frontend registry to consume provider-keyed models + rehydrate dates + provider field. |
| frontend/src/core/ai/tests/model-registry.test.ts | Updates mocks/expectations for provider-keyed models and provider-owned entries. |
| frontend/src/components/app-config/ai-config.tsx | Switches “thinking” badge to capabilities.includes("thinking"). |
| frontend/src/components/ai/ai-model-dropdown.tsx | Switches “thinking” indicators to capabilities.includes("thinking"). |
| frontend/src/components/ai/tests/ai-utils.test.ts | Updates models.json mock to provider-keyed format + new fields. |
![cubic-dev-ai[bot]](https://cdn.statically.io/img/avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
9 issues found across 19 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/llm-info/src/cli.ts">
<violation number="1" location="packages/llm-info/src/cli.ts:35">
P2: `--mode replace` is ignored. Use `getFlag()` here so the spaced form does not fall back to append mode.</violation>
<violation number="2" location="packages/llm-info/src/cli.ts:73">
P1: Empty provider list should fail. In `--replace` mode, a typo can rewrite `models.yml` from an empty result set.</violation>
</file>
<file name="packages/llm-info/src/sync-models.ts">
<violation number="1" location="packages/llm-info/src/sync-models.ts:237">
P2: Cap is not per final provider. `google` can get trimmed entries from both `google` and `google-vertex`, so `-n 5` can still append 10 models. Trim after merging into one bucket.</violation>
</file>
Architecture diagram
sequenceDiagram
participant CLI as Sync Script (cli.ts)
participant Sync as sync-models.ts
participant Merge as sources/merge.ts
participant ModelsDev as sources/models-dev.ts
participant API as models.dev API
participant YAML as data/models.yml
participant Codegen as generate.ts
participant JSON as data/generated/models.json
participant Frontend as Frontend Components
participant Registry as model-registry.ts
Note over CLI,Registry: Model Sync and Consumption Flow
CLI->>Sync: pnpm sync-models [--replace] [-n 10] [-p openai,google]
Sync->>ModelsDev: fetchModelsDev()
ModelsDev->>API: GET https://models.dev/api.json
API-->>ModelsDev: JSON response
ModelsDev->>ModelsDev: parseModelsDev() – validate with Zod schema
ModelsDev-->>Sync: ModelsDevApi object
Sync->>YAML: readFileSync(models.yml)
YAML-->>Sync: YAML text
Sync->>Sync: parseExistingModels() – extract provider-model pairs
Sync->>Merge: mergeModels(existing, modelsDev, options)
Merge->>Merge: For each provider in PROVIDER_MAP:
Merge->>Merge: - Build AiModel entries from API data
Merge->>Merge: - Derive roles, capabilities, cost, modalities
Merge->>Merge: - Sort newest-first, cap at maxPerProvider
Merge->>Merge: - Skip models that already exist locally
Merge-->>Sync: MergeSummary (newEntries, preservedCount)
alt mode === "append"
Sync->>Sync: appendIntoDocument() – append new entries to existing YAML
else mode === "replace"
Sync->>Sync: renderFresh() – generate entirely new YAML
end
Sync->>YAML: writeFileSync(models.yml)
Note over Codegen: Codegen runs separately
Codegen->>YAML: readFileSync + parse
Codegen->>Codegen: Validate with ModelsByProviderSchema
Codegen->>JSON: writeFileSync(models.json)
Note over Frontend: Runtime consumption
Frontend->>JSON: import models.json
JSON-->>Frontend: { models: { providerId: AiModel[] } }
Frontend->>Registry: getKnownModelMaps()
Registry->>Registry: Flatten per-provider arrays into single Map
Registry->>Registry: Each model gets its own provider field
Registry-->>Frontend: QualifiedModelId → AiModel
alt User opens model dropdown
Frontend->>Registry: getModelsByProvider(provider)
Registry-->>Frontend: AiModel[]
Frontend->>Frontend: Check model.capabilities.includes("thinking")
end
Note over Registry: Key schema changes
Note over Registry: providers: string[] → provider: string (single)
Note over Registry: thinking: boolean → capabilities: string[]
Note over Registry: New fields: input_types, output_types, release_date, cost
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Contributor
There was a problem hiding this comment.
1 issue found across 9 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/llm-info/src/sync-models.ts">
<violation number="1" location="packages/llm-info/src/sync-models.ts:237">
P2: Cap is not per final provider. `google` can get trimmed entries from both `google` and `google-vertex`, so `-n 5` can still append 10 models. Trim after merging into one bucket.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Contributor
There was a problem hiding this comment.
1 issue found across 4 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
Light2Dark
force-pushed
the
sham/automate-models-yml-scrape
branch
from
b199bb0 to
21e909d
Compare
## 📝 Summary <!-- If this PR closes any issues, list them here by number (e.g., Closes #123). Detail the specific changes made in this pull request. Explain the problem addressed and how it was resolved. If applicable, provide before and after comparisons, screenshots, or any relevant details to help reviewers understand the changes easily. --> Closes # ## 📋 Pre-Review Checklist <!-- These checks need to be completed before a PR is reviewed --> - [ ] For large changes, or changes that affect the public API: this change was discussed or approved through an issue, on [Discord](https://marimo.io/discord?ref=pr), or the community [discussions](https://github.com/marimo-team/marimo/discussions) (Please provide a link if applicable). - [ ] Any AI generated code has been reviewed line-by-line by the human PR author, who stands by it. - [ ] Video or media evidence is provided for any visual changes (optional). <!-- PR is more likely to be merged if evidence is provided for changes made --> ## ✅ Merge Checklist - [ ] I have read the [contributor guidelines](https://github.com/marimo-team/marimo/blob/main/CONTRIBUTING.md). - [ ] Documentation has been updated where applicable, including docstrings for API changes. - [ ] Tests have been added for the changes made. Co-authored-by: Cursor <cursoragent@cursor.com>
|
|
||
| export type ModelsDevApi = Record<string, ModelsDevProvider>; | ||
|
|
||
| const DEFAULT_URL = "https://models.dev/api.json"; |
Member
There was a problem hiding this comment.
most providers have their own urls which might have most up to date info. It might be worth it just maintain those urls instead.
Member
There was a problem hiding this comment.
Also how much work would it be to fetch this live? i.e instead of reading from models.json, make this query
Member
Author
There was a problem hiding this comment.
Unfortunately, it doesn't have the full information we need like cost and capability. Different providers give a different structure. So, this is more unified.
Member
Author
There was a problem hiding this comment.
kirangadhave
approved these changes
May 29, 2026
kirangadhave
left a comment
kirangadhave
left a comment
Member
There was a problem hiding this comment.
🚀 This is good. Should we have a job that run this periodically and opens a PR? Say weekly or monthly?
Member
Author
Yeah, I can follow-up on that. Thanks! |
Contributor
|
🚀 Development release published. You may be able to view the changes at https://marimo.app?v=0.23.9-dev22 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📝 Summary
Pulling this data from https://models.dev. It has an open-source API. Considered openrouter as well, but it has a slightly different structure. Anyway, easy to changeover if needed.
Some models don't exist on the API, maybe we can cross-check. I've removed them manually for now (eg.
gpt-5.5-codex-spark)We could put this into a github actions workflow in the future.
📋 Pre-Review Checklist
✅ Merge Checklist