Skip to content

feat: LLM Cost Tracking and Budgeting feature in passmark#41

Open
devhimanshuu wants to merge 1 commit into
bug0inc:mainfrom
devhimanshuu:features
Open

feat: LLM Cost Tracking and Budgeting feature in passmark#41
devhimanshuu wants to merge 1 commit into
bug0inc:mainfrom
devhimanshuu:features

Conversation

@devhimanshuu

Copy link
Copy Markdown

🌟 Summary

Introduces an enterprise-grade LLM Cost Tracking & Budgeting system. This prevents "bill shock" by strictly monitoring API costs across all parallel test workers and halting execution if a user-defined budget is exceeded.

❓ Why is this needed?

AI-driven tests can accumulate high API costs rapidly, especially during loops or parallel test execution in CI/CD. Teams need strict, observable cost controls to confidently run Passmark in production pipelines without fear of runaway expenses.

🛠️ Key Features

  • Distributed Budget Enforcer: Added maxCostPerRun to configure(). If the budget is breached, it safely aborts the suite with a BudgetExceededError.
  • Global Worker Sync (Redis): Uses Redis (INCRBYFLOAT) with a 24-hour TTL to precisely aggregate costs across all parallel Playwright workers, ensuring true global limits.
  • Custom Pricing Overrides: Added a pricing config option to define custom token rates (useful for enterprise gateways or self-hosted models).
  • OpenTelemetry Observability: Automatically attaches token counts and cost metrics (ai.usage.cost) to active OpenTelemetry traces for rich Datadog/Axiom dashboarding.
  • CI/CD Artifacts: Generates a passmark-cost-<pid>.json summary in the uploads directory and logs the total cost upon process exit.
  • Universal Tracking: Intercepts Vercel AI SDK usage and raw OpenAI Responses API (CUA mode) calls universally.

💻 Usage Example

import { configure } from "passmark";

configure({
  maxCostPerRun: 2.00, // Halts the test suite if AI cost exceeds $2.00 globally
  pricing: {
    // Optional: override default pricing
    "my-internal-model": { input: 0.10, output: 0.20 }
  }
});
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant