This project contains CI packages used for the transformers project.

transformersci.otel, contains the OpenTelemetry support for pytest tests, that can be used from the main transformers repository.

The main pieces are:

• transformersci.otel.cli: configures OTEL env vars and launches pytest
• transformersci.otel.resource_plugin: optional per-test CPU, RSS, and CUDA memory sampling
• transformersci.otel.trace_exporter: converts recent Tempo traces into Prometheus metrics

Installing the package exposes:

• configure-ci-otel
• pytest-trace-exporter

Example:

configure-ci-otel \
  --job local_smoke \
  --service-name transformers-tests \
  --protocol grpc \
  --otlp-endpoint http://localhost:5317 \
  -- python3 -m pytest tests/test_cli.py -q

The resource plugin is registered through pytest11 and stays inert unless resource collection is explicitly enabled with one of:

• --resource-metrics-file <path>
• PYTEST_RESOURCE_METRICS_FILE=<path>
• TRANSFORMERS_TEST_RESOURCE_METRICS_FILE=<path>

Example:

configure-ci-otel \
  --job resource_demo \
  --service-name pytest-observability-demo \
  --protocol grpc \
  --otlp-endpoint http://localhost:5317 \
  -- \
  python3 -m pytest tests/test_demo_workload.py -q \
  --resource-metrics-file dashboard/data/pytest-resource-metrics.jsonl

To use transformers-ci in a GitHub Actions workflow:

1. Add repository secrets:

   • OTEL_EXPORTER_OTLP_ENDPOINT - The OTLP endpoint URL (e.g., https://transformers-ci-traces.lor-e.huggingface.cool)
   • OTEL_EXPORTER_OTLP_TOKEN - Raw bearer token, without the Authorization=Bearer prefix

   configure-ci-otel can now set OTEL_TRACES_EXPORTER, OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_EXPORTER_OTLP_ENDPOINT, and OTEL_EXPORTER_OTLP_HEADERS for you from CLI flags. Use the traces collector host here, not the Grafana UI host: transformers-ci-traces.lor-e.huggingface.cool, not transformers-ci.lor-e.huggingface.cool.

2. Update your workflow:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install transformers-ci
        run: pip install transformers-ci[otel]

      - name: Run tests with OpenTelemetry tracing
        run: >-
          configure-ci-otel
          --service-name transformers-tests
          --protocol http
          --otlp-endpoint "${OTEL_EXPORTER_OTLP_ENDPOINT}"
          --token "${OTEL_EXPORTER_OTLP_TOKEN}"
          -- pytest tests/ -v
        env:
          OTEL_EXPORTER_OTLP_ENDPOINT: ${{ secrets.OTEL_EXPORTER_OTLP_ENDPOINT }}
          OTEL_EXPORTER_OTLP_TOKEN: ${{ secrets.OTEL_EXPORTER_OTLP_TOKEN }}

To send every span to a second (staging) backend in addition to the primary, add --staging-endpoint (and --staging-token if staging uses its own auth). This attaches a second span processor inside the pytest run, so each span is exported to both backends; a flaky staging box does not affect the primary export. Staging reuses the primary --protocol unless you override it with --staging-protocol (e.g. prod over http but the stage box only speaks grpc).

      - name: Run tests with OpenTelemetry tracing (mirrored to staging)
        run: >-
          configure-ci-otel
          --service-name transformers-tests
          --protocol http
          --otlp-endpoint "${OTEL_EXPORTER_OTLP_ENDPOINT}"
          --token "${OTEL_EXPORTER_OTLP_TOKEN}"
          --staging-endpoint "${OTEL_STAGING_OTLP_ENDPOINT}"
          --staging-protocol grpc
          --staging-token "${OTEL_STAGING_OTLP_TOKEN}"
          -- pytest tests/ -v
        env:
          OTEL_EXPORTER_OTLP_ENDPOINT: ${{ secrets.OTEL_EXPORTER_OTLP_ENDPOINT }}
          OTEL_EXPORTER_OTLP_TOKEN: ${{ secrets.OTEL_EXPORTER_OTLP_TOKEN }}
          OTEL_STAGING_OTLP_ENDPOINT: ${{ secrets.OTEL_STAGING_OTLP_ENDPOINT }}
          OTEL_STAGING_OTLP_TOKEN: ${{ secrets.OTEL_STAGING_OTLP_TOKEN }}

The endpoint/token/protocol can also come from the environment instead of flags via TRANSFORMERS_TEST_OTEL_STAGING_ENDPOINT, TRANSFORMERS_TEST_OTEL_STAGING_TOKEN, and TRANSFORMERS_TEST_OTEL_STAGING_PROTOCOL. If --staging-token is omitted, staging falls back to the primary token; if --staging-protocol is omitted, it falls back to the primary protocol.

To test the instrumentation locally without sending traces to a remote endpoint:

# Install with dev dependencies
pip install -e ".[dev]"

# Run tests with local resource metrics collection (no OTLP endpoint needed)
configure-ci-otel --force-export-traces --job local_test -- \
  pytest tests/test_demo_workload.py -v \
  --resource-metrics-file /tmp/pytest-metrics.jsonl

This writes per-test CPU, RSS, and CUDA metrics to a local JSONL file without requiring an OTLP endpoint.

An initial Grafana dashboard implementation lives under dashboard/.