Cost management and billing

Adaptive Telemetry

Adaptive Metrics

Configure segmentation

Grafana Cloud

Configure segmentation

Segmentation allows you to operate Adaptive Metrics on a per team, service, department, or system basis. Each segment receives its own recommendations; recommendations, rules, and exemptions can then all be managed on a per-segment basis. This team-centric approach allows you to manage Adaptive Metrics considering each team’s specific data and requirements.

By default all recommendations, custom rules, and exemptions are applied to all metrics sent to Adaptive Metrics regardless of which labels they include.

You can create and manage segments using the UI or configure and manage segments using the Segment API or Terraform.

Segments

Think of a segment as a container for a specific group of your metrics. You define this group based on a label and values, or selectors.

Each segment has a:

Name
Represents a human-readable name for this label values grouping.
Selector or label, and values:
Tells Adaptive Metrics which metrics belong to the segment.
In the UI, you use a label and values. In the API, you use selectors (each selector contains a label and values).
Note
- Only one label matcher is allowed. For example, ({team=“billing”}).
- All segments must reference the same label name. Creating a segment with a different label name results in an error.
- Only equality matchers, for example, ({team=“billing”}), or multi-literal regular expression matchers, for example, ({team=~(alerting|alerting-dev)}), are allowed.
Fallback to default
A setting that decides if the general, default rules should apply to metrics in this segment if no specific rule is found for them.
(API only) ID
A unique code automatically assigned when you create it. You use this to refer to the segment later.

Segmentation workflow

Define segments.
Using the UI, API, or Terraform, you create segments, give them descriptive names, and add labels and values. Grafana uses these to identify which metrics belong to this segment, effectively grouping metrics into separate, segment-specific containers.
Note
Multiple label values can be applied to a single selector, for example, {team=~"(alerting|alerting-dev|alerting-prod)"}, to enable managing segments on a per-team,service, department, or system basis.
Adaptive Metrics allows for a maximum of 50 segments. To increase the limit, contact Customer Support."
View recommendations and manage rules per segment.
Within the Grafana UI or via the API, when you view recommendations or manage aggregation, drop, and exemption rules, you can filter or scope the view to a specific segment. This allows each team to see and act upon recommendations that pertain only to their metrics.

Create segments using the UI

Create, update, and manage rule segments through the Grafana user interface. This provides you with a more visual way of defining segments and managing your rules. It is the recommended way for initial set up and smaller organizations and it is also easier to make ad-hoc adjustments.

To create a new segment, complete the following steps.

Navigate to Administration > Cost Management > Metrics Cost Management > Adaptive Metrics > Configuration.
Click Add new segment.
Enter a name.
Give your segment a descriptive name that clearly identifies the team it applies to.
Enter a label.
Specify the metric label that distinguishes the metrics belonging to a particular team. In the context of Kubernetes, this is often namespace.
Enter a value.
Define the criteria for matching the label you specified. This is where you tell Grafana which values of that label correspond to the current team’s metrics. In the UI, you can select values using a dropdown. Enter the exact name of the Kubernetes namespace(s) or label value(s) for the team (e.g., web-frontend, web-backend). If a team owns multiple namespaces, you can enter multiple values.
Note
Each value can only be used once. Therefore, once a value is added to a segment, it cannot be used in another.
(Optional): Select Fallback to default rules checkbox.
Select this checkbox to apply the general, default aggregation rules to any metrics that do not belong to a specific team segment. When unchecked (default), these unmatched metrics are ignored.
Click Add.

View recommendations per segment

Find and view the rule recommendations that Adaptive Metrics generates for each of your segments using the Adaptive Metrics UI.

Navigate to Administration > Cost Management > Metrics Cost Management > Adaptive Metrics > Rules.
Choose a segment from the Segment dropdown.
The recommendations for that segment are displayed in the list view. Apply all recommendations is restricted to the selected segment.
Note
If you choose the Default segment, every recommendation that is not assigned to a segment is displayed.

Create segments using Terraform

Manage your Adaptive Metrics rule segmentation as part of your infrastructure-as-code strategy using the grafana-adaptive-metrics Terraform provider.

This allows you to define and import your metric segments, ensuring a consistent, versioned, and maintainable setup for Adaptive Metrics within your Grafana environment.

For more information, refer to the Terraform Provider repository.

Prerequisites

Ensure that you have Terraform installed.
Ensure that the Grafana provider is configured within your Terraform project, including the necessary API keys or authentication details to interact with your Grafana Cloud instance.
If you want to take advantage of version control, ensure that you have a version control system where you can store the Adaptive Metrics configurations.

Steps

To create segments using Terraform, complete the following steps.

Organize your Adaptive Metrics configuration files following a clear structure. The Terraform provider comes with a resource type to define Adaptive Metrics segments called grafana-adaptive-metrics_segment.
Each team has a corresponding segment and therefore it is recommended to split segments into separate files using the following file structure.
- aggregation_rules.tf
- rules-mimir.json
- rules-loki.json
- rules.json
- segments.json
Define your segments in he segments.json file. Each segment should include:
- name: A user-friendly name for the segment (often the team name).
- selector: A PromQL-style label selector that matches the metrics belonging to this segment, for example, based on Kubernetes namespace.
- fallback_to_default: A boolean indicating whether to apply the default rules if no specific rule matches for this segment.
Example:
json
```
[
  {
    "fallback_to_default": false,
    "name": "mimir",
    "selector": "{namespace=~\"(adaptive-metrics|mimir-cluster-1|     mimir-cluster-2)\"}"
  },
  {
    "fallback_to_default": false,
    "name": "loki",
    "selector": "{namespace=~\"(loki-cluster-1|loki-cluster-2)\"}"
  }
]
```
Define fallback rules in the rules.json file.
This file contains the aggregation and drop rules that are applied to any metric series that do not match any of your defined segments, if fallback_to_default is set to true for a segment or if no segment matches.
Example:
json
```
[
  {
    "metric": "node_network_protocol_type",
    "drop_labels": ["agent_hostname", "instance"],
    "aggregations": ["sum:counter"]
  }
]
```
Define team-specific rules in rules-\*.json files.
Create separate JSON files for each team or segment. These files contain the aggregation and drop rules that should only be applied to the metrics belonging to that specific segment.
Example:
json
```
[
  {
    "metric": "activity_tracker_free_slots",
    "drop_labels": ["instance", "pod"],
    "aggregations": ["sum:counter"]
  }
]
```

Create your Terraform configuration in the aggregation_rules.tf file.

This file contains the Terraform code to:

Read the contents of your JSON files.
Define the grafana-adaptive-metrics_segment resources based on segments.json.
Define the default grafana-adaptive-metrics_ruleset resource using rules.json.
Define grafana-adaptive-metrics_ruleset resources for each team-specific rule file, linking them to the corresponding grafana-adaptive-metrics_segment.

Example:

locals {
 rules_json = jsondecode(file("${path.module}/rules.json"))
 all_rules_files = fileset("${path.module}", "rules-*.json")
 segments = jsondecode(file("${path.module}/segments.json"))


 segmented_rules_json = {
   for file in local.all_rules_files :
   regex("rules-(.+).json", file)[0] => jsondecode(file("${path.module}/${file}"))
 }
}


resource "grafana-adaptive-metrics_ruleset" "all_rules" {
 rules = local.rules_json
}


resource "grafana-adaptive-metrics_segment" "all_segments" {
 for_each = { for segment in module.common.segments : segment.name => segment }


 name                = each.value.name
 selector            = each.value.selector
 fallback_to_default = each.value.fallback_to_default
}


resource "grafana-adaptive-metrics_ruleset" "segmented_rules" {
 for_each = local.segmented_rules_json


 segment = grafana-adaptive-metrics_segment.all_segments[each.key].id
 rules   = each.value
}

Apply the Terraform configuration.

Create segments using Segment API

Create rule segments using the Segment API. This is useful for automation, version control and scalability for managing a large number of teams and integrating with existing as-code practices.

If you have already defined a set of aggregation rules and recommendation exemptions that apply to all teams in your organization, identify which exemptions correspond to each team so that you can later assign these exemptions to the corresponding segment.

To simplify this process, consider copying the set of exemption rules defined in the default segment to all teams and allowing each team to independently remove or edit any rules that do not apply.

To apply the rules from the default segment if no matching rule applies to an incoming series, set the fallback_to_default setting to true.

To configure segments using the Segment API, complete the following steps.

Create segments by sending a POST request to the following endpoint: /aggregations/rules/segments with the following scope: adaptive-metrics-segments:write.
Example payload:
json
```
{
  "name": "Staging",
  "selector": "{env=\"staging\"}",
  "fallback_to_default": false
}
```
Note
After you create a segment, all of the endpoints available for listing and editing rules, including recommendations and exemptions, allow the optional sending of a URL query parameter named segment.
This parameter specifies the particular rule segment to perform an operation on. If this parameter is empty, the API uses the default segment.
After creating each segment, define the corresponding set of exemptions for the recommendations service.
After the recommendations service runs, apply the generated recommendations one-by-one, and then set the fallback_to_default setting to false to ignore the default segment.
As a best practice, migrate segments with a smaller total volume of series first, and then gradually proceed with the teams that have a larger volume of series. Additionally, to avoid errors, consider automating this process as much as possible.

Update or delete segments

Update or delete segments that you have created.

Update existing segments by sending a PUT request to the following endpoint /aggregations/rules/segments?segment=<segment ID> with the following scope: adaptive-metrics-segments:write.
Example payload:
json
```
{
  "name": "Staging",
  "selector": "{env=\"staging\"}",
  "fallback_to_default": false
}
```
Delete segments by sending a DELETE request to the following endpoint /aggregations/rules/segments?segment=<segment ID> with the following scope: adaptive-metrics-segments:delete.
You specify the segment to delete in the request body.
Before you can delete a segment, you must first remove all the aggregation rules and exemptions that are associated with it.

View segments and rules

View either a list of all the segments you have created or a list of all the aggregation rules that are currently applied, organized by which segment they belong to.

/aggregations/rules/segments: Shows you the segments you have created.
/aggregations/segmented_rules: Shows you the contents (aggregation rules) inside those containers (segments), along with the default rules.
/recommendations/segmented_exemptions: Shows the exemption rules distributed by segment.

View existing segments by sending a GET request to the following endpoint /aggregations/rules/segments with the following scope: adaptive-metrics-segments:read.
This retrieves a list of all rule segments configured by the user and returns an array of segments.
Example response:
json
```
[
  {
    "id": "01J35VCQXJHNF68C3JV3C91T0G",
    "name": "Development",
    "selector": "{env=\"dev\"}",
    "fallback_to_default": true
  }
]
```
View a list of segmented aggregation rules.
List all the aggregation rules that are currently applied, organized by which segment they belong to by sending a GET request to the following endpoint /aggregations/segmented_rules.
Example response:
json
```
[
  {
   "segment": {..},
   "rules": [...]
  },
  {
   "segment": {...},
   "rules": [...]
  },
  {
   "segment": {...},
   "rules": [...]
  },
  {
   "segment": {"name": "default"},
   "rules": [...]
  }
]
```
View a list of segmented exemptions.
List the exemption rules that you have configured for each of your defined segments by sending a GET request to the following endpoint /v1/recommendations/segmented_exemptions.
Example response:
json
```
[
  {
   "segment": {...},
   "exemptions": [...]
  },
  {
   "segment": {"name": "default" },
   "exemptions": [...]
  }
]
```

Best practices

Follow these best practices for segmenting rules in the Segment API.

Choose an appropriate label

There is a limit of 50 segments per instance. The same label must be used for all segments, and multiple label values can be chosen per segment. Select a label that is used to attribute by team, system, or organization, and take advantage of the ability to use multiple values per segment to make the feature easily manageable.

Define one segment per team

While it’s possible to have more than one segment for the same team, this approach is not recommended. Those segments would need to replicate the same number of aggregation rules and exemptions, adding an unnecessary layer of complexity by requiring these sets of rules to remain in sync.

If a single team has more than one label value assigned, define a single selector using a multi-literal regular expression. For example, {namespace=~"(team_dev|team_staging|team)"}.

Migrate one segment at a time

Incorrectly defining segments during migration can affect the ratio of aggregated series. As a result, it’s recommended to migrate one segment at a time and to check during the process that the percentage of total series versus aggregated series doesn’t change.

Begin with the segments that have the least potential for negative impact. Then, refine the process before proceeding with the segments that have a higher volume of time series.

It’s also recommended use an automation tool to minimize risks during the migration.

Regularly update segment definitions

If a new label value is assigned to a team, update its associated segment as soon as possible. Otherwise, the default segment is applied to time series with that label value. It’s also recommended to use an automation tool for this process.

Example

The following example defines a segment for the machine learning team. In this example, migration is not required.

You can identify the machine learning team’s time series using the label namespace. There are three possible values associated with the team, ml-dev, ml-staging, and ml.

The following steps show how to define both the segment and its associated rules using the API.

Create the machine-learning segment.

curl -u $TENANT:$KEY -X POST -H "Content-Type: application/json" -d '{
"name": "ml",
"selector": "{namespace=~\"(ml-dev|ml-staging|ml)\"}"
}' $URL/aggregations/rules/segments

The output of this command shows the value of the new segment, including the unique identifier in the id field.

Apply segment recommendations. After you create the segment, the recommendations service starts generating per-segment recommendations that you can apply. For example, if the segment id for the machine-learning team is 01J35VC3XPZD6GBB0QXJ86KS1E, you can download the recommendations list for that segment. For example:
bash
```
curl -u $TENANT:$KEY -X GET $URL/aggregations/recommendations?segment=01J35VC3XPZD6GBB0QXJ86KS1E -o ml.recommendations.json
```
After you download the recommended rules, you can be apply them by referencing the corresponding segment when you update the rules. For example:
bash
```
curl -u $TENANT:$KEY -X POST $URL/aggregations/rules?segment=01J35VC3XPZD6GBB0QXJ86KS1E -d @ml.recommendations.json -H "Content-Type: application/json"
```

Create per-segment exemptions. You can define a set of exemptions on a per-segment basis. For example:

curl -u $TENANT:$KEY -X POST $URL/recommendations/exemptions?segment=01J35VC3XPZD6GBB0QXJ86KS1E -d @ml.exemptions.json -H "Content-Type: application/json"

Update the segment, as required. For example:

curl -u $TENANT:$KEY -X PUT -H "Content-Type:   application/json" -d '{
 "name": "machine-learning",
 "selector": "{namespace=~\"(ml-dev|ml-staging|ml| machine-learning)\"}"
}' $URL/aggregations/rules/segments

[
  {
    "fallback_to_default": false,
    "name": "mimir",
    "selector": "{namespace=~\"(adaptive-metrics|mimir-cluster-1|mimir-cluster-2)\"}"
  },
  {
    "fallback_to_default": false,
    "name": "loki",
    "selector": "{namespace=~\"(loki-cluster-1|loki-cluster-2)\"}"
  }
]

The rules.json file contains the default fallback rules for Adaptive Metrics if a series doesn’t match any defined segments. This file uses the same format as the per-team rules files. For example:

[
  {
    "metric": "node_network_protocol_type",
    "drop_labels": ["agent_hostname", "instance"],
    "aggregations": ["sum:counter"]
  }
]

The rules-\*.json files contain one rule set for each team, which is only applied to that team’s metrics. For example:

[
  {
    "metric": "activity_tracker_free_slots",
    "drop_labels": ["instance", "pod"],
    "aggregations": ["sum:counter"]
  }
]

The Terraform configuration in the aggregation_rules.tf file imports these JSON files and builds a segmented rule set based on their content. For example:

locals {
 rules_json = jsondecode(file("${path.module}/rules.json"))
 all_rules_files = fileset("${path.module}", "rules-*.json")
 segments = jsondecode(file("${path.module}/segments.json"))


 segmented_rules_json = {
   for file in local.all_rules_files :
   regex("rules-(.+).json", file)[0] => jsondecode(file("${path.module}/${file}"))
 }
}


resource "grafana-adaptive-metrics_ruleset" "all_rules" {
 rules = local.rules_json
}


resource "grafana-adaptive-metrics_segment" "all_segments" {
 for_each = { for segment in module.common.segments : segment.name => segment }


 name                = each.value.name
 selector            = each.value.selector
 fallback_to_default = each.value.fallback_to_default
}


resource "grafana-adaptive-metrics_ruleset" "segmented_rules" {
 for_each = local.segmented_rules_json


 segment = grafana-adaptive-metrics_segment.all_segments[each.key].id
 rules   = each.value
}

Was this page helpful?

Email docs@grafana.com

Help and support

Community

Configure segmentation

Segments

Segmentation workflow

Create segments using the UI

View recommendations per segment

Create segments using Terraform

Prerequisites

Steps

Create segments using Segment API

Update or delete segments

View segments and rules

Best practices

Choose an appropriate label

Define one segment per team

Migrate one segment at a time

Regularly update segment definitions

Example

Was this page helpful?

Related resources from Grafana Labs