Skip to content

Commit 2d0bbfd

Browse files
Jayclifford345JSticklerpetewall
authored
feat: Meta Monitoring Loki v2 (Part 1) - Telemetry Collection (#16665)
Signed-off-by: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Co-authored-by: J Stickler <julie.stickler@grafana.com> Co-authored-by: Pete Wall <pete@petewall.net>
1 parent 6824f4e commit 2d0bbfd

File tree

2 files changed

+312
-0
lines changed

2 files changed

+312
-0
lines changed

‎production/helm/meta-monitoring/README.md

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
# Meta-monitoring Loki - Kubernetes Helm Chart
2+
3+
This Helm chart provides comprehensive monitoring for Loki deployments in Kubernetes, based on the [Grafana Kubernetes Monitoring Helm Chart](https://github.com/grafana/k8s-monitoring-helm/tree/main).
4+
5+
## Overview
6+
7+
The Meta-monitoring chart collects metrics and logs from Loki deployments using Grafana Alloy (an OpenTelemetry Collector distribution) and sends them to your preferred observability backend. It's designed to work with the pre-compiled Loki mixins to provide dashboards and alerts specifically tailored for Loki monitoring.
8+
9+
## Prerequisites
10+
11+
- [kubectl](https://kubernetes.io/docs/reference/kubectl/)
12+
- Kubernetes cluster
13+
- Helm 3.x
14+
- Access to Grafana Cloud or a self-managed Prometheus/Loki/Grafana stack
15+
- For self-managed destinations: credentials stored in Kubernetes secrets
16+
17+
## Installation
18+
19+
1. Create the required secrets for your observability backend:
20+
21+
```bash
22+
# For Grafana Cloud or self-managed stack
23+
kubectl create namespace meta
24+
kubectl create secret generic metrics --namespace meta \
25+
--from-literal=username='YOUR_PROMETHEUS_USERNAME' \
26+
--from-literal=password='YOUR_PROMETHEUS_PASSWORD'
27+
28+
kubectl create secret generic logs --namespace meta \
29+
--from-literal=username='YOUR_LOKI_USERNAME' \
30+
--from-literal=password='YOUR_LOKI_PASSWORD'
31+
```
32+
33+
2. Install the Helm chart:
34+
35+
```bash
36+
helm repo add grafana https://grafana.github.io/helm-charts
37+
helm install meta-loki grafana/k8s-monitoring \
38+
--namespace meta \
39+
-f values.yaml
40+
```
41+
42+
## Configuration
43+
44+
The default configuration is set up for Grafana Cloud, but you can direct it to your self-managed monitoring stack by modifying the destinations in `values.yaml`:
45+
46+
```yaml
47+
destinations:
48+
- name: prometheus
49+
type: prometheus
50+
url: https://<PROMETHEUS-ENDPOINT>/api/prom/push
51+
# Configure authentication as needed
52+
53+
- name: loki
54+
type: loki
55+
url: https://<LOKI-ENDPOINT>/loki/api/v1/push
56+
# Configure authentication as needed
57+
```
58+
59+
Note: Authentication is based on a pre-configured secret. The Helm chart does support more advanced authentication methods. Examples are provided in the [k8s-monitoring documentation](https://github.com/grafana/k8s-monitoring-helm/tree/main/charts/k8s-monitoring/docs/examples/auth)
60+
61+
## Key Features
62+
63+
- **Cluster Event Collection**: Captures Kubernetes events as logs
64+
- **Metrics Collection**: Uses cadvisor, kubelet, and kube-state-metrics
65+
- **Pod Log Collection**: Collects logs from Loki pods
66+
- **Integrated Dashboards**: Works with pre-compiled Loki mixin dashboards
67+
- **Alert Rules**: Pre-configured alerting rules for Loki components
68+
69+
## Loki Mixin Compiled Files
70+
71+
This chart is designed to work with the Loki mixins found in the `loki-mixin-compiled` directory, which contains:
72+
73+
### Files Overview
74+
75+
- **dashboards/**: Pre-compiled Grafana dashboards for visualizing Loki performance and health
76+
- **alerts.yaml**: Pre-configured alerting rules for Loki components
77+
- **rules.yaml**: Recording rules that create metrics used by dashboards and alerts
78+
79+
### Using the Dashboards
80+
81+
The `dashboards/` directory contains JSON files that can be imported directly into Grafana:
82+
83+
1. From your Grafana UI, go to **Dashboards → Import**.
84+
1. Upload the JSON file or paste its contents.
85+
1. Configure the data source (should match your Prometheus).
86+
1. Click **Import**.
87+
88+
Alternatively, use the Grafana API to programmatically import dashboards:
89+
90+
```bash
91+
# Example using curl to import a dashboard
92+
curl -X POST -H "Content-Type: application/json" -H "Authorization: Bearer YOUR_API_KEY" \
93+
-d @path/to/dashboard.json \
94+
https://your-grafana-instance/api/dashboards/db
95+
```
96+
97+
### Loading Alert and Recording Rules
98+
99+
For Grafana Cloud Prometheus or Grafana Mimir, use `[mimirtool](https://grafana.com/docs/mimir/latest/manage/tools/mimirtool/#installation)` to load the rules:
100+
101+
```bash
102+
# Install mimirtool
103+
go install github.com/grafana/mimir/pkg/mimirtool@latest
104+
105+
# Load alert rules
106+
mimirtool rules load alerts.yaml \
107+
--address=https://prometheus-prod-xxx.grafana.net/api/prom \
108+
--id=<Your-Stack-ID> \
109+
--key=<Your-API-Key>
110+
111+
# Load recording rules
112+
mimirtool rules load rules.yaml \
113+
--address=https://prometheus-prod-xxx.grafana.net/api/prom \
114+
--id=<Your-Stack-ID> \
115+
--key=<Your-API-Key>
116+
```
117+
118+
For self-managed Prometheus:
119+
120+
```bash
121+
# For Prometheus
122+
cp rules.yaml alerts.yaml /etc/prometheus/rules/
123+
# Then reload Prometheus configuration
124+
curl -X POST http://prometheus:9090/-/reload
125+
```
126+
127+
## Components Monitored
128+
129+
The meta-monitoring chart monitors:
130+
131+
- Loki components via label selection (`app.kubernetes.io/name: loki`)
132+
- Alloy collectors in the `meta` namespace
133+
- Kubernetes resources in the `loki` and `meta` namespaces
134+
135+
## Advanced Configuration
136+
137+
See the [k8s-monitoring documentation](https://github.com/grafana/k8s-monitoring-helm/tree/main/charts/k8s-monitoring/docs) for additional configuration options for:
138+
139+
- Authentication methods
140+
- Collection tuning
141+
- Metric processing
142+
- Log processing
143+
144+
## References
145+
146+
- [Grafana Kubernetes Monitoring](https://github.com/grafana/k8s-monitoring-helm/tree/main)
147+
- [Alloy integration](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/charts/feature-integrations/docs/integrations/alloy.md)
148+
- [Loki integration](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/charts/feature-integrations/docs/integrations/loki.md)
149+
- [Mimirtool documentation](https://grafana.com/docs/mimir/latest/manage/tools/mimirtool/)

‎production/helm/meta-monitoring/values.yaml

Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
---
2+
# Sets the global scrape interval for Alloy components
3+
global:
4+
scrapeInterval: 15s
5+
6+
# Global Label to be added to all telemetry data. Should reflect a recognizable name for the cluster.
7+
cluster:
8+
name: loki-meta-monitoring-cluster
9+
10+
# Destinations for telemetry data (metrics, logs)
11+
# The credentials are stored in the secrets metrics and logs
12+
# Further authentication methods are supported, see the documentation (https://github.com/grafana/k8s-monitoring-helm/tree/main/charts/k8s-monitoring/docs/examples/auth)
13+
destinations:
14+
- name: prometheus
15+
type: prometheus
16+
url: https://<PROMETHEUS-ENDPOINT>/api/prom/push
17+
auth:
18+
type: basic
19+
usernameKey: username
20+
passwordKey: password
21+
secret:
22+
create: false
23+
name: metrics
24+
namespace: meta
25+
26+
- name: loki
27+
type: loki
28+
url: https://<LOKI-ENDPOINT>/loki/api/v1/push
29+
auth:
30+
type: basic
31+
usernameKey: username
32+
passwordKey: password
33+
secret:
34+
create: false
35+
name: logs
36+
namespace: meta
37+
38+
# Components to be monitored by the meta-monitoring Helm chart.
39+
# Two integrations are being used:
40+
# - alloy: https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/charts/feature-integrations/docs/integrations/alloy.md
41+
# - loki: https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/charts/feature-integrations/docs/integrations/loki.md
42+
integrations:
43+
collector: alloy-singleton
44+
alloy:
45+
instances:
46+
# monitor the collectors gathering and sending metrics/logs to the local cluster
47+
- name: alloy
48+
labelSelectors:
49+
app.kubernetes.io/name: [alloy-singleton]
50+
namespaces:
51+
- meta
52+
53+
loki:
54+
instances:
55+
- name: loki
56+
namespaces:
57+
- loki
58+
labelSelectors:
59+
app.kubernetes.io/name: loki
60+
logs:
61+
tuning:
62+
# extract logfmt fields and set them as structured metadata
63+
structuredMetadata:
64+
caller:
65+
tenant:
66+
org_id:
67+
user:
68+
# (Optional) Kubernetes events are captured as logs and are annotated with additional metadata to make them easier to search and filter.
69+
clusterEvents:
70+
enabled: true
71+
collector: alloy-singleton
72+
namespaces:
73+
- meta
74+
- loki
75+
# A collection of metric collectors that gather metrics from various sources in the cluster.
76+
# (Required) cadvisor - Used to collect Loki pod metrics. Cadvisor is automatically deployed.
77+
# kubelet - Kubernetes information on each node
78+
# kubeletResource - Scrape resource metrics from the Kubelet.
79+
# kube-state-metrics - A simple service that listens to the Kubernetes API server and generates metrics about the state of the objects.
80+
clusterMetrics:
81+
enabled: true
82+
collector: alloy-singleton
83+
kubelet:
84+
enabled: true
85+
kubeletResource:
86+
enabled: true
87+
cadvisor:
88+
enabled: true
89+
metricsTuning:
90+
includeNamespaces:
91+
- loki
92+
- meta
93+
apiServer:
94+
enabled: false
95+
kubeControllerManager:
96+
enabled: false
97+
kubeDNS:
98+
enabled: false
99+
kubeProxy:
100+
enabled: false
101+
kubeScheduler:
102+
enabled: false
103+
kube-state-metrics:
104+
enabled: true
105+
namespaces: loki,meta
106+
metricsTuning:
107+
useDefaultAllowList: false
108+
includeMetrics: [(.+)]
109+
node-exporter:
110+
enabled: true
111+
deploy: true
112+
metricsTuning:
113+
useIntegrationAllowList: true
114+
windows-exporter:
115+
enabled: false
116+
deploy: false
117+
kepler:
118+
enabled: false
119+
deploy: false
120+
opencost:
121+
enabled: false
122+
deploy: false
123+
124+
nodeLogs:
125+
enabled: false
126+
127+
# Enable pod log collection for the cluster. Will collect logs from all pods in both the meta and loki namespace.
128+
podLogs:
129+
enabled: true
130+
collector: alloy-singleton
131+
labelsToKeep:
132+
- app
133+
- app_kubernetes_io_name
134+
- component
135+
- container
136+
- job
137+
- level
138+
- namespace
139+
- pod
140+
- service_name
141+
- cluster
142+
gatherMethod: kubernetesApi
143+
namespaces:
144+
- meta
145+
- loki
146+
147+
# Collectors
148+
149+
# The Alloy Singleton is a single instance of the Alloy Collector that is deployed in the cluster.
150+
alloy-singleton:
151+
enabled: true
152+
153+
alloy-metrics:
154+
enabled: false
155+
156+
alloy-logs:
157+
enabled: false
158+
159+
alloy-profiles:
160+
enabled: false
161+
162+
alloy-receiver:
163+
enabled: false

0 commit comments

Comments
 (0)