Skip to content

Commit ff88f3c

Browse files
fix: Updated Loki Otlp Ingest Configuration (#12648)
Co-authored-by: J Stickler <julie.stickler@grafana.com>
1 parent 0d8ff9e commit ff88f3c

File tree

1 file changed

+72
-61
lines changed

1 file changed

+72
-61
lines changed

‎docs/sources/send-data/otel/_index.md

Lines changed: 72 additions & 61 deletions
Original file line numberDiff line numberDiff line change
@@ -16,14 +16,21 @@ For ingesting logs to Loki using the OpenTelemetry Collector, you must use the [
1616

1717
When logs are ingested by Loki using an OpenTelemetry protocol (OTLP) ingestion endpoint, some of the data is stored as [Structured Metadata]({{< relref "../../get-started/labels/structured-metadata" >}}).
1818

19+
You must set `allow_structured_metadata` to `true` within your Loki config file. Otherwise, Loki will reject the log payload as malformed.
20+
21+
```yaml
22+
limits_config:
23+
allow_structured_metadata: true
24+
```
25+
1926
## Configure the OpenTelemetry Collector to write logs into Loki
2027
2128
You need to make the following changes to the [OpenTelemetry Collector config](https://opentelemetry.io/docs/collector/configuration/) to write logs to Loki on its OTLP ingestion endpoint.
2229
2330
```yaml
2431
exporters:
2532
otlphttp:
26-
endpoint: http://<loki-addr>/otlp
33+
endpoint: http://<loki-addr>:3100/otlp
2734
```
2835
2936
And enable it in `service.pipelines`:
@@ -50,7 +57,7 @@ exporters:
5057
otlphttp:
5158
auth:
5259
authenticator: basicauth/otlp
53-
endpoint: http://<loki-addr>/otlp
60+
endpoint: http://<loki-addr>:3100/otlp
5461
5562
service:
5663
extensions: [basicauth/otlp]
@@ -118,50 +125,52 @@ It currently only supports changing the storage of Attributes. Here is how the c
118125

119126
```yaml
120127
# OTLP log ingestion configurations
121-
otlp_config:
122-
# Configuration for Resource Attributes to store them as index labels or
123-
# Structured Metadata or drop them altogether
124-
resource_attributes:
125-
# Configure whether to ignore the default list of resource attributes set in
126-
# 'distributor.otlp.default_resource_attributes_as_index_labels' to be
127-
# stored as index labels and only use the given resource attributes config
128-
[ignore_defaults: <boolean>]
129-
130-
[attributes_config: <list of attributes_configs>]
131-
132-
# Configuration for Scope Attributes to store them as Structured Metadata or
133-
# drop them altogether
134-
[scope_attributes: <list of attributes_configs>]
135-
136-
# Configuration for Log Attributes to store them as Structured Metadata or
137-
# drop them altogether
138-
[log_attributes: <list of attributes_configs>]
139-
140-
attributes_config:
141-
# Configures action to take on matching Attributes. It allows one of
142-
# [structured_metadata, drop] for all Attribute types. It additionally allows
143-
# index_label action for Resource Attributes
144-
[action: <string> | default = ""]
145-
146-
# List of attributes to configure how to store them or drop them altogether
147-
[attributes: <list of strings>]
148-
149-
# Regex to choose attributes to configure how to store them or drop them
150-
# altogether
151-
[regex: <Regexp>]
128+
limits_config:
129+
otlp_config:
130+
# Configuration for Resource Attributes to store them as index labels or
131+
# Structured Metadata or drop them altogether
132+
resource_attributes:
133+
# Configure whether to ignore the default list of resource attributes set in
134+
# 'distributor.otlp.default_resource_attributes_as_index_labels' to be
135+
# stored as index labels and only use the given resource attributes config
136+
[ignore_defaults: <boolean>]
137+
138+
[attributes_config: <list of attributes_configs>]
139+
140+
# Configuration for Scope Attributes to store them as Structured Metadata or
141+
# drop them altogether
142+
[scope_attributes: <list of attributes_configs>]
143+
144+
# Configuration for Log Attributes to store them as Structured Metadata or
145+
# drop them altogether
146+
[log_attributes: <list of attributes_configs>]
147+
148+
attributes_config:
149+
# Configures action to take on matching Attributes. It allows one of
150+
# [structured_metadata, drop] for all Attribute types. It additionally allows
151+
# index_label action for Resource Attributes
152+
[action: <string> | default = ""]
153+
154+
# List of attributes to configure how to store them or drop them altogether
155+
[attributes: <list of strings>]
156+
157+
# Regex to choose attributes to configure how to store them or drop them
158+
# altogether
159+
[regex: <Regexp>]
152160
```
153161

154162
Here are some example configs to change the default mapping of OTLP to Loki format:
155163

156164
#### Example 1:
157165

158166
```yaml
159-
otlp_config:
160-
resource_attributes:
161-
attributes_config:
162-
- action: index_label
163-
attributes:
164-
- service.group
167+
limits_config:
168+
otlp_config:
169+
resource_attributes:
170+
attributes_config:
171+
- action: index_label
172+
attributes:
173+
- service.group
165174
```
166175

167176
With the example config, here is how various kinds of Attributes would be stored:
@@ -172,12 +181,13 @@ With the example config, here is how various kinds of Attributes would be stored
172181
#### Example 2:
173182

174183
```yaml
175-
otlp_config:
176-
resource_attributes:
177-
ignore_defaults: true
178-
attributes_config:
179-
- action: index_label
180-
regex: service.group
184+
limits_config:
185+
otlp_config:
186+
resource_attributes:
187+
ignore_defaults: true
188+
attributes_config:
189+
- action: index_label
190+
regex: service.group
181191
```
182192

183193
With the example config, here is how various kinds of Attributes would be stored:
@@ -188,25 +198,26 @@ With the example config, here is how various kinds of Attributes would be stored
188198
#### Example 2:
189199

190200
```yaml
191-
otlp_config:
192-
resource_attributes:
193-
attributes_config:
194-
- action: index_label
195-
regex: service.group
196-
scope_attributes:
197-
- action: drop
198-
attributes:
199-
- method.name
200-
log_attributes:
201-
- action: structured_metadata
202-
attributes:
203-
- user.id
204-
- action: drop
205-
regex: .*
201+
limits_config:
202+
otlp_config:
203+
resource_attributes:
204+
attributes_config:
205+
- action: index_label
206+
regex: service.group
207+
scope_attributes:
208+
- action: drop
209+
attributes:
210+
- method.name
211+
log_attributes:
212+
- action: structured_metadata
213+
attributes:
214+
- user.id
215+
- action: drop
216+
regex: .*
206217
```
207218

208219
With the example config, here is how various kinds of Attributes would be stored:
209220
* Store all 17 Resource Attributes mentioned earlier and `service.group` Resource Attribute as index labels.
210221
* Store remaining Resource Attributes as Structured Metadata.
211222
* Drop Scope Attribute named `method.name` and store all other Scope Attributes as Structured Metadata.
212-
* Store Log Attribute named `user.id` as Structured Metadata and drop all other Log Attributes.
223+
* Store Log Attribute named `user.id` as Structured Metadata and drop all other Log Attributes.

0 commit comments

Comments
 (0)