Skip to content

Commit 226e9f1

Browse files
cyriltovenacyriltovena
authored
feat: add kafka development environment (#15603)
1 parent b2f46de commit 226e9f1

File tree

5 files changed

+266
-0
lines changed

5 files changed

+266
-0
lines changed

‎.gitignore

+1
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ dist
3535
.idea
3636
pkg/loki/wal
3737
tools/lambda-promtail/main
38+
tools/dev/kafka/data/
3839

3940
# Submodule added by `act` CLI
4041
_shared-workflows-dockerhub-login

‎tools/dev/kafka/README.md

+124
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
# Loki Kafka Development Setup
2+
3+
This directory contains the development environment for testing Loki with Kafka integration. The setup provides supporting services (Kafka, Grafana, and a log generator) via docker-compose, while Loki itself needs to be run manually from source code for development purposes.
4+
5+
## Quick Start
6+
7+
1. Start the supporting services (Kafka, Grafana, Log Generator):
8+
```bash
9+
docker-compose up -d
10+
```
11+
12+
2. Run Loki manually with Kafka configuration:
13+
```bash
14+
# From the root of the Loki repository
15+
go run ./cmd/loki/main.go --config.file=tools/dev/kafka/loki-local-config.debug.yaml --log.level=debug -target=all
16+
```
17+
18+
Note: Loki is not included in docker-compose as it's intended to be run directly from source code for development.
19+
20+
## Services
21+
22+
### Kafka
23+
- Broker accessible at `localhost:9092`
24+
- Uses KRaft (no ZooKeeper required)
25+
- Single broker setup for development
26+
- Topic `loki` is used for log ingestion
27+
28+
### Kafka UI
29+
- Web interface available at http://localhost:8080
30+
- Monitor topics, messages, and consumer groups
31+
- No authentication required
32+
33+
### Grafana
34+
- Available at http://localhost:3000
35+
- Anonymous access enabled (Admin privileges)
36+
- Pre-configured with Loki data source
37+
- Features enabled:
38+
- Loki logs dataplane
39+
- Explore logs shard splitting
40+
- Loki explore app
41+
42+
### Log Generator
43+
- Automatically sends sample logs to Loki
44+
- Useful for testing and development
45+
- Configured to push logs directly to Loki's HTTP endpoint
46+
47+
## Configuration Files
48+
49+
- `docker-compose.yaml`: Service definitions and configuration
50+
- `loki-local-config.debug.yaml`: Loki configuration with Kafka enabled
51+
- Kafka ingestion enabled
52+
- Local storage in `/tmp/loki`
53+
- Debug logging enabled
54+
55+
## Debugging
56+
57+
### VSCode Configuration
58+
Create `.vscode/launch.json` in the root directory:
59+
```json
60+
{
61+
"version": "0.2.0",
62+
"configurations": [
63+
{
64+
"name": "Launch Loki (Kafka)",
65+
"type": "go",
66+
"request": "launch",
67+
"mode": "auto",
68+
"program": "${workspaceFolder}/cmd/loki/main.go",
69+
"args": [
70+
"--config.file=../../tools/dev/kafka/loki-local-config.debug.yaml",
71+
"--log.level=debug",
72+
"-target=all"
73+
],
74+
"buildFlags": "-mod vendor"
75+
}
76+
]
77+
}
78+
```
79+
80+
## Common Tasks
81+
82+
### View Logs
83+
```bash
84+
# All services
85+
docker-compose logs -f
86+
87+
# Specific service
88+
docker-compose logs -f kafka
89+
docker-compose logs -f grafana
90+
```
91+
92+
### Reset Environment
93+
```bash
94+
# Stop and remove containers
95+
docker-compose down
96+
97+
# Remove local Loki data
98+
rm -rf /tmp/loki
99+
100+
# Start fresh
101+
docker-compose up -d
102+
```
103+
104+
### Verify Setup
105+
1. Check Kafka UI (http://localhost:8080) for:
106+
- Broker health
107+
- Topic creation
108+
- Message flow
109+
110+
2. Check Grafana (http://localhost:3000):
111+
- Navigate to Explore
112+
- Select Loki data source
113+
- Query logs using LogQL
114+
115+
## Troubleshooting
116+
117+
- **Loki fails to start**: Check if port 3100 is available
118+
- **No logs in Grafana**:
119+
- Verify Kafka topics are created
120+
- Check log generator is running
121+
- Verify Loki is receiving data through Kafka UI
122+
- **Kafka connection issues**:
123+
- Ensure broker is running (`docker-compose ps`)
124+
- Check broker logs (`docker-compose logs broker`)

‎tools/dev/kafka/docker-compose.yaml

+53
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
services:
2+
grafana:
3+
image: grafana/grafana-enterprise:latest
4+
environment:
5+
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
6+
- GF_AUTH_ANONYMOUS_ENABLED=true
7+
- GF_AUTH_BASIC_ENABLED=false
8+
- GF_FEATURE_TOGGLES_ENABLE=accessControlOnCall lokiLogsDataplane exploreLogsShardSplitting
9+
- GF_INSTALL_PLUGINS=https://storage.googleapis.com/integration-artifacts/grafana-lokiexplore-app/grafana-lokiexplore-app-latest.zip;grafana-lokiexplore-app
10+
ports:
11+
- 3000:3000/tcp
12+
volumes:
13+
- ./provisioning:/etc/grafana/provisioning/
14+
- ./data/grafana/:/var/lib/grafana/
15+
extra_hosts:
16+
- 'host.docker.internal:host-gateway'
17+
kafka-ui:
18+
image: provectuslabs/kafka-ui:latest
19+
ports:
20+
- 8080:8080
21+
environment:
22+
KAFKA_CLUSTERS_0_NAME: local
23+
KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: broker:29092
24+
KAFKA_CLUSTERS_0_METRICS_PORT: 9997
25+
depends_on:
26+
- broker
27+
extra_hosts:
28+
- 'host.docker.internal:host-gateway'
29+
broker:
30+
image: apache/kafka:latest
31+
hostname: broker
32+
container_name: broker
33+
ports:
34+
- 9092:9092
35+
environment:
36+
KAFKA_BROKER_ID: 1
37+
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT,CONTROLLER:PLAINTEXT
38+
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
39+
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
40+
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
41+
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
42+
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
43+
KAFKA_PROCESS_ROLES: broker,controller
44+
KAFKA_NODE_ID: 1
45+
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker:29093
46+
KAFKA_LISTENERS: PLAINTEXT://broker:29092,CONTROLLER://broker:29093,PLAINTEXT_HOST://0.0.0.0:9092
47+
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
48+
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
49+
KAFKA_LOG_DIRS: /tmp/kraft-combined-logs
50+
CLUSTER_ID: MkU3OEVBNTcwNTJENDM2Qk
51+
generator:
52+
image: ctovena/log-generator:latest
53+
command: -url http://host.docker.internal:3100/loki/api/v1/push

‎tools/dev/kafka/loki-local-config.debug.yaml

+77
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
auth_enabled: false
2+
3+
server:
4+
http_listen_port: 3100
5+
grpc_listen_port: 9096
6+
log_level: info
7+
grpc_server_max_concurrent_streams: 1000
8+
9+
common:
10+
instance_addr: 127.0.0.1
11+
path_prefix: /tmp/loki
12+
storage:
13+
filesystem:
14+
chunks_directory: /tmp/loki/chunks
15+
rules_directory: /tmp/loki/rules
16+
replication_factor: 1
17+
ring:
18+
instance_id: local
19+
kvstore:
20+
store: inmemory
21+
22+
kafka_config:
23+
topic: "loki"
24+
25+
querier:
26+
query_partition_ingesters: true
27+
28+
ingester:
29+
kafka_ingestion:
30+
enabled: true
31+
32+
distributor:
33+
kafka_writes_enabled: true
34+
ingester_writes_enabled: false
35+
36+
query_range:
37+
results_cache:
38+
cache:
39+
embedded_cache:
40+
enabled: false
41+
max_size_mb: 100
42+
43+
limits_config:
44+
metric_aggregation_enabled: true
45+
46+
schema_config:
47+
configs:
48+
- from: 2020-10-24
49+
store: tsdb
50+
object_store: filesystem
51+
schema: v13
52+
index:
53+
prefix: index_
54+
period: 24h
55+
56+
pattern_ingester:
57+
enabled: true
58+
metric_aggregation:
59+
loki_address: localhost:3100
60+
61+
ruler:
62+
alertmanager_url: http://localhost:9093
63+
64+
frontend:
65+
encoding: protobuf
66+
# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
67+
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
68+
#
69+
# Statistics help us better understand how Loki is used, and they show us performance
70+
# levels for most users. This helps us prioritize features and documentation.
71+
# For more information on what's sent, look at
72+
# https://github.com/grafana/loki/blob/main/pkg/analytics/stats.go
73+
# Refer to the buildReport method to see what goes into a report.
74+
#
75+
# If you would like to disable reporting, uncomment the following lines:
76+
#analytics:
77+
# reporting_enabled: false

‎tools/dev/kafka/provisioning/datasources/default.yml

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
apiVersion: 1
2+
3+
datasources:
4+
- name: gdev-testdata
5+
isDefault: true
6+
type: testdata
7+
- name: gdev-loki
8+
type: loki
9+
uid: gdev-loki
10+
access: proxy
11+
url: http://host.docker.internal:3100

0 commit comments

Comments
 (0)