Hệ thống ELK Stack với tính năng filtering database metrics, sử dụng whitelist approach để bảo mật dữ liệu.

Metricbeat → Logstash (Filter) → Elasticsearch → Kibana
                ↓
        ┌─────────────────┐
        │  Filter Logic   │
        │  (Whitelist)    │
        └─────────────────┘
                ↓
    ┌─────────────────────────┐
    │  metricbeat-filtered-*  │  ← Allowed events
    │  block-metric-*         │  ← Blocked events
    └─────────────────────────┘

cloudwatch/
├── docker-compose.yml              # ELK Stack configuration
├── logstash/
│   ├── config/
│   │   ├── logstash.yml            # Logstash main config
│   │   ├── pipelines.yml           # Pipeline configuration
│   │   └── database-filter-rules.yml # Filter rules (whitelist)
│   └── pipeline/
│       └── simple.conf             # Main pipeline with filter logic
├── metricbeat/
│   └── metricbeat.yml              # Metricbeat configuration
└── scripts/
    ├── monitor-whitelist.sh        # Monitor filter results
    ├── monitor-results.sh          # General monitoring
    └── test-filter-simple.sh       # Test filter system

docker-compose up -d

# Elasticsearch
curl -X GET "http://localhost:9200/_cluster/health" -u "elastic:changeme123"

# Kibana
open http://localhost:5601

# Logstash
docker logs logstash --tail 20

./scripts/monitor-whitelist.sh

• Default Action: DENY (block all by default)
• Allowed Services: MySQL, PostgreSQL, Elasticsearch
• Blocked Services: System, Docker, Kibana, Logstash, Unknown services

• MySQL: status, galera_status, performance
• PostgreSQL: database, bgwriter, activity
• Elasticsearch: node, cluster_stats, index

• Allowed Events → metricbeat-filtered-*
• Blocked Events → block-metric-*

# Monitor filter results
./scripts/monitor-whitelist.sh

# Continuous monitoring
./scripts/monitor-whitelist.sh --continuous

curl -X GET "http://localhost:9200/_cat/indices?v" -u "elastic:changeme123"

# Allowed events count
curl -X GET "http://localhost:9200/metricbeat-filtered-*/_count" -u "elastic:changeme123"

# Blocked events count
curl -X GET "http://localhost:9200/block-metric-*/_count" -u "elastic:changeme123"

./scripts/test-filter-simple.sh

1. Normal MySQL metrics → Should be allowed
2. MySQL with sensitive data → Fields should be removed
3. PostgreSQL with sensitive query → Query should be blocked
4. High connection count → Should trigger alert/block
5. Invalid data → Should be blocked
6. Unknown service types → Should be blocked

• Blocked Fields: *password*, *secret*, *key*, *token*
• Query Filtering: SQL injection patterns, sensitive content
• Value Limits: Connection thresholds, rate limiting

• default_deny_policy
• unknown_service_type:*
• metricset_not_in_whitelist:*
• missing_required_field:*
• connection_limit_exceeded

Edit logstash/config/database-filter-rules.yml:

global:
  enabled: true
  default_action: "deny"  # Whitelist mode

mysql:
  enabled: true
  allowed_metricsets:
    - "status"
    - "galera_status"
    - "performance"

docker restart logstash

• Single Pipeline: Unified processing for all inputs
• Efficient Filtering: Ruby-based filter logic
• Separate Indices: Clean data separation
• Monitoring: Real-time stats and alerts

docker logs logstash --tail 50

docker logs logstash | grep -E "(filter_status|blocked|allowed)"

# Test TCP input
echo '{"test": "message"}' | nc localhost 5001

# Test HTTP input
curl -X POST "http://localhost:8090" -H "Content-Type: application/json" -d '{"test": "message"}'

• Whitelist Mode: Only explicitly allowed services/metricsets pass through
• Audit Trail: All blocked events are logged with reasons
• Scalable: Easy to add/remove allowed services
• Secure: Sensitive data automatically filtered out

• Database Monitoring: Secure MySQL/PostgreSQL metrics collection
• Compliance: Ensure no sensitive data leakage
• Performance: Monitor only relevant metrics
• Security: Block unknown/suspicious data sources