Decoupling EPR Image from Integration Packages #1907

Executive Summary

This request proposes decoupling the Elastic Package Registry (EPR) OCI image from integration packages to enable hot-loading of new versions and packages without image rebuilds. This evolution would significantly improve operational efficiency in air-gapped environments while maintaining security and compatibility requirements.

Key Benefits:

• 95% reduction in deployment time for integration updates (from hours to minutes)
• 80% reduction in image size for frequent deployments
• Enhanced development velocity for internal integrations
• Maintained air-gap compatibility with signing mechanisms

1. Context and Problem Statement

Current State

In air-gapped environments, the Elastic Package Registry (EPR) is distributed as "distribution" images (production/lite/versioned) containing a snapshot of packages. This architecture creates several operational challenges:

Technical Constraints:

• Strong coupling between EPR OCI image and package inventory
• Massive EPR images (hundreds of GB) difficult to transport and deploy in isolated environments
• Systematic image rebuild and redeployment required for any package update
• High operational delays and costs for often incremental and minor updates

Business Impact:

• Slow iteration cycle on internal integrations despite Kibana/Fleet supporting fine-grained version management
• Reduced agility for security teams developing frequently updated dashboards, rules, and detection assets
• Increased infrastructure costs and bandwidth consumption
• Developer productivity bottlenecks

Target Architecture

Enable integration package distribution independent of EPR image lifecycle with capabilities to:

1. Hot-load packages without EPR image reconstruction
2. Reduce image size and rebuild frequency while maintaining air-gap and signature compatibility
3. Provide simple, supported mechanisms for publishing and activating internal packages atomically with audit trails

2. Proposed Solution Architecture

2.1 External Content Volume Support

Implementation:

# EPR Configuration
epr:
  content:
    external_path: "/var/epr/packages"
    watch_enabled: true
    signature_verification: true
    public_key_path: "/etc/epr/signing-keys/"

Features:

• Externalized data directory via volume (PVC, hostPath, CSI) for package discovery/serving
• Filesystem watch or admin endpoint (/admin/reload) for catalog refresh without restart
• Security controls: read-only volume lock with signed format verification (sigstore, GPG)
• Trust parameter configuration for third-party package signature validation

2.2 Remote Backend Integration

Implementation:

# Remote Backend Configuration
epr:
  backends:
    internal:
      type: "s3"
      endpoint: "https://internal-registry.company.com"
      sync_interval: "300s"
      whitelist_patterns: ["internal/*", "security/*"]

Features:

• HTTP(S)/S3-compatible private backend configuration
• Periodic EPR synchronization with local mirror caching
• Incremental metadata indexing strategy
• Whitelist/pinning strategies for internal/authorized packages only

2.3 Package Administration API

Endpoints:

# Package Management
POST   /admin/packages              # Upload package
PUT    /admin/packages/{id}/verify  # Verify signature/schema
PATCH  /admin/packages/{id}/status  # Activate/retire
DELETE /admin/packages/{id}         # Remove package
GET    /admin/packages              # List versions with status

# System Operations
POST   /admin/reindex               # Trigger partial reindex
GET    /admin/audit                 # Access audit logs

Security Features:

• Package upload, verification (signature, schema), activation, unpublishing
• Version listing, state management (active/retired), partial reindexing
• Minimal audit journal (timestamp, technical identities) for compliance in restricted environments

2.4 Compatibility and Constraints Strategy

Air-Gapped Mode:

• All controls and signatures operable without Internet access
• Strict mode with comprehensive signature validation
• Maintain production/lite distributions as optional base with internal package overlay

Performance Parameters:

epr:
  performance:
    max_concurrent_uploads: 5
    index_cache_size: "512MB"
    package_quota_per_namespace: 100

3. Use Cases and Impact Analysis

Detailed Use Cases

1. Security Team Rapid Iteration

• Frequent evolution of dashboards, detections, and security assets
• Internal versioned package publication via API/volume
• Immediate activation in Kibana/Fleet without EPR rebuild

2. Multi-Team Organization

• EPR maintains "lite" foundation
• Each team publishes packages in internal namespace
• Dynamic activation for targeted Kibana spaces

3. Size Constraint Optimization

• Transition from >100GB EPR images to minimal base image + versioned internal volume
• Incremental updates of few MB instead of full image replacement