This project provides an end-to-end template for deploying a robust multi-service Kubernetes cluster. It now includes improvements in Kafka topic management, enhanced metrics filtering, and refined deployment strategies. With updated services and tighter integration between components, the cluster is optimized for real-time monitoring and data ingestion.

• datagenerator
  A Go-based helper service that continuously seeds the system with real-time metrics. It now features improved performance, error handling, and logging for smoother operations.

• frontend
  A modern React-based web frontend with an enhanced responsive design. Build scripts and a Dockerfile are provided to quickly generate production-ready images.

• k8s
  Contains comprehensive Kubernetes manifests for deploying and managing various services and infrastructure components:

  • ingress.yaml: Updated to support new domain rules for Prometheus and training-related metrics.
  • database: Manifests for PostgreSQL deployment with fine-tuned resource and storage settings.
  • frontend: Updated manifests to match the new deployment requirements of the frontend.
  • kafka: Manifests now include a dedicated topic for training metrics along with the standard topics.
  • privatebox, publicbox, rss-exporter: Standalone backend services written in Go with improved internal business logic, error management, and logging. Notably, privatebox now integrates a training metrics filter.

• privatebox, publicbox, rss-exporter
  These services now boast refined message handling, enhanced Docker configurations, and updated Go modules. They include:

  • A Dockerfile for containerization.
  • Up-to-date Makefiles.
  • Server code (server.go) and additional internal logic organized under internal/.

• prometheus
  Contains configuration for Prometheus monitoring with refined scrape targets and dashboards to better observe multi-service metrics.

The top-level Makefile simplifies deploying the updated cluster:

• up:

  • Builds Docker images for rss-exporter, publicbox, privatebox, and frontend.
  • Pulls the latest PostgreSQL amd other images.
  • Sets up Kafka
  • Applies all Kubernetes manifests.

• down:

  • Tears down all deployed Kubernetes resources and cleans up volumes and images.

• ports-export:

  • For development, establishes port forwarding for PostgreSQL.

1. Update /etc/hosts Before starting the services, update your /etc/hosts file by adding

   192.168.49.2 frontend.local
   192.168.49.2 publicbox.local
   192.168.49.2 privatebox.local
   192.168.49.2 rss-exporter.local
   192.168.49.2 prometheus.local
   

   to it.

2. Deploy the Cluster:
   Run the following command in the project root:

   make up

3. Run migrations
   Run http://publicbox.local/api/v1/migrate?migration_password=admin in your browser or API client.

4. Access the Services:

   • Frontend: http://frontend.local/
   • PostgreSQL: Accessible on port 5432
   • Kafka: Accessible on port 9092
   • Other services via their respective hosts (e.g., publicbox.local, privatebox.local, rss-exporter.local, prometheus.local)

5. Add sample data:
   In datagenerator directory run go run .. This command will add new real data.

6. Monitor the System:
   Open the Prometheus UI at http://prometheus.local/.

7. Teardown the Cluster:
   Run:

   make down

This project leverages a diverse set of modern technologies to ensure scalability, reliability, and performance:

• Kubernetes:
  Orchestrates containerized microservices for seamless deployment and scaling.

• Docker:
  Containerizes application components to maintain consistent environments across different stages.

• Go:
  Implements backend services (datagenerator, privatebox, publicbox, rss-exporter) with a focus on performance and concurrency.

• React:
  Powers the frontend with a modern, responsive design and efficient UI rendering.

• Kafka:
  Manages real-time message streaming, including dedicated topics for training metrics.

• Prometheus:
  Monitors and collects metrics from various cluster components for robust observability.

• PostgreSQL:
  Serves as the primary relational database with fine-tuned performance and storage settings.