In Part 1, we covered how Elastic's Kubernetes integration ships telemetry via the EDOT Collector into Elasticsearch. In this post, we go further with an MCP (Model Context Protocol) app server that exposes that telemetry as AI-callable tools, complete with interactive React UIs rendered inline. We'll also cover how to take it further with Elastic Workflows: automated runbooks that handle the full root cause analysis loop from alert to remediation proposal.

Observability MCP App that renders where you work

The Elastic Observability MCP App (tech preview) ships six views, one per tool. Each renders inline when the tool returns, and each surfaces opinionated next-step prompts as clickable buttons so you don't have to guess the right follow-up. MCP Apps take it further than standalone agent workflows — they render live, interactive views directly inside your chat or IDE, inline in the conversation, without a context switch to Kibana.

Cluster health rollup

Ask "what's broken?" or "give me a status report" and get a one-shot orientation: overall health badge, degraded services with reasons, top pod memory consumers, anomaly severity breakdown, and service throughput — all in one inline view.

The view adapts based on what your deployment supports. APM gives you service health. Kubernetes metrics add pod and node context. ML jobs layer in anomalies. If a signal isn't present, the view tells you what's missing rather than failing. We'll begin with a status report of the Kubernetes cluster:

Compound reports like the health summary have condensed data presentation with detail-expansion so that you get to choose the appropriate amount of information to view at once. Suggested investigation actions provide guidance for both specific information being returned, as well as orienting users to other tools to run.

Service dependency graph

Ask "what calls checkout?" or "show me the topology" and get a layered dependency graph — upstream callers, downstream dependencies, protocols, call volume, and latency per edge. Hover over an edge to highlight the full call path. Let's ask Claude to "Show me the service dependencies of the frontend":

Zoom, pan, and hover to get all the details you need to understand the complex service relationships:

Anomaly Details

Ask "what's anomalous?" or "is anything unusual in checkout?" and get one of two views, chosen automatically. If multiple entities are affected, the overview mode shows severity counts, affected entities, and a by-job breakdown. If a single entity is the focus, the detail mode shows score, actual vs. typical values with a comparison bar, deviation percentage, and a time-series when available. Let's check on the frontend service:

This isn't an ESQL query — it's an explanation of results of a previously-defined anomaly detection job. As discussed in Part 1 of this blog series, the Kubernetes integration ships with a few for you to enable. This tool will help you make the most of them.

Observe

Observe is the agent's primary access primitive for Elastic — one tool, with two modes for three different needs. Say "what is the network throughput of each of my kubernetes clusters" for a table or chart of results. Say "tell me when memory drops below 80MB" or "watch the frontend memory for anything unusual for the next 10 minutes" and it blocks until the condition fires or the window expires.

The view adapts to the mode: a results table for one-shot queries, a live trend chart with current/peak/baseline stats for sampling and threshold conditions, and a severity-scored trigger card for anomaly mode. We'll use it here to identify the busiest Kubernetes node:

Assess risk with a blast radius

Ask "what happens if this node goes down?" and get a radial impact diagram: the target node at center, full-outage deployments in red, degraded in amber, unaffected in gray. A floating summary card shows pods at risk and rescheduling feasibility. Single-replica deployments are flagged as single points of failure. What would happen if our busy node were to fail:

Alert Management

With the alert management tool, you can create, list, get info, and delete alerts. We'll create an alert next, but first use Observe once more to take a quick baseline so we know the alert will make sense:

Say "alert me if frontend memory goes above 75MB" and the agent creates a persistent Kibana alerting rule — a saved object that keeps running after the conversation ends. The view renders a live rule card: rule name, condition, window, check interval, KQL filter, and tags. Next-step buttons offer to verify the rule, watch the metric stabilize, or check current cluster health. The agent confirms what was created and where to find it in Kibana:

MCP App Architecture

The app is composed of a Node.js server, six model-facing tools wired to six single-file view resources, app-only tools for re-queries, and vite-plugin-singlefile bundling. Tools are grouped by deployment backend (Universal, APM-dependent, K8s-dependent, ML-dependent), so the agent and the user both know up front which tools apply to a given deployment instead of discovering capability gaps at call time. The repo includes six Skills as separate .zip artifacts that teach the agent when and how to call each tool.

The following diagram shows the three components that make up the app: the MCP host (Claude Desktop, VS Code, or similar), which holds the LLM and the Claude skills that teach it how to use the tools; the MCP app server, a single Node.js process that exposes the tool registry, bundles the React UI views, and handles all communication with Elastic; and the Elastic Stack itself, where Elasticsearch and Kibana serve as the live data and alerting backends.

The diagram below traces the flow of a user request: Claude reads the relevant skill file to understand which tool to call and how to fill its parameters, calls the tool which triggers server-side queries against Elasticsearch and Kibana, and receives back a compact text summary alongside a React UI resource that renders inline as an interactive widget.

From alert to root cause: Investigation Workflows

Alert rules tell you something is wrong. ML modules tell you the pattern. Elastic Workflows run the diagnosis — automatically, the moment an alert fires.

We're shipping a Kubernetes Investigation Workflow (technical preview) that triggers on a Kubernetes alert and returns a structured root cause summary before you've opened a single dashboard. The SRE who gets paged opens the alert and finds the investigation already done.

The workflow is a directed graph of steps that queries multiple data sources — primarily via Elasticsearch Query Language (ES|QL), with an Elasticsearch search for the ML anomaly lookup. if steps branch on query results, choosing which corroboration to run (ML memory anomaly vs log classification) and whether to assess upstream health (only when APM dependencies exist). AI steps appear at three points: classifying log patterns on the non-OOM path, classifying upstream degraded-vs-healthy, and a final ai.summarize that synthesizes all structured evidence into a root-cause narrative.

What the investigation workflow looks like in practice

The example execution below is based on the OpenTelemetry Astronomy Shop running against Elastic — 16 services, Kafka, PostgreSQL, all pre-instrumented via OTLP. Alongside the Shop's real telemetry, we injected a synthetic OOMKill cascade, which writes synthetic K8s and APM signals into the same namespace via the EDOT data streams. The workflow can't tell our signals from real ones — it just investigates the alert.

Alert fires: CrashLoopBackOff — app-deployment in oteldemo-esyox-default. Restart count: 6.

Workflow step 1 — Characterize pod and container context

The workflow queries K8s metrics for restart count, last termination reason, and utilization against declared limits.

Result: Last termination reason OOMKilled, restart count 6. (Note: kubeletstats utilization was unavailable for this pod/window — the workflow continues gracefully.)

Workflow branches: Termination reason is OOMKilled, so the workflow takes the memory-investigation path, not the log-investigation path.

Workflow step 2a — Consult ML anomaly results

Rather than recomputing memory trends, the workflow queries the ML anomaly index for an active k8s_pod_memory_growth anomaly.

Result: No anomaly — the spike is flagged load-driven, not a suspected leak.

Workflow step 3 — Check upstream service health

The workflow enumerates upstream dependencies from APM service_destination.1m aggregates, then compares current error rate and mean latency against the same hour 7 days ago. An AI classification step decides whether upstream degradation preceded the alert. Result: One upstream — api-gateway. Current mean latency 15.13 ms, error rate 41.26%. Baseline (168h ago): identical. Classification: upstream_healthy — within 5× error / 3× latency thresholds. Upstream is ruled out.

Workflow step 4 — Correlate with recent K8s changes

Event log for the namespace shows a tight cycle of Pulled → Created → Started → Killing → BackOff repeating roughly every 60–90 seconds. No deployments or scaling events in the past two hours.

Workflow output:

ROOT CAUSE HYPOTHESIS (confidence: high)

app-deployment is OOMKilling under memory pressure. The pod has restarted
6 times with termination reason OOMKilled. ML flagged the memory spike as
load-driven (no leak). Upstream api-gateway is healthy at current vs 7-day
baseline. This is a resource-allocation issue — the container's memory
limit is too low for its real working set.

Evidence:
- 6 restarts, last termination reason OOMKilled
- No ML memory-growth anomaly → leak_suspected=false (load-driven)
- Upstream api-gateway unchanged vs 7d baseline (15.13 ms, 41.26%) → healthy
- K8s events show tight Pulled/Created/Started/Killing/BackOff cycles;
  no deployments in the last 2h

Likely cause: memory limit insufficient for actual working set under load.

Recommended next steps:
1. Raise the app-deployment memory limit based on observed usage
2. Review application code for memory-optimization opportunities
3. Consider graceful degradation on high-load paths

Downstream impact: none identified from APM destination metrics.

The output above is what the alert looks like when you open it — not a link to a bunch of logs or a dashboard, but an answer.

The same workflow is accessible as an MCP tool from Claude Desktop, VS Code, or any MCP-compatible client. When a developer asks "why is checkout erroring?" from their IDE, the agent calls the workflow and returns the same structured output inline — same evidence, same root cause, without leaving the editor.

Here's an animated walkthrough of the workflow execution:

Observability Skill for Kubernetes investigations

We're also shipping a single, comprehensive investigation Skill (observability-k8s-investigation) that encodes the full diagnostic protocol for Kubernetes workload, node, and control-plane issues. It is an opinionated investigation methodology that includes the reasoning an experienced SRE applies instinctively but rarely writes down. You'll get this by keeping Kibana up to date, as it's baked into our AI Agent skills. It starts with governing principles that prevent the most common misdiagnoses:

• Absence of evidence is not evidence. If log queries return zero rows, report no_logs_available — don't infer a failure mode from empty results.
• OOMKilled does not mean memory leak by default. Compare current usage against a 7-day baseline before claiming a leak. The limit may simply be undersized.
• Average CPU metrics hide throttling. A pod can look healthy at 40–60% average utilization while being severely throttled at p99. Look at max and p95, not just average.
• Co-symptoms are not causes. Two services degrading simultaneously usually share an upstream cause. Only attribute causation when one service's degradation clearly precedes the other's and the delta is large.

From there, the Skill encodes a failure-mode taxonomy covering 16 distinct K8s failure patterns across workload, node, control-plane, autoscaling, and networking layers — from OOMKilled and CFS throttling through admission webhook blocks and StatefulSet split-brain. Each mode has a pivotal signal that identifies it and a corroboration checklist that confirms it.

The investigation flow follows a structured arc: orient (resolve the target pod, namespace, deployment), characterize (get restart count, termination reasons, utilization), classify (match against the taxonomy), corroborate (pull events, logs, APM, baseline comparisons), and synthesize (produce a root cause hypothesis at calibrated confidence — high, medium, or low — with explicit evidence and recommended next steps).

When two failure modes fit the evidence, the Skill names both and says which it believes is causal and why. When evidence is ambiguous, it says so. "Competing hypotheses are a valid output" is an explicit design principle — manufacturing false confidence is treated as a failure mode of the investigation itself.

Getting started

These capabilities build on the Kubernetes integration described in Part 1. Once you have dashboards and data collection running:

Step 1 — Enable investigation workflows (technical preview). Import the Kubernetes Crashloop Investigation Workflow from the Workflows page in Kibana, and optionally configure it to trigger on an alert rule.

Step 2 — Install the MCP App on an MCP-compatible client (technical preview). The MCP App for Observability repo can be found on GitHub (see the Releases page for downloads). When installing the app, don't forget to also install and enable the included skills. Access the Example MCP App's tools from your favorite agentic client — instructions are in the README at the GitHub link above.

Step 3 — Leverage the K8s Investigation Skill (technical preview). This one is a freebie if you're using Agent Builder, because it's baked into AI Agent Skills. The Skill teaches the agent when and how to call the underlying tools and workflows, ensuring consistent diagnostics in conversational contexts.

What's next

Investigation workflows diagnose what's broken in the services you're monitoring. The next question is harder: what about the services you're not monitoring?

We're thinking about topology-aware coverage intelligence — automatically discovering every workload deployed in your cluster via the Kubernetes API, cross-referencing against telemetry flowing into Elastic, and surfacing the gap. "You have 47 services. 11 have no distributed traces. Here's your riskiest blind spot." That capability is under consideration and will likely be the subject of a future post.

In parallel, we're extending workflows toward remediation — not just diagnosis but action: creating a case with the investigation summary attached, proposing a rollback for human approval, or scaling a workload to buy time while the root cause is addressed.

If you're running Kubernetes on Elastic today, tell us which investigation steps you repeat manually on every incident, which remediations you'd trust a workflow to propose, and which MCP tools we should build next. You can join the Elastic Community Discussion Here.

In an era where APIs serve as the backbone of modern applications, having the means to maintain visibility and control over these vital components is absolutely essential. In this blog post, we dive deep into the comprehensive observability solution offered by Elastic^®, ensuring real-time visibility, advanced analytics, and actionable insights, empowering you to fine-tune your API Gateway for optimal performance.

For application owners and developers, this integration stands as a beacon of empowerment. Elastic's meticulous orchestration of the seamless merging of metrics, logs, and traces, built upon the robust ELK Stack foundation, equips them with potent real-time monitoring and analysis tools. These tools facilitate precise performance optimization and swift issue resolution, all within a secure and dependable environment.

With Elastic's AWS API Gateway integration, application owners and developers unlock the capability to proactively identify and resolve problems, fine-tune resource utilization, and provide extraordinary digital experiences to their users.

Architecture

Why the AWS API Gateway integration matters

API Gateway now serves as the foundation of contemporary application development, simplifying the process of creating and overseeing APIs on a large scale. Yet, monitoring and troubleshooting these API endpoints can be challenging. With the new AWS API Gateway integration introduced by Elastic, you can gain the following:

• Unprecedented visibility: Monitor your API Gateway endpoints' performance, error rates, and usage metrics in real time. Get a comprehensive view of your APIs' health and performance.
• Log analysis: Dive deep into API Gateway logs with ease. Our integration enables you to collect and analyze logs for HTTP, REST, and Websocket API types, helping you troubleshoot issues and gain valuable insights.
• Rapid issue resolution: Identify and resolve issues in your API Gateway workflows faster than ever. Elastic Observability's powerful search and analytics tools help you pinpoint problems with ease.
• Alerting and notifications: Set up custom alerts based on API Gateway metrics and logs. Receive notifications when performance thresholds are breached, ensuring that you can take action promptly.
• Optimized costs: Visualize resource usage and performance metrics for your API Gateway deployments. Use these insights to optimize resource allocation and reduce operational costs.
• Custom dashboards: Create customized dashboards and visualizations tailored to your API Gateway monitoring needs. Stay in control with real-time data and actionable insights.
• Effortless integration: Seamlessly connect your AWS API Gateway to our observability solution. Our intuitive setup process ensures a smooth integration experience.
• Scalability: Whether you have a handful of APIs or a complex API Gateway landscape, our observability solution scales to meet your needs. Grow confidently as your API infrastructure expands.

How to get started

Getting started with the AWS API Gateway integration in Elastic Observability is seamless. Here's a quick overview of the steps:

Prerequisites and configurations

If you intend to follow the steps outlined in this blog post, there are a few prerequisites and configurations that you should have in place beforehand.

1. You will need an account on Elastic Cloud and a deployed stack and agent. Instructions for deploying a stack on AWS can be found here. This is necessary for AWS API Gateway logging and analysis.

2. You will also need an AWS account with the necessary permissions to pull data from AWS. Details on the required permissions can be found in our documentation.

3. You can monitor API execution by using CloudWatch, which collects and processes raw data from API Gateway into readable, near-real-time metrics and logs. Details on the required steps to enable logging can be found here.

Step 1. Create an account with Elastic

Create an account on Elastic Cloud by following the steps provided.

Step 2. Add integration

• Log in to your Elastic Cloud deployment.

• Click on Add integrations. You will be navigated to a catalog of supported integrations.

• Search and select AWS API Gateway.

Step 3. Configure integration

• Click on the Add AWS API Gateway button and provide the required details.
• If this is your first time adding an AWS integration, you’ll need to configure and enroll the Elastic Agent on an AWS instance.

• Then complete the “Configure integration” form, providing all the necessary information required for agents to collect the AWS API Gateway metrics and associated CloudWatch logs. Multiple AWS credential methods are supported, including access keys, temporary security credentials, and IAM role ARN. Please see the IAM security and access documentation for more details. You can choose to collect API Gateway metrics, API Gateway logs via S3, or API Gateway logs via CloudWatch.
• Click on the Save and continue button at the bottom of the page.

Step 4. Analyze and monitor

Explore the data using the out-of-the-box dashboards available for the integration. Select Discover from the Elastic Cloud top-level menu.

Or, create custom dashboards, set up alerts, and gain actionable insights into your API Gateway service performance.

Here are key monitoring metrics collected through this integration across Rest APIs, HTTP APIs, and Websocket APIs:

• 4XXError – The number of client-side errors captured in a given period
• 5XXError – The number of server-side errors captured in a given period
• CacheHitCount – The number of requests served from the API cache in a given period
• CacheMissCount – The number of requests served from the backend in a given period, when API caching is enabled
• Count – The total number of API requests in a given period
• IntegrationLatency – The time between when API Gateway relays a request to the backend and when it receives a response from the backend
• Latency – The time between when API Gateway receives a request from a client and when it returns a response to the client — the latency includes the integration latency and other API Gateway overhead
• DataProcessed – The amount of data processed in bytes
• ConnectCount – The number of messages sent to the $connect route integration
  MessageCount – The number of messages sent to the WebSocket API, either from or to the client

Conclusion

The native integration of AWS API Gateway into Elastic Observability marks a significant advancement in streamlining the monitoring and management of your APIs. With this integration, you gain access to a wealth of insights, real-time visibility, and powerful analytics tools, empowering you to optimize your API performance, enhance security, and troubleshoot with ease. Don't miss out on this opportunity to take your API management to the next level, ensuring your digital assets operate at their best, all while providing a seamless experience for your users. Embrace this integration, and stay at the forefront of API observability in the ever-evolving world of digital technology.

Visit our documentation to learn more about Elastic Observability and the AWS API Gateway integration, or contact our sales team to get started!

Start a free trial today

Start your own 7-day free trial by signing up via AWS Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on AWS around the world. Your AWS Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with AWS.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Cloud is becoming the de facto deployment option for today’s applications. Many cloud deployments choose to host their applications on AWS for the globally diverse set of regions it covers and the myriad of services (for faster development and innovation) available, as well as to drive operational and capital costs down. On AWS, development teams are finding additional value in migrating to Kubernetes on Amazon EKS, testing out the latest serverless options, and improving traditional, tiered applications with better services.

Elastic Observability offers 30 out-of-the-box integrations for AWS services with more to come.

A quick review highlighting some of the integrations and capabilities can be found in a previous post:

• Elastic and AWS: Seamlessly ingest logs and metrics into a unified platform with ready-to-use integrations.

Some additional posts on key AWS service integrations on Elastic are:

• APM (metrics, traces and logs) for serverless functions on AWS Lambda with Elastic
• Log ingestion from AWS Services into Elastic via serverless forwarder on Lambda
• Elastic’s Amazon S3 Storage Lens Integration: Simplify management, control costs, and reduce risk
• Ingest your container logs into Elastic Cloud with AWS FireLens

A full list of AWS integrations can be found in Elastic’s online documentation:

• Full list of AWS integrations

In addition to our native AWS integrations, Elastic Observability aggregates not only logs but also metrics for AWS services and the applications running on AWS compute services (EC2, Lambda, EKS/ECS/Fargate). All this data can be analyzed visually and more intuitively using Elastic’s advanced machine learning capabilities, which help detect performance issues and surface root causes before end users are affected.

For more details on how Elastic Observability provides application performance monitoring (APM) capabilities such as service maps, tracing, dependencies, and ML based metrics correlations:

• APM correlations in Elastic Observability: Automatically identifying probable causes of slow or failed transactions
• Elastic and AWS: Get the most value from your data sets

That’s right, Elastic offers metrics ingest, aggregation, and analysis for AWS services and applications on AWS compute services (EC2, Lambda, EKS/ECS/Fargate). Elastic is more than logs — it offers a unified observability solution for AWS environments.

In this blog, I’ll review how Elastic Observability can monitor metrics for a simple AWS application running on AWS services which include:

• AWS EC2
• AWS ELB
• AWS RDS (AuroraDB)
• AWS NAT Gateways

As you will see, once the integration is installed, metrics will arrive instantly and you can immediately start reviewing metrics.

Prerequisites and config

If you plan on following this blog, here are some of the components and details we used to set up this demonstration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).
• Ensure you have an AWS account with permissions to pull the necessary data from AWS. See details in our documentation.
• We used AWS’s three tier app and installed it as instructed in git.
• We’ll walk through installing the general Elastic AWS Integration, which covers the four services we want to collect metrics for.
  (Full list of services supported by the Elastic AWS Integration)
• We will not cover application monitoring given other blogs cover application AWS monitoring (metrics, logs, and tracing). Instead we will focus on how AWS services can be easily monitored.
• In order to see metrics, you will need to load the application. We’ve also created a playwright script to drive traffic to the application.

Three tier application overview

Before we dive into the Elastic configuration, let's review what we are monitoring. If you follow the instructions for aws-three-tier-web-architecture-workshop, you will have the following deployed.

What’s deployed:

• 1 VPC with 6 subnets
• 2 AZs
• 2 web servers per AZ
• 2 application servers per AZ
• 1 External facing application load balancer
• 1 Internal facing application load balancer
• 2 NAT gateways to manage traffic to the application layer
• 1 Internet gateway
• 1 RDS Aurora DB with a read replica

At the end of the blog, we will also provide a Playwright script to implement to load this app. This will help drive metrics to “light up” the dashboards.

Setting it all up

Let’s walk through the details of how to get the application, AWS integration on Elastic, and what gets ingested.

Step 0: Load up the AWS Three Tier application and get your credentials

Follow the instructions listed out in AWS’s Three Tier app and instructions in the workshop link on git. The workshop is listed here.

Once you’ve installed the app, get credentials from AWS. This will be needed for Elastic’s AWS integration.

There are several options for credentials:

• Use access keys directly
• Use temporary security credentials
• Use a shared credentials file
• Use an IAM role Amazon Resource Name (ARN)

For more details on specifics around necessary credentials and permissions.

Step 1: Get an account on Elastic Cloud

Follow the instructions to get started on Elastic Cloud.

Step 2: Install the Elastic AWS integration

Navigate to the AWS integration on Elastic.

Select Add AWS integration.

This is where you will add your credentials and it will be stored as a policy in Elastic. This policy will be used as part of the install for the agent in the next step.

As you can see, the general Elastic AWS Integration will collect a significant amount of data from 30 AWS services. If you don’t want to install this general Elastic AWS Integration, you can select individual integrations to install.

Step 3: Install the Elastic Agent with AWS integration

Now that you have created an integration policy, navigate to the Fleet section under Management in Elastic.

Select the name of the policy you created in the last step.

Follow step 3 in the instructions in the Add agent window. This will require you to:

1: Bring up an EC2 instance

• t2.medium is minimum
• Linux - your choice of which
• Ensure you allow for Open reservation on the EC2 instance when you Launch it

2: Log in to the instance and run the commands under Linux Tar tab (below is an example)

curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.5.0-linux-x86_64.tar.gz
tar xzvf elastic-agent-8.5.0-linux-x86_64.tar.gz
cd elastic-agent-8.5.0-linux-x86_64
sudo ./elastic-agent install --url=https://37845638732625692c8ee914d88951dd96.fleet.us-central1.gcp.cloud.es.io:443 --enrollment-token=jkhfglkuwyvrquevuytqoeiyri

Step 4: Run traffic against the application

While getting the application running is fairly easy, there is nothing to monitor or observe with Elastic unless you add a load on the application.

Here is a simple script you can also run using Playwright to add traffic to the website for the AWS three tier application:

import { test, expect } from "@playwright/test";

test("homepage for AWS Threetierapp", async ({ page }) => {
  await page.goto(
    "http://web-tier-external-lb-1897463036.us-west-1.elb.amazonaws.com/#/db"
  );

  await page.fill(
    "#transactions > tbody > tr > td:nth-child(2) > input",
    (Math.random() * 100).toString()
  );
  await page.fill(
    "#transactions > tbody > tr > td:nth-child(3) > input",
    (Math.random() * 100).toString()
  );
  await page.waitForTimeout(1000);
  await page.click(
    "#transactions > tbody > tr:nth-child(2) > td:nth-child(1) > input[type=button]"
  );
  await page.waitForTimeout(4000);
});

This script will launch three browsers, but you can limit this load to one browser in playwright.config.ts file.

For this exercise, we ran this traffic for approximately five hours with an interval of five minutes while testing the website.

Step 5: Go to AWS dashboards

Now that your Elastic Agent is running, you can go to the related AWS dashboards to view what’s being ingested.

To search for the AWS Integration dashboards, simply search for them in the Elastic search bar. The relevant ones for this blog are:

• [Metrics AWS] EC2 Overview
• [Metrics AWS] ELB Overview
• [Metrics AWS] RDS Overview
• [Metrics AWS] NAT Gateway

Let's see what comes up!

All of these dashboards are out-of-the-box and for all the following images, we’ve narrowed the views to only the relevant items from our app.

Across all dashboards, we’ve limited the timeframe to when we ran the traffic generator.

Once we filtered for our 4 EC2 instances (2 web servers and 2 application servers), we can see the following:

1: All 4 instances are up and running with no failures in status checks.

2: We see the average CPU utilization across the timeframe and nothing looks abnormal.

3: We see the network bytes flow in and out, aggregating over time as the database is loaded with rows.

While this exercise shows a small portion of the metrics that can be viewed, more are available from AWS EC2. The metrics listed on AWS documentation are all available, including the dimensions to help narrow the search for specific instances, etc.

For the ELB dashboard, we filter for our 2 load balancers (external web load balancer and internal application load balancer).

With the out-of-the-box dashboard, you can see application ELB-specific metrics. A good portion of the application ELB specific metrics listed in AWS Docs are available to add graphs for.

For our two load balancers, we can see:

1: Both the hosts (EC2 instances connected to the ELBs) are healthy.

2: Load Balancer Capacity Units (how much you are using) and request counts both went up as expected during the traffic generation time frame.

3: We picked to show 4XX and 2XX counts. 4XX will help identify issues with the application or connectivity with the application servers.

For AuroraDB, which is deployed in RDS, we’ve filtered for just the primary and secondary instances of Aurora on the dashboard.

Just as with EC2, ELB, most RDS metrics from Cloudwatch are also available to create new charts and graphs. In this dashboard, we’ve narrowed it down to showing:

1: Insert throughput & Select throughput

2: Write latency

3: CPU usage

4: General number of connections during the timeframe

We filtered to look only at our 2 NAT instances which are fronting the application servers. As with the other dashboards, other metrics are available to build graphs and /charts as needed.

For the NAT dashboard we can see the following:

1: The NAT Gateways are doing well due to no packet drops

2: An expected number of active connections from the web server

3: Fairly normal set of metrics for bytes in and out

Congratulations, you have now started monitoring metrics from key AWS services for your application!

What to monitor on AWS next?

Add logs from AWS Services

Now that metrics are being monitored, you can also now add logging. There are several options for ingesting logs.

1. The AWS Integration in the Elastic Agent has logs setting. Just ensure you turn on what you wish to receive. Let’s ingest the Aurora Logs from RDS. In the Elastic agent policy, we simply turn on Collect logs from CloudWatch (see below). Next, update the agent through the Fleet management UI.

2. You can install the Lambda logs forwarder. This option will pull logs from multiple locations. See the architecture diagram below.

A review of this option is also found in the following blog.

Analyze your data with Elastic Machine Learning

Once metrics and logs (or either one) are in Elastic, start analyzing your data through Elastic’s ML capabilities. A great review of these features can be found here:

• Correlating APM Telemetry to determine root causes in transactions
• Introduction to Elastic Machine Learning

And there are many more videos and blogs on Elastic’s Blog.

Conclusion: Monitoring AWS service metrics with Elastic Observability is easy!

I hope you’ve gotten an appreciation for how Elastic Observability can help you monitor AWS service metrics, here’s a quick recap of lessons and what you learned:

• Elastic Observability supports ingest and analysis of AWS service metrics
• It’s easy to set up ingest from AWS Services via the Elastic Agent
• Elastic Observability has multiple out-of-the-box (OOTB) AWS service dashboards you can use to preliminarily review information, then modify for your needs
• 30+ AWS services are supported as part of AWS Integration on Elastic Observability, with more services being added regularly
• As noted in related blogs, you can analyze your AWS service metrics with Elastic’s machine learning capabilities

Start your own 7-day free trial by signing up via AWS Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on AWS around the world. Your AWS Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with AWS.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Elastic can make it easier for organizations to transform data into actionable insights and stop threats quickly with unified visibility across your environment — so mission-critical applications can keep running smoothly no matter what. From a free trial and fast deployment to sending logs to Elastic securely and frictionlessly, all you need to do is point and click to capture, store, and search data from your AWS services.

Monitoring EMR via Elastic Observability

In this article, we will delve into the following key aspects:

• Enabling EMR cluster metrics for Elastic integration: Learn the intricacies of configuring an EMR cluster to emit metrics that Elastic can effectively extract, paving the way for insightful analysis.
• Harnessing Kibana ^® dashboards for EMR workload analysis: Discover the potential of utilizing Kibana dashboards to dissect metrics related to an EMR workload. By gaining a deeper understanding, we open the doors to optimization opportunities.

Key benefits of AWS EMR integration

• Comprehensive monitoring: Monitor the health and performance of your EMR clusters in real time. Track metrics related to cluster status and utilization, node status, IO, and many others, allowing you to identify bottlenecks and optimize your data processing.
• Log analysis: Dive deep into EMR logs with ease. Our integration enables you to collect and analyze logs from your clusters, helping you troubleshoot issues and gain valuable insights.
• Cost optimization: Understand the cost implications of your EMR clusters. By monitoring resource utilization, you can identify opportunities to optimize your cluster configurations and reduce costs.
• Alerting and notifications: Set up custom alerts based on EMR metrics and logs. Receive notifications when performance thresholds are breached, ensuring that you can take action promptly.
• Seamless integration: Our integration is designed for ease of use. Getting started is simple, and you can start monitoring your EMR clusters quickly.

Accompanying these discussions is an illustrative solution architecture diagram, providing a visual representation of the intricacies and interactions within the proposed solution.

How to get started

Getting started with AWS EMR integration in Observability is easy. Here's a quick overview of the steps:

Prerequisites and configurations

If you intend to follow the steps outlined in this blog post, there are a few prerequisites and configurations that you should have in place beforehand.

1. You will need an account on Elastic Cloud and a deployed stack and agent. Instructions for deploying a stack on AWS can be found here. This is necessary for AWS EMR logging and analysis.

2. You will also need an AWS account with the necessary permissions to pull data from AWS. Details on the required permissions can be found in our documentation.

3. Finally, be sure to turn on EMR monitoring for the EMR cluster when you deploy the cluster.

Step 1: Create an account with Elastic

Create an account on Elastic Cloud by following the steps provided.

Step 2: Add integration

1. Log in to your Elastic Cloud on AWS deployment.

2. Click on Add Integration. You will be navigated to a catalog of supported integrations.

3. Search and select Amazon EMR.

Step 3: Configure integration

1. Click on the Add Amazon EMR button and provide the required details.

2. Provide the required access credentials to connect to your EMR instance.

3. You can choose to collect EMR metrics, EMR logs via S3, or EMR logs via Cloudwatch.

4. Click on the Save and continue button at the bottom of the page.

Step 4: Analyze and monitor

Explore the data using the out-of-the-box dashboards available for the integration. Select Discover from the Elastic Cloud top-level menu.

Or, create custom dashboards, set up alerts, and gain actionable insights into your EMR clusters' performance.

This integration streamlines the collection of vital metrics and logs, including Cluster Status, Node Status, IO, and Cluster Capacity. Some metrics gathered include:

• IsIdle: Indicates that a cluster is no longer performing work, but is still alive and accruing charges
• ContainerAllocated: The number of resource containers allocated by the ResourceManager
• ContainerReserved: The number of containers reserved
• CoreNodesRunning: The number of core nodes working
• CoreNodesPending: The number of core nodes waiting to be assigned
• MRActiveNodes: The number of nodes presently running MapReduce tasks or jobs
• MRLostNodes: The number of nodes allocated to MapReduce that have been marked in a LOST state
• HDFSUtilization: The percentage of HDFS storage currently used
• HDFSBytesRead/Written: The number of bytes read/written from HDFS (This metric aggregates MapReduce jobs only, and does not apply for other workloads on Amazon EMR.)
• TotalUnitsRequested/TotalNodesRequested/TotalVCPURequested: The target total number of units/nodes/vCPUs in a cluster as determined by managed scaling

Conclusion

Elastic is committed to fulfilling all your observability requirements, offering an effortless experience. Our integrations are designed to simplify the process of ingesting telemetry data, granting you convenient access to critical information for monitoring, analytics, and observability. The native AWS EMR integration underscores our dedication to delivering seamless solutions for your data needs. With this integration, you'll find the confidence to monitor, analyze, and optimize your EMR clusters, opening up exciting opportunities for your data-driven initiatives.

Start a free trial today

Start your own 7-day free trial by signing up via AWS Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on AWS around the world. Your AWS Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with AWS.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Collecting JMX metrics with OpenTelemetry can be done in two main ways depending on your environment, requirements and constraints:

• from inside the JVM with the OpenTelemetry Instrumentation Java agent (or EDOT Java)
• from outside the JVM with the jmx-scraper.

Thorough this article, we will use the term "Java agent" to refer to the OpenTelemetry Java instrumentation agent, this also applies to the Elastic own distribution (EDOT Java) which is based on it and provides the same features.

This walkthrough uses a Tomcat server as the target and shows how to validate which metrics are emitted with the logging exporter.

The configuration examples in this article use Java system properties that must be passed using -D flags in the JVM startup command, equivalent environment variables can also be used for configuration.

Prerequisites

• A local Tomcat install (or any JVM app you can start with custom JVM flags)
• Java 8+ on the host, the Tomcat version used might require a more recent version though.
• An OpenTelemetry Collector endpoint if you want to ship metrics beyond local logging

Choosing between the Java agent and jmx-scraper

Use the Java agent (or EDOT Java) when you can modify JVM startup flags and want in-process collection with full context from the running application: this allows to capture traces, logs and metrics with a single tool deployment.

Use jmx-scraper when you cannot install an agent on the JVM or prefer out-of-process collection from a separate host. This requires the JVM and the network to be configured for remote JMX access and also dealing with authentication and credentials.

Both approaches rely on the same JMX metric mappings and can use the logging exporter for validation and then use OTLP to send metrics to the collector / an OTLP endpoint.

Option 1: Collect JMX metrics inside the JVM with the Java agent

OpenTelemetry Java instrumentation ships with a curated set of JMX metric mappings. For Tomcat, you just need to enable the Java agent and set otel.jmx.target.system=tomcat.

Step 1 - Download the OpenTelemetry Java agent

The agent is downloaded in /opt/otel but you can choose any location on the host. Make sure the path is consistent with the -javaagent flag in the next step.

mkdir -p /opt/otel
curl -L -o /opt/otel/opentelemetry-javaagent.jar \
  https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar

Step 2 - Configure Tomcat with bin/setenv.sh

Create or update bin/setenv.sh so Tomcat launches with the agent and JMX target system enabled.

#!/bin/bash
export CATALINA_OPTS="$CATALINA_OPTS \
  -javaagent:/opt/otel/opentelemetry-javaagent.jar \
  -Dotel.service.name=tomcat-demo \
  -Dotel.metrics.exporter=otlp,logging \
  -Dotel.jmx.target.system=tomcat"

This will configure the agent to log metrics (using the logging exporter) in addition to sending them to the Collector.

Step 3 - Validate the emitted metrics

Start Tomcat and watch stdout.

./bin/catalina.sh run

By defaults metrics are sampled and emitted every minute, so you might have to wait a bit for the metrics to be logged. If needed, you can use otel.metric.export.interval configuration to increase or reduce the frequency.

You should see logging exporter output with JVM and Tomcat metrics. Look for lines containing the LoggingMetricExporter class name.

INFO io.opentelemetry.exporter.logging.LoggingMetricExporter - MetricData{name=tomcat.threadpool.currentThreadsBusy, ...}
INFO io.opentelemetry.exporter.logging.LoggingMetricExporter - MetricData{name=jvm.memory.used, ...}

Step 4 - Send metrics to a Collector

Once metric capture is validated, you should be ready to send metrics to a collector.

You will have to:

• remove the logging exporter as it's no longer necessary for production
• configure the OTLP endpoint (otel.exporter.otlp.endpoint) and headers (otel.exporter.otlp.headers) if needed

The bin/setenv.sh file should be modified to look like this:

#!/bin/bash
export CATALINA_OPTS="$CATALINA_OPTS \
  -javaagent:/opt/otel/opentelemetry-javaagent.jar \
  -Dotel.service.name=tomcat-demo \
  -Dotel.jmx.target.system=tomcat \
  -Dotel.exporter.otlp.endpoint=https://your-collector:4317 \
  -Dotel.exporter.otlp.headers=Authorization=Bearer <your-token>"

When using the Java agent, the JVM metrics are automatically captured by the runtime-telemetry module, it is thus not necessary to include jvm in the otel.jmx.target.system configuration option.

Option 2: Collect JMX metrics from outside the JVM with jmx-scraper

When you cannot install an agent in the JVM or if only metrics are required, jmx-scraper lets you query JMX remotely and export metrics to an OTLP endpoint.

Step 1 - Enable remote JMX on Tomcat

Add JMX remote options to bin/setenv.sh and create access/password files.

Warning: This uses trivial credentials and disables SSL. Do not use this configuration in production.

mkdir -p /opt/jmx
cat <<EOF > ${CATALINA_HOME}/jmxremote.access
monitorRole readonly
EOF

cat <<EOF > ${CATALINA_HOME}/jmxremote.password
monitorRole monitorPass
EOF

chmod 600 ${CATALINA_HOME}/jmxremote.password

export CATALINA_OPTS="$CATALINA_OPTS \
  -Dcom.sun.management.jmxremote \
  -Dcom.sun.management.jmxremote.port=9010 \
  -Dcom.sun.management.jmxremote.rmi.port=9010 \
  -Dcom.sun.management.jmxremote.authenticate=true \
  -Dcom.sun.management.jmxremote.ssl=false \
  -Dcom.sun.management.jmxremote.access.file=${CATALINA_HOME}/jmxremote.access \
  -Dcom.sun.management.jmxremote.password.file=${CATALINA_HOME}/jmxremote.password \
  -Djava.rmi.server.hostname=127.0.0.1"

Step 2 - Download jmx-scraper

The jmx-scraper is downloaded in /opt/otel but you can choose any location on the host.

mkdir -p /opt/otel
curl -L -o /opt/otel/opentelemetry-jmx-scraper.jar \
  https://github.com/open-telemetry/opentelemetry-java-contrib/releases/latest/download/opentelemetry-jmx-scraper.jar

Step 3 - Check the JMX connection

Run jmx-scraper with credentials from previous step to confirm it can reach Tomcat. If the credentials are wrong, you will see authentication errors.

java -jar /opt/otel/opentelemetry-jmx-scraper.jar \
  -Dotel.jmx.service.url=service:jmx:rmi:///jndi/rmi://localhost:9010/jmxrmi
  -Dotel.jmx.username=monitorRole \
  -Dotel.jmx.password=monitorPass \
  -Dotel.jmx.target.system=tomcat \
  -test

You should get in the standard output:

• JMX connection test OK if the connection and authentication is successful
• JMX connection test ERROR otherwise

Step 4 - Validate the emitted metrics

Using the logging exporter allows to inspect metrics and attributes before sending them to a collector.

In order to capture both Tomcat and JVM metrics, it is required to set otel.jmx.target.system to tomcat,jvm.

java -jar /opt/otel/opentelemetry-jmx-scraper.jar \
  -Dotel.jmx.service.url=service:jmx:rmi:///jndi/rmi://localhost:9010/jmxrmi
  -Dotel.jmx.username=monitorRole \
  -Dotel.jmx.password=monitorPass \
  -Dotel.jmx.target.system=tomcat,jvm \
  -Dotel.metrics.exporter=logging

Step 5 - Send metrics to a Collector

After validation, to send metrics to an OTLP endpoint, you will have to:

• remove the -Dotel.metrics.exporter to restore the otlp default value.
• configure the OTLP endpoint (otel.exporter.otlp.endpoint) and headers (otel.exporter.otlp.headers) if needed

java -jar /opt/otel/opentelemetry-jmx-scraper.jar \
  -Dotel.jmx.service.url=service:jmx:rmi:///jndi/rmi://localhost:9010/jmxrmi
  -Dotel.jmx.username=monitorRole \
  -Dotel.jmx.password=monitorPass \
  -Dotel.jmx.target.system=tomcat,jvm \
  -Dotel.exporter.otlp.endpoint=https://your-collector:4317
  -Dotel.exporter.otlp.headers="Authorization=Bearer <your-token>"

Customizing the JMX Metrics Collection

Once the built-in Tomcat and JVM mappings are flowing, you can add custom rules with otel.jmx.config. Create a YAML file and pass its path alongside otel.jmx.target.system.

For example, the following custom.yaml file allows to capture the custom.jvm.thread.count metric from the java.lang:type=Threading MBean:

---
rules:
  - bean: "java.lang:type=Threading"
    mapping:
      ThreadCount:
        metric: custom.jvm.thread.count
        type: gauge
        unit: "{thread}"
        desc: Current number of live threads.

For complete reference on the configuration format and syntax, refer to jmx-metrics module in Opentelemetry Java instrumentation.

This custom configuration can be used both with jmx-scraper and Java agent, both support the otel.jmx.config configuration option, for example with jmx-scraper:

java -jar /opt/otel/opentelemetry-jmx-scraper.jar \
  -Dotel.jmx.service.url=service:jmx:rmi:///jndi/rmi://localhost:9010/jmxrmi
  -Dotel.jmx.username=monitorRole \
  -Dotel.jmx.password=monitorPass \
  -Dotel.jmx.target.system=tomcat,jvm \
  otel.jmx.config=/opt/otel/jmx/custom.yaml

You can pass multiple custom files as a comma-separated list to otel.jmx.config when you need to organize metrics by team or component.

Using the JMX Metrics in Kibana

Once you have collected the JMX metrics using one of the approaches described in this article, you can start using them in Kibana. You can build custom dashboards and visualizations to explore and analyze the metrics, create custom alerts on top of them or build MCP tools and AI Agents to use them in your agentic workflows.

Here is an example of how you can use the JMX metrics in Kibana through ES|QL:

TS metrics*
| WHERE telemetry.sdk.language == "java"
| WHERE service.name == ?instance
| STATS
    request_rate = SUM(RATE(tomcat.request.count))
  BY Time = BUCKET(@timestamp, 100, ?_tstart, ?_tend)

You can use the native metric and dimension names of the JMX metrics to build your queries. With the TS command you get first-class support for time series aggregation functions and dimensions on your metrics. This kind of queries constitute the building blocks for your dashboards, alerts, workflows and AI agent tools.

Here is an example of a dashboard that visualizes the typical JMX metrics for Apache Tomcat:

Conclusion

In this article, we have seen how to collect JMX metrics with OpenTelemetry using the Java agent or jmx-scraper. We have also seen how to use the JMX metrics in Kibana through ES|QL to build custom dashboards, alerts, workflows and AI agent tools.

This is just the beginning of what you can do with the JMX metrics and Elastic Observability. Try it out yourself and explore the full potential of your JMX metrics when combined with powerful features provided by the Elastic Observability platform.

AWS Fargate is a serverless pay-as-you-go engine used for Amazon Elastic Container Service (ECS) to run Docker containers without having to manage servers or clusters. The goal of Fargate is to containerize your application and specify the OS, CPU and memory, networking, and IAM policies needed for launch. Additionally, AWS Fargate can be used with Elastic Kubernetes Service (EKS) in a similar manner.

Although the provisioning of servers would be handled by a third party, the need to understand the health and performance of containers within your serverless environment becomes even more vital in identifying root causes and system interruptions. Serverless still requires observability. Elastic Observability can provide observability for not only AWS ECS with Fargate, as we will discuss in this blog, but also for a number of AWS services (EC2, RDS, ELB, etc). See our previous blog on managing an EC2-based application with Elastic Observability.

Gaining full visibility with Elastic Observability

Elastic Observability is governed by the three pillars involved in creating full visibility within a system: logs, metrics, and traces. Logs list all the events that have taken place in the system. Metrics keep track of data that will tell you if the system is down, like response time, CPU usage, memory usage, and latency. Traces give a good indication of the performance of your system based on the execution of requests.

These pillars by themselves offer some insight, but combining them allows for you to see the full scope of your system and how it handles increases in load or traffic over time. Connecting Elastic Observability to your serverless environment will help you deal with outages quicker and perform root cause analysis to prevent any future problems.

In this article, we’ll guide you through how to install the Elastic Agent with the AWS Fargate integration as a sidecar container to send host metrics and logs to Elastic Observability.

Prerequisites:

• AWS account with AWS CLI configured
• GitHub account
• Elastic Cloud account
• An app running on a container in AWS

This tutorial is divided into two parts:

1. Set up the Fleet server to be used by the sidecar container in AWS.
2. Create the sidecar container in AWS Fargate to send data back to Elastic Observability.

Part I: Set up the Fleet server

First, let’s log in to Elastic Cloud.

You can either create a new deployment or use an existing one.

From the Home page, use the side panel to scroll to Management > Fleet > Agent policies. Click Add policy.

Click Create agent policy. Here we’ll create a policy to attach to the Fleet agent.

Give the policy a name and save changes.

Click Create agent policy. You should see the agent policy AWS Fargate in the list of policies.

Now that we have an agent policy, let’s add the integration to collect logs and metrics from the host. Click on AWS Fargate -> Add integration.

We’ll be adding to the policy AWS to collect overall AWS metrics and AWS Fargate to collect metrics from this integration. You can find each one by typing them in the search bar.

Once you click on the integration, it will take you to its landing page, where you can add it to the policy.

For the AWS integration, the only collection settings that we will configure are Collect billing metrics, Collect logs from CloudWatch, Collect metrics from CloudWatch, Collect ECS metrics, and Collect Usage metrics. Everything else can be left disabled.

Another thing to keep in mind when using this integration is the set of permissions required to collect data from AWS. This can be found on the AWS integration page under AWS permissions. Take note of these permissions, as we will use them to create an IAM policy.

Next, we will add the AWS Fargate integration, which doesn’t require further configuration settings.

Now that we have created the agent policy and attached the proper integrations, let’s create the agent that will implement the policy. Navigate back to the main Fleet page and click Add agent.

Since we’ll be connecting to AWS Fargate through ECS, the host type should be set to this value. All the other default values can stay the same.

Lastly, let’s create the enrollment token and attach the agent policy. This will enable AWS ECS Fargate to access Elastic and send data.

Once created, you should be able to see policy name, secret, and agent policy listed.

We’ll be using our Fleet credentials in the next step to send data to Elastic from AWS Fargate.

Part II: Send data to Elastic Observability

It’s time to create our ECS Cluster, Service, and task definition in order to start running the container.

Log in to your AWS account and navigate to ECS.

We’ll start by creating the cluster.

Add a name to the Cluster. And for subnets, only select the first two for us-east-1a and us-eastlb.

For the sake of the demo, we’ll keep the rest of the options set to default. Click Create.

We should see the cluster we created listed below.

Now that we’ve created our cluster to host our container, we want to create a task definition that will be used to set up our container. But before we do this, we will need to create a task role with an associated policy. This task role will allow for AWS metrics to be sent from AWS to the Elastic Agent.

Navigate to IAM in AWS.

Go to Policies -> Create policy.

Now we will reference the AWS permissions from the Fleet AWS integration page and use them to configure the policy. In addition to these permissions, we will also add the GetAtuhenticationToken action for ECR.

You can configure each one using the visual editor.

Or, use the JSON option. Don’t forget to replace the <account_id> with your own.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "sqs:DeleteMessage",
        "sqs:ChangeMessageVisibility",
        "sqs:ReceiveMessage",
        "ecr:GetDownloadUrlForLayer",
        "ecr:UploadLayerPart",
        "ecr:PutImage",
        "sts:AssumeRole",
        "rds:ListTagsForResource",
        "ecr:BatchGetImage",
        "ecr:CompleteLayerUpload",
        "rds:DescribeDBInstances",
        "logs:FilterLogEvents",
        "ecr:InitiateLayerUpload",
        "ecr:BatchCheckLayerAvailability"
      ],
      "Resource": [
        "arn:aws:iam::<account_id>:role/*",
        "arn:aws:logs:*:<account_id>:log-group:*",
        "arn:aws:sqs:*:<account_id>:*",
        "arn:aws:ecr:*:<account_id>:repository/*",
        "arn:aws:rds:*:<account_id>:target-group:*",
        "arn:aws:rds:*:<account_id>:subgrp:*",
        "arn:aws:rds:*:<account_id>:pg:*",
        "arn:aws:rds:*:<account_id>:ri:*",
        "arn:aws:rds:*:<account_id>:cluster-snapshot:*",
        "arn:aws:rds:*:<account_id>:cev:*/*/*",
        "arn:aws:rds:*:<account_id>:og:*",
        "arn:aws:rds:*:<account_id>:db:*",
        "arn:aws:rds:*:<account_id>:es:*",
        "arn:aws:rds:*:<account_id>:db-proxy-endpoint:*",
        "arn:aws:rds:*:<account_id>:secgrp:*",
        "arn:aws:rds:*:<account_id>:cluster:*",
        "arn:aws:rds:*:<account_id>:cluster-pg:*",
        "arn:aws:rds:*:<account_id>:cluster-endpoint:*",
        "arn:aws:rds:*:<account_id>:db-proxy:*",
        "arn:aws:rds:*:<account_id>:snapshot:*"
      ]
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": [
        "sqs:ListQueues",
        "organizations:ListAccounts",
        "ec2:DescribeInstances",
        "tag:GetResources",
        "cloudwatch:GetMetricData",
        "ec2:DescribeRegions",
        "iam:ListAccountAliases",
        "sns:ListTopics",
        "sts:GetCallerIdentity",
        "cloudwatch:ListMetrics"
      ],
      "Resource": "*"
    },
    {
      "Sid": "VisualEditor2",
      "Effect": "Allow",
      "Action": "ecr:GetAuthorizationToken",
      "Resource": "arn:aws:ecr:*:<account_id>:repository/*"
    }
  ]
}

Review your changes.

Now let’s attach this policy to a role. Navigate to IAM -> Roles. Click Create role.

Select AWS service as Trusted entity type and select EC2 as Use case. Click Next.

Under permissions policies, select the policy we just created, as well as CloudWatchLogsFullAccess and AmazonEC2ContainerRegistryFullAccess. Click Next.

Give the task role a name and description.

Click Create role.

Now it’s time to create the task definition. Navigate to ECS -> Task definitions. Click Create new task definition.

Let’s give this task definition a name.

After giving the task definition a name, you’ll add the Fleet credentials to the container section, which you can obtain from the Enrollment Tokens section of the Fleet section in Elastic Cloud. This allows us to host the Elastic Agent on the ECS container as a sidecar and send data to Elastic using Fleet credentials.

• Container name: elastic-agent-container

• Image: docker.elastic.co/beats/elastic-agent:8.19.13

Now let’s add the environment variables:

• FLEET_ENROLL: yes

• FLEET_ENROLLMENT_TOKEN: <enrollment-token>

• FLEET_URL: <fleet-server-url>

For the sake of the demo, leave Environment, Monitoring, Storage, and Tags as default values. Now we will need to create a second container to run the image for the golang app stored in ECR. Click Add more containers.

For Environment, we will reserve 1 vCPU and 3 GB of memory. Under Task role, search for the role we created that uses the IAM policy.

Review the changes, then click Create.

You should see your new task definition included in the list.

The final step is to create the service that will connect directly to the fleet server.
Navigate to the cluster you created and click Create under the Service tab.

Let’s get our service environment configured.

Set up the deployment configuration. Here you should provide the name of the task definition you created in the previous step. Also, provide the service with a unique name. Set the number of desired tasks to 2 instead of 1.

Click Create. Now your service is running two tasks in your cluster using the task definition you provided.

To recap, we set up a Fleet server in Elastic Cloud to receive AWS Fargate data. We then created our AWS Fargate cluster task definition with the Fleet credentials implemented within the container. Lastly, we created the service to send data about our host to Elastic.

Now let’s verify our Elastic Agent is healthy and properly receiving data from AWS Fargate.

We can also view a better breakdown of our agent on the Observability Overview page.

If we drill down to hosts, by clicking on host name we should be able to see more granular data. For instance, we can see the CPU Usage of the Elastic Agent itself that is deployed in our AWS Fargate environment.

Lastly, we can view the AWS Fargate dashboard generated using the data collected by our Elastic Agent. This is an out-of-the-box dashboard that can also be customized based on the data you would like to visualize.

As you can see in the dashboard we’re able to filter based on running tasks, as well as see a list of containers running in our environment. Something else that could be useful to show is the CPU usage per cluster as shown under CPU Utilization per Cluster.

The dashboard can pull data from different sources and in this case shows data for both AWS Fargate and the greater ECS cluster. The two containers at the bottom display the CPU and memory usage directly from ECS.

Conclusion

In this article, we showed how to send data from AWS Fargate to Elastic Observability using the Elastic Agent and Fleet. Serverless architectures are quickly becoming industry standard in offloading the management of servers to third parties. However, this does not alleviate the responsibility of operations engineers to manage the data generated within these environments. Elastic Observability provides a way to not only ingest the data from serverless architectures, but also establish a roadmap to address future problems.

Start your own 7-day free trial by signing up via AWS Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on AWS around the world. Your AWS Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with AWS.

More resources on serverless and observability and AWS:

• Analyze your AWS application’s service metrics on Elastic Observability (EC2, ELB, RDS, and NAT)
• Get visibility into AWS Lambda serverless functions with Elastic Observability
• Trace-based testing with Elastic APM and Tracetest
• Sending AWS logs into Elastic via AWS Firehose

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Elastic's Agent skills are open source packages that give your AI coding agent native Elastic expertise. If you're already using Elastic Agent Builder, you get AI agents that work natively with your Observability data. The Elastic Agent Skills deliver native platform expertise directly to your AI coding agent, so you can stop debugging AI-generated errors and start shipping production-ready code with the full depth of Elastic.

Skills can be used for specialized tasks across the Elastic stack — Elasticsearch, Kibana, Elastic Security, Elastic Observability, and more. Each skill lives in its own folder with a SKILL.md file containing metadata and instructions the agent follows.

Observability is releasing five skills that together cover the core workflows SREs and developers perform daily.Running Elastic Observability today involves a wide surface area: configuring OpenTelemetry instrumentation, writing ES|QL queries to search logs and metrics, defining SLOs with the correct indicator types and equation syntax, tand stitching together service health from multiple signals. Each of these tasks requires domain expertise and familiarity with specific APIs, index patterns, and Kibana workflows. For teams managing dozens of services across multiple environments, this is repetitive, error-prone, and time-consuming.

This article walks through the current Observability skill set, shows an end-to-end workflow, and highlights where these skills are useful in day-to-day operations.

Why this matters for observability teams

Modern observability work is usually ad hoc and cross-cutting. In one hour, you may instrument a new service, inspect logs for an incident, check error-budget status, and validate service health across several signals.

Each step often needs different APIs, index patterns, and Kibana workflows. Agent Skills package this task knowledge into reusable units so an agent can execute these steps consistently.

The observability skills

The observability set currently focuses on five connected workflows:

1. Instrument applications Adds the Elastic Distributions of OpenTelemetry to Python, Java, or .NET services (tracing, metrics, logs) or helps migrate from the classic Elastic APM agents to EDOT, with correct OTLP endpoints and configuration
2. Search logs Provides visibility into Elastic Streams — the data routing and processing layer for observability data.
3. Manage SLOs Creates and manages Service-Level Objectives in Elastic Observability via the Kibana API — from data exploration through SLO definition, creation, and lifecycle management.
4. Assess service health Provides a unified view of service health by combining signals from APM, infrastructure metrics, logs, SLOs, and alerts into a single assessment.
5. Observe LLM applications Monitors and troubleshoots LLM-powered applications — tracking token usage, latency, error rates, and model performance across inference calls.

What Agent Skills are

Agent Skills are self-contained folders with instructions, scripts, and resources that an AI agent loads dynamically for a specific task. Elastic publishes official skills in elastic/agent-skills, based on the Agent Skills standard.

At a practical level, this means:

• You describe the goal.
• The agent selects the relevant skill or you specify it.
• The skill applies known consistent steps and API patterns, Elastic recommendeds, for that job.

Practical example: from incident question to root-cause

As an SRE, you're notified that a specific customer is experiencing errors. Support has been trying to trouble shoot, but they need help. Support provides a transaction ID to investigate.

You've loaded Elastic's Agent Skills to Claude. You ask Claude:

Find out why transaction with id 01ba6cf8e60253bdeb26026caa3278a1 is having issues over the last 24 hours.

Claude, with Elastic O11y Skills added, analyzes the issue for that specific transaction with Elastic.

1. it uses the log-search skill to narrow down likely causes
2. the root cause is identified
3. and a potential remediation is recommended

How to get started

Install Elastic skills with the skills CLI:

npx skills add elastic/agent-skills

Install a specific skill directly:

npx skills add elastic/agent-skills --skill logs-search 

Then run your agent and give it an outcome-focused request, for example:

My cart service is experiencing some slowness, are there any errors over the last 3 hours? Please give me a summary of these logs.

The key shift is that the request is outcome-first. The skill captures implementation details such as API order, field expectations, and verification steps.

What is next

The planned scope includes broader workflow coverage. As skills mature, teams can combine them into repeatable operating patterns that still support ad hoc investigation.

If you want to try this model now, get Elastic's Agent Skills, start with one service and one workflow:

1. Assess service health.
2. Run guided log investigation for one real incident.
3. Add SLO management after baseline telemetry quality is in place.
4. Understand how well your LLM is performing for your developers.

This gives you a concrete way to evaluate agent-assisted observability work without changing your full operating model in one step.

What is Elastic’s Managed OTLP Endpoint?

The managed OTLP endpoint delivers on that promise offering a fully hosted OpenTelemetry ingestion path that’s scalable, reliable, and designed from the ground up for OpenTelemetry.

Data from OpenTelemetry SDKs, OpenTelemetry Collectors, or any OTLP service can send data to the OTLP endpoint. The OTLP endpoint is available on Elastic Cloud Serverless, and is fully managed by Elastic. This helps minimize the burden on customers of managing the OpenTelemetry ingestion layer. Whenever your production environment scales, the OTLP end point will also auto scale without any management from an SRE.

OpenTelemetry data is stored without any schema translation, preserving both semantic conventions and resource attributes. Additionally, it supports ingesting OTLP logs, metrics, and traces in a unified manner, ensuring consistent treatment across all telemetry data. This marks a significant improvement over the existing functionality, which primarily focuses on traces and APM use cases.

Hence, SREs gain: 

• Native OTLP ingestion with Elastic-managed reliability and scale

• OTel-native data storage, enabling richer analytics and future-proof observability

• Elastic-grade scaling, ready for production and multi-tenant workloads

• Frictionless onboarding, with a drop-in endpoint for logs, metrics and traces..

Native OTLP ingestion

Whether you are using native OTel SDKs, OpenTelemetry Collector, EDOT, or other OpenTelemetry instrumentation, the OTLP endpoint will ingest any native OTLP data.

The managed OTLP endpoint will automatically scale with Observability data that is notoriously bursty. A sudden spike in requests, a scaling event in Kubernetes, or a deployment gone sideways can lead to massive surges in telemetry, often when you need visibility the most. That’s exactly what the managed OTLP endpoint in Elastic Observability Serverless is built to handle.

This isn’t just a thin wrapper on a collector. It’s a multi-tenant, auto-scaling service architected to absorb high volumes of OpenTelemetry data without you having to manage infrastructure, pre-provision capacity, or worry about dropped data.

Whether you’re routing data directly from OpenTelemetry SDKs or via an intermediate Collector, Elastic handles the scale behind the scenes. The endpoint is designed to scale with your telemetry traffic and recover gracefully from bursts, giving you one less thing to monitor. Just point your instrumentation at the endpoint and let Elastic take care of the rest.

Natively stored OpenTelemetry 

With this feature, developers can now send OpenTelemetry signals directly to an Elastic Cloud Serverless project using the OTLP output of a collector or SDK regardless of the distribution contrib, EDOT and any other distribution will work). 

The endpoint also supports data forwarded from any OpenTelemetry Collectors, SDKs or OTLP compliant forwarder. This gives teams full control to send directly from an SDK or route, enrich, or batch telemetry when needed. Elasticsearch stores OpenTelemetry data using the OpenTelemetry data model, including resource attributes, to identify emitting entities and enable ES|QL queries that correlate logs, metrics, and traces.

Faster time-to-insight

Whether you’re building in serverless, Kubernetes, or classic VMs, this endpoint lets you focus on instrumentation and insights—not ingestion plumbing. It dramatically shortens the time from telemetry to value, while embracing the OpenTelemetry data model by preserving the original attributes and built-in correlation

Easy connectivity to Managed OTLP Endpoint

Connecting to the Managed OTLP endpoint is as simple as setting your SDK or the OTel collector OTLP export setting to the Elastic Managed OTLP Endpoint URL, and authentication key. Getting your endpoint is extremely straight-forward, go to project management, then edit alias and you will find your project’s OTLP endpoint. 

Get Started Today

The managed OTLP endpoint can be used today on Elastic Observability Serverless. Support for Elastic Cloud Hosted deployments is coming soon.

For more detail and examples, follow this guide.

Whether you’re running microservices in Kubernetes, workloads in serverless, or apps on classic VMs, the OTLP endpoint helps you streamline your observability pipeline, standardize on OpenTelemetry, and accelerate your mean time to resolution (MTTR).

Also check out our OTel resources about instrumenting and ingesting OTel into Elastic

Elastic Distributions of OpenTelemetry

Monitoring Kubernetes with Elastic and OpenTelemetry

Dynamic workload discovery with EDOT Collector

Assembling an OpenTelemetry NGINX Ingress Controller Integration

These latest updates, available in v9.3 and in Serverless, introduce significant performance gains, sophisticated time series functions, and native OpenTelemetry exponential histogram support that directly benefit SREs and Observability practitioners.

Query Performance and Storage Optimizations

Speed is paramount when diagnosing incidents. Compared to prior releases, we have achieved a 5x+ improvement in query latency when wildcarding or filtering by dimensions. Additionally, storage efficiency for OpenTelemetry metrics data has improved by approximately 2x, significantly reducing the infrastructure footprint required to retain high-volume observability data. If you’re hungry to learn more about what architectural updates are driving these optimizations, stay tuned… Tech blogs are on their way! 

Expanded Time Series Analytics in ES|QL

The ESQL TS source command, which targets time series indices and enables time series aggregation functions, has been significantly enhanced to support complex analytics capabilities.

We have expanded the library of time series functions to include essential tools for identifying anomalies and trends.

• PERCENTILE_OVER_TIME, STDDEV_OVER_TIME, VARIANCE_OVER_TIME: Calculate the percentile, standard deviation, or variance of a field over time, which is critical for understanding distribution and variability in service latency or resource usage.

Example: Seeing the worst-case latency in 5-minute intervals.

TS metrics*  | STATS MAX(PERCENTILE_OVER_TIME(kafka.consumer.fetch_latency_avg, 99))
  BY TBUCKET(5m)

• DERIV: This command calculates the derivative of a numeric field over time using linear regression, useful for analyzing the rate of change in system metrics.

Example: trending gauge values over time.

TS metrics*  | STATS AVG(DERIV(container.memory.available))
  BY TBUCKET(1 hour)

• CLAMP: To handle noisy data or outliers, this function limits sample values to a specified lower and upper bound.

Example: handling saturation metrics (like CPU or Memory utilization) where spikes or measurement errors can occasionally report values over 100%, making the rest of the data look like a flat line at the bottom of the chart.\

TS metrics*  | STATS AVG(CLAMP(k8s.pod.memory.node.utilization, 0, 100))
  BY k8s.pod.name

• TRANGE: This new filter function allows you to filter data for a specific time range using the @timestamp attribute, simplifying query syntax for time-bound investigations.

Example: Filtering and showing metrics for the last 4 hours.

TS metrics*  | WHERE TRANGE(4h) | STATS AVG(host.cpu.pct)
  BY TBUCKET(5m)

Window Functions To smoothen results over specific periods, ES|QL now introduces window functions. Most time series aggregation functions now accept an optional second argument that specifies a sliding time window. For example, you can calculate a rate over a 10-minute sliding window while bucketing results by minute.

Example: Calculating the average rate of requests per host for every minute, using values over a sliding window of 5 minutes.

TS metrics*  | STATS AVG(RATE(app.frontend.requests, 5m))
  BY TBUCKET(1m)

Accepted window values are currently limited to multiples of the time bucket interval in the BY clause. Windows that are smaller than the time bucket interval or larger but not a multiple of the time bucket interval will be supported in feature releases. 

Native OpenTelemetry Exponential Histograms

Elastic now provides native support for OpenTelemetry exponential histograms, enabling efficient ingest, querying, and downsampling of high-fidelity distribution data.

We have introduced a new exponential_histogram field type designed to capture distributions with fixed, exponentially spaced bucket boundaries. Because these fields are primarily intended for aggregations, the histogram is stored as compact doc values and is not indexed, optimizing storage efficiency. These fields are fully supported in ES|QL aggregation functions such as PERCENTILES, AVG, MIN, MAX, and SUM.

You can index documents with exponential histograms automatically through our OTLP endpoint or manually. For example, let’s create an index with an exponential histogram field and a keyword field:

PUT my-index-000001
{
  "settings": {
    "index": {
      "mode": "time_series",
      "routing_path": ["http.path"],
      "time_series": {
        "start_time": "2026-01-21T00:00:00Z",
        "end_time": "2026-01-25T00:00:00Z"
     }
    }
  },
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "http.path": {
        "type": "keyword",
        "time_series_dimension": true
      },
      "responseTime": {
        "type": "exponential_histogram",
        "time_series_metric": "histogram"
      }
    }
  }
}

Index a document with a full exponential histogram payload:

POST my-index-000001/_doc
{
  "@timestamp": "2026-01-22T21:25:00.000Z",
  "http.path": "/foo",
  "responseTime": {
    "scale":3,
    "sum":73.2,
    "min":3.12,
    "max":7.02,
    "positive": {
      "indices":[13,14,15,16,17,18,19,20,21,22],
      "counts":[1,1,2,2,1,2,1,3,1,1]
    }
  }
}

POST my-index-000001/_doc
{
  "@timestamp": "2026-01-22T21:26:00.000Z",
  "http.path": "/bar",
  "responseTime": {
    "scale":3,
    "sum":45.86,
    "min":2.15,
    "max":5.1,
    "positive": {
      "indices":[8,9,10,11,12,13,14,15,16,17,18],
      "counts":[1,1,1,1,1,1,1,2,1,1,2]
    }
  }
}

And finally, query the time series index using ES|QL and the TS source command:

TS my-index-000001  | STATS MIN(responseTime), MAX(responseTime),
        AVG(responseTime), MEDIAN(responseTime),
        PERCENTILE(responseTime, 90)
  BY http.path

Enhanced Downsampling

Downsampling is essential for long-term data retention. We have introduced a new "last value" downsampling mode. This method exchanges accuracy for storage efficiency and performance by keeping only the last sample value, providing a lightweight alternative to calculating aggregate metrics.

You can configure a time series data stream for last value downsampling in a similar way as regular downsampling, just by setting the downsampling_method to last_value. For example, by using a data stream lifecycle:

PUT _data_stream/my-data-stream/_lifecycle
{
  "data_retention": "7d",
  "downsampling_method": "last_value",
  "downsampling": [
     {
       "after": "1m",
       "fixed_interval": "10m"
      },
      {
        "after": "1d",
        "fixed_interval": "1h"
      }
   ]
}

In Conclusion

These enhancements mark a significant step forward in Elastic's metrics analytics capabilities, delivering 5x+ faster query latency, 2x storage efficiency and specialized commands like DERIV, CLAMP, and PERCENTILE_OVER_TIME. With native support for OpenTelemetry exponential histograms and expanded downsampling options, SREs can now perform richer, more cost-effective analysis on their observability data. This release empowers teams to detect anomalies faster and manage long-term metrics retention with greater efficiency.

We welcome you to try the new features today!

The Elastic MongoDB Atlas integration transforms how you monitor and troubleshoot your Atlas infrastructure by providing deep insights into every aspect of your deployment—from real-time alerts and audit trails to detailed performance metrics and organizational activities. This integration empowers teams to minimize Mean Time to Detect (MTTD) and Mean Time to Resolve (MTTR) while gaining actionable insights for capacity planning and performance optimization.

Why MongoDB Atlas Observability Matters

MongoDB Atlas abstracts much of the operational complexity of running MongoDB, but this doesn't eliminate the need for monitoring. Modern applications demand:

• Proactive Issue Detection: Identify performance bottlenecks, resource constraints, and security threats before they impact users
• Comprehensive Audit Trails: Track database operations, user activities, and configuration changes for compliance and security
• Performance Optimization: Monitor query performance, resource utilization, and capacity trends to optimize costs and user experience
• Operational Insights: Understand organizational activities, project changes, and infrastructure events across your multi-cloud deployments

The Elastic MongoDB Atlas integration addresses these needs by collecting comprehensive telemetry data and presenting it through powerful visualizations and alerting capabilities.

Integration Architecture and Data Streams

The MongoDB Atlas integration leverages the Atlas Administration API to collect eight distinct data streams, each providing specific insights into different aspects of your Atlas deployment:

Log Data Streams

Alert Logs: Capture real-time alerts generated by your Atlas instances, covering resource utilization thresholds (CPU, memory, disk space), database operations, security issues, and configuration changes. These alerts provide immediate visibility into critical events that require attention.

Database Logs: Collect comprehensive operational logs from MongoDB instances, including incoming connections, executed commands, performance diagnostics, and issues encountered. These logs are invaluable for troubleshooting performance problems and understanding database behavior.

MongoDB Audit Logs: Enable administrators to track system activity across deployments with multiple users and applications. These logs capture detailed events related to database operations including insertions, updates, deletions, user authentication, and access patterns—essential for security compliance and forensic analysis.

Organization Logs: Provide enterprise-level visibility into organizational activities, enabling tracking of significant actions involving database operations, billing changes, security modifications, host management, encryption settings, and user access management across teams.

Project Logs: Offer project-specific event tracking, capturing detailed records of configuration modifications, user access changes, and general project activities. These logs are crucial for project-level auditing and change management.

Metrics Data Streams

Hardware Metrics: Collect comprehensive hardware performance data including CPU usage, memory consumption, JVM memory utilization, and overall system resource metrics for each process in your Atlas groups.

Disk Metrics: Monitor storage performance with detailed insights into I/O operations, read/write latency, and space utilization across all disk partitions used by MongoDB Atlas. These metrics help identify storage bottlenecks and plan capacity expansion.

Process Metrics: Gather host-level metrics per MongoDB process, including detailed CPU usage patterns, I/O operation counts, memory utilization, and database-specific performance indicators like connection counts, operation rates, and cache utilization.

Implementation Guide

Setting Up the Integration

Getting started with MongoDB Atlas observability requires establishing API access and configuring the integration in Kibana:

1. Generate Atlas API Keys: Create programmatic API keys with Organization Owner permissions in the Atlas console, then invite these keys to your target projects with appropriate roles (Project Read Only for alerts/metrics, Project Data Access Read Only for audit logs).

2. Enable Prerequisites: Enable database auditing in Atlas for projects where you want to collect audit and database logs. Gather your Project ID and Organization ID from the Atlas UI.

3. Configure in Kibana: Navigate to Management > Integrations, search for "MongoDB Atlas," and add the integration using your API credentials.

The integration supports different permission levels for each data stream, ensuring you can collect operational metrics with minimal privileges while protecting sensitive audit data with elevated permissions.

Considerations and Limitations

• Cluster Support: Log collection doesn't support M0 free clusters, M2/M5 shared clusters, or serverless instances
• Historical Data: Most log streams collect the previous 30 minutes of historical data
• Performance Impact: Large time spans may cause request timeouts; adjust HTTP Client Timeout accordingly

Real-World Use Cases and Benefits

Security and Compliance Monitoring

Audit Trail Management: Organizations in regulated industries leverage the audit logs to maintain comprehensive records of database access and modifications. The integration automatically parses and indexes audit events, making it easy to search for specific user activities, failed authentication attempts, or unauthorized access patterns.

Security Incident Response: When security events occur, teams can quickly correlate alert logs with audit trails to understand the scope and timeline of incidents.

Performance Optimization and Capacity Planning

Proactive Resource Management: By monitoring disk, hardware, and process metrics, teams can identify resource constraints before they impact application performance. For example, tracking disk I/O latency trends helps predict when storage upgrades are needed.

Query Performance Analysis: Database logs combined with process metrics provide insights into slow queries, connection patterns, and resource utilization that enable database performance tuning.

Operational Excellence

Multi-Environment Monitoring: Organizations running Atlas across development, staging, and production environments can standardize monitoring across all environments while maintaining environment-specific alerting thresholds.

Change Management: Project and organization logs provide complete audit trails for infrastructure changes, enabling teams to correlate application issues with recent configuration modifications.

Let's Try It!

The MongoDB Atlas integration delivers comprehensive database observability that enables proactive management and optimization of your Atlas deployments. With pre-built dashboards and alerting capabilities, teams can gain immediate value while leveraging rich data streams for advanced analytics and custom monitoring solutions.

Deploy a cluster on Elastic Cloud or Elastic Serverless, or download the Elasticsearch stack, then spin up the MongoDB Atlas Integration, open the curated dashboards in Kibana and start monitoring your service!

Elastic Observability offers 16 out-of-the-box integrations for Google Cloud services with more on the way. A full list of Google Cloud integrations can be found in our online documentation.

In addition to our native Google Cloud integrations, Elastic Observability aggregates not only logs but also metrics for Google Cloud services and the applications running on Google Cloud compute services (Compute Engine, Cloud Run, Cloud Functions, Kubernetes Engine). All this data can be analyzed visually and more intuitively using Elastic®’s advanced machine learning (ML) capabilities, which help detect performance issues and surface root causes before end users are affected.

For more details on how Elastic Observability provides application performance monitoring (APM) capabilities such as service maps, tracing, dependencies, and ML based metrics correlations, read: APM correlations in Elastic Observability: Automatically identifying probable causes of slow or failed transactions.

That’s right, Elastic offers metrics ingest, aggregation, and analysis for Google Cloud services and applications on Google Cloud compute services. Elastic is more than logs — it offers a unified observability solution for Google Cloud environments.

In this blog, I’ll review how Elastic Observability can monitor metrics for a three-tier web application running on Google Cloud services, which include:

• Google Cloud Run
• Google Cloud SQL for PostgreSQL
• Google Cloud Memorystore for Redis
• Google Cloud VPC Network

As you will see, once the integration is installed, metrics will arrive instantly and you can immediately start reviewing metrics.

Prerequisites and config

Here are some of the components and details we used to set up this demonstration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).
• Ensure you have a Google Cloud project and a Service Account with permissions to pull the necessary data from Google Cloud (see details in our documentation).
• We used Google Cloud’s three-tier app and deployed it using the Google Cloud console.
• We’ll walk through installing the general Elastic Google Cloud Platform Integration, which covers the services we want to collect metrics for.
• We will not cover application monitoring; instead, we will focus on how Google Cloud services can be easily monitored.
• In order to see metrics, you will need to load the application. We’ve also created a playwright script to drive traffic to the application.

Three-tier application overview

Before we dive into the Elastic configuration, let's review what we are monitoring. If you follow the Jump Start Solution: Three-tier web app instructions fordeploying the task-tracking app, you will have the following deployed.

What’s deployed:

• Cloud Run frontend tier that renders an HTML client in the user's browser and enables user requests to be sent to the task-tracking app
• Cloud Run middle tier API layer that communicates with the frontend and the database tier
• Memorystore for Redis instance in the database tier, caching and serving data that is read frequently
• Cloud SQL for PostgreSQL instance in the database tier, handling requests that can't be served from the in-memory Redis cache

At the end of the blog, we will also provide a Playwright script that can be run to send requests to this app in order to load it with example data and exercise its functionality. This will help drive metrics to “light up” the dashboards.

Setting it all up

Let’s walk through the details of how to get the application, Google Cloud integration on Elastic, and what gets ingested.

Step 0: Get an account on Elastic Cloud

Follow the instructions to get started on Elastic Cloud.

Step 1: Deploy the Google Cloud three-tier application

Follow the instructions listed out in Jump Start Solution: Three-tier web app choosing the Deploy through the console option for deployment.

Step 2: Create a Google Cloud Service Account and download credentials file

Once you’ve installed the app, the next step is to create a Service Account with a Role and a Service Account Key that will be used by Elastic’s integration to access data in your Google Cloud project.

Go to Google Cloud IAM Roles to create a Role with the necessary permissions. Click the CREATE ROLE button.

Give the Role a Title and an ID. Then add the 10 assigned permissions listed here.

• cloudsql.instances.list
• compute.instances.list
• monitoring.metricDescriptors.list
• monitoring.timeSeries.list
• pubsub.subscriptions.consume
• pubsub.subscriptions.create
• pubsub.subscriptions.get
• pubsub.topics.attachSubscription
• redis.instances.list
• run.services.list

These permissions are a minimal set of what’s required for this blog post. You should add permissions for all the services for which you would like to collect metrics. If you need to add or remove permissions in the future, the Role’s permissions can be updated as many times as necessary.

Click the CREATE button.

Go to Google Cloud IAM Service Accounts to create a Service Account that will be used by the Elastic integration for access to Google Cloud. Click the CREATE SERVICE ACCOUNT button.

Enter a Service account name and a Service account ID. Click the CREATE AND CONTINUE button.

Then select the Role that you created previously and click the CONTINUE button.

Click the DONE button to complete the Service Account creation process.

Next select the Service Account you just created to see its details page. Under the KEYS tab, click the ADD KEY dropdown and select Create new key.

In the Create private key dialog window, with the Key type set as JSON, click the CREATE button.

The JSON credentials file key will be automatically downloaded to your local computer’s Downloads folder. The credentials file will be named something like:

your-project-id-12a1234b1234.json

You can rename the file to be something else. For the purpose of this blog, we’ll rename it to:

credentials.json

Step 3: Create a Google Cloud VM instance

To create the Compute Engine VM instance in Google Cloud, go to Compute Engine. Then select CREATE INSTANCE.

Enter the following values for the VM instance details:

• Enter a Name of your choice for the VM instance.
• Expand the Advanced Options section and the Networking sub-section.
  • Enter allow-ssh as the Networking tag.
  • Select the Network Interface to use the tiered-web-app-private-network , which is the network on which the Google Cloud three-tier web app is deployed.

Click the CREATE button to create the VM instance.

Step 4: SSH in to the Google Cloud VM instance and upload the credentials file

In order to SSH into the Google Cloud VM instance you just created in the previous step, you’ll need to create a Firewall rule in tiered-web-app-private-network , which is the network where the VM instance resides.

Go to the Google Cloud Firewall policies page. Click the CREATE FIREWALL RULE button.

Enter the following values for the Firewall Rule.

• Enter a firewall rule Name.
• Select tiered-web-app-private-network for the Network.
• Enter allow-ssh for Target Tags.
• Enter 0.0.0.0/0 for the Source IPv4 ranges.Click TCP and set the Ports to 22.

Click CREATE to create the firewall rule.

After the new Firewall rule is created, you can now SSH into your VM instance. Go to the Google Cloud VM instances and select the VM instance you created in the previous step to see its details page. Click the SSH button.

Once you are SSH’d inside the VM instance terminal window, click the UPLOAD FILE button.

Select the credentials.json file located on your local computer and click the Upload Files button to upload the file.

In the VM instance’s SSH terminal, run the following command to get the full path to your Google Cloud Service Account credentials file.

realpath credentials.json

This should return the full path to your Google Cloud Service Account credentials file.

Copy the credentials file’s full path and save it in a handy location to be used in a later step.

Step 5: Add the Elastic Google Cloud integration

Navigate to the Google Cloud Platform integration in Elastic by selecting Integrations from the top-level menu. Search for google and click the Google Cloud Platform tile.

Click Add Google Cloud Platform.

Click Add integration only (skip agent installation).

Update the Project Id input text box to be your Google Cloud Project ID. Next, paste in the credentials file’s full path into the Credentials File input text box.

As you can see, the general Elastic Google Cloud Platform Integration will collect a significant amount of data from 16 Google Cloud services. If you don’t want to install this general Elastic Google Cloud Platform Integration, you can select individual integrations to install. Click Save and continue.

You’ll be presented with a confirmation dialog window. Click Add Elastic Agent to your hosts.

This will display the instructions required to install the Elastic agent. Copy the command under the Linux Tar tab.

Next you will need to use SSH to log in to the Google Cloud VM instance and run the commands copied from Linux Tar tab. Go to Compute Engine. Then click the name of the VM instance that you created in Step 2. Log in to the VM by clicking the SSH button.

Once you are SSH’d inside the VM instance terminal window, run the commands copied previously from Linux Tar tab in the Install Elastic Agent on your host instructions.

When the installation completes, you’ll see a confirmation message in the Install Elastic Agent on your host form. Click the Add the integration button.

Excellent! The Elastic agent is sending data to Elastic Cloud. Now let’s observe some metrics.

Step 6: Run traffic against the application

While getting the application running is fairly easy, there is nothing to monitor or observe with Elastic unless you add a load on the application.

Here is a simple script you can also run using Playwright to add traffic and exercise the functionality of the Google Cloud three-tier application:

import { test, expect } from "@playwright/test";

test("homepage for Google Cloud Threetierapp", async ({ page }) => {
  await page.goto("https://tiered-web-app-fe-zg62dali3a-uc.a.run.app");
  // Insert 2 todo items
  await page.fill("id=todo-new", (Math.random() * 100).toString());
  await page.keyboard.press("Enter");
  await page.waitForTimeout(1000);
  await page.fill("id=todo-new", (Math.random() * 100).toString());
  await page.keyboard.press("Enter");
  await page.waitForTimeout(1000);
  // Click one todo item
  await page.getByRole("checkbox").nth(0).check();
  await page.waitForTimeout(1000);
  // Delete one todo item
  const deleteButton = page.getByText("delete").nth(0);
  await deleteButton.dispatchEvent("click");
  await page.waitForTimeout(4000);
});

Step 7: Go to Google Cloud dashboards in Elastic

With Elastic Agent running, you can go to Elastic Dashboards to view what’s being ingested. Simply search for “dashboard” in Elastic and choose Dashboards.

This will open the Elastic Dashboards page.

In the Dashboards search box, search for GCP and click the [Metrics GCP] CloudSQL PostgreSQL Overview dashboard, one of the many out-of-the-box dashboards available. Let’s see what comes up.

On the Cloud SQL dashboard, we can see the following sampling of some of the many available metrics:

• Disk write ops
• CPU utilization
• Network sent and received bytes
• Transaction count
• Disk bytes used
• Disk quota
• Memory usage
• Disk read ops

Next let’s take a look at metrics for Cloud Run.

We’ve created a custom dashboard using the Create dashboard button on the Elastic Dashboards page. Here we see a few of the numerous available metrics:

• Container instance count
• CPU utilization for the three-tier app frontend and API
• Request count for the three-tier app frontend and API
• Bytes in and out of the API

This is a custom dashboard created for MemoryStore where we can see the following sampling of the available metrics:

• Network traffic to the Memorystore Redis instance
• Count of the keys stored in Memorystore Redis
• CPU utilization of the Memorystore Redis instance
• Memory usage of the Memorystore Redis instance

Congratulations, you have now started monitoring metrics from key Google Cloud services for your application!

What to monitor on Google Cloud next?

Add logs from Google Cloud Services

Now that metrics are being monitored, you can also now add logging. There are several options for ingesting logs.

The Google Cloud Platform Integration in the Elastic Agent has four separate logs settings: audit logs, firewall logs, VPC Flow logs, and DNS logs. Just ensure you turn on what you wish to receive.

Analyze your data with Elastic machine learning

Once metrics and logs (or either one) are in Elastic, start analyzing your data through Elastic’s ML capabilities. A great review of these features can be found here:

• Correlating APM Telemetry to determine root causes in transactions
• Introduction to Elastic Machine Learning

Conclusion: Monitoring Google Cloud service metrics with Elastic Observability is easy!

I hope you’ve gotten an appreciation for how Elastic Observability can help you monitor Google Cloud service metrics. Here’s a quick recap of lessons and what you learned:

• Elastic Observability supports ingest and analysis of Google Cloud service metrics.
• It’s easy to set up ingest from Google Cloud services via the Elastic Agent.
• Elastic Observability has multiple out-of-the-box Google Cloud service dashboards you can use to preliminarily review information and then modify for your needs.
• For metrics not covered by out-of-the-box dashboards, custom dashboards can be easily created to visualize metrics that are important to you.
• 16 Google Cloud services are supported as part of Google Cloud Platform Integration on Elastic Observability, with more services being added regularly.
• As noted in related blogs, you can analyze your Google Cloud service metrics with Elastic’s machine learning capabilities.

Try it out for yourself by signing up via Google Cloud Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on Google Cloud around the world. Your Google Cloud Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with Google Cloud.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Elastic Observability offers over 25 out-of-the-box integrations for Microsoft Azure services with more on the way. A full list of Azure integrations can be found in our online documentation.

Elastic Observability aggregates not only logs but also metrics for Azure services and the applications running on Azure compute services (Virtual Machines, Functions, Kubernetes Service, etc.). All this data can be analyzed visually and more intuitively using Elastic®’s advanced machine learning (ML) capabilities, which help detect performance issues and surface root causes before end users are affected.

For more details on how Elastic Observability provides application performance monitoring (APM) capabilities such as service maps, tracing, dependencies, and ML-based metrics correlations, read APM correlations in Elastic Observability: Automatically identifying probable causes of slow or failed transactions.

That’s right, Elastic offers capabilities to collect, aggregate, and analyze metrics for Microsoft Azure services and applications running on Azure. Elastic Observability is for more than just capturing logs — it offers a unified observability solution for Microsoft Azure workloads.

In this blog, we’ll review how Elastic Observability can monitor metrics for a three-tier web application running on Microsoft Azure and leveraging:

• Microsoft Azure Virtual Machines
• Microsoft Azure SQL database
• Microsoft Azure Virtual Network

As you will see, once the integration is installed, metrics will arrive instantly and you can immediately start deriving insights from metrics.

Prerequisites and config

Here are some of the components and details we used to set up this demonstration:

• Ensure you have a Microsoft Azure account and an Azure service principal with permission to read monitoring data from Microsoft Azure (see details in our documentation).
• This post does not cover application monitoring; instead, we will focus on how Microsoft Azure services can be easily monitored. If you want to get started with examples of application monitoring, see our Hello World observability code samples.
• In order to see metrics, you will need to load the application. We’ve also created a Playwright script to drive traffic to the application.

Three-tier application overview

Before we dive into the Elastic deployment setup and configuration, let's review what we are monitoring. If you follow the Microsoft Learn N-tier example app instructions for deploying the "What's for Lunch?" app, you will have the following deployed.

What’s deployed:

• Microsoft Azure VM presentation tier that renders an HTML client in the user's browser and enables user requests to be sent to the “What’s for Lunch?” app
• Microsoft Azure VM application tier that communicates with the presentation and the database tier
• Microsoft Azure SQL instance in the database tier, handling requests from the application tier to store and serve data

At the end of the blog, we will also provide a Playwright script that can be run to send requests to this app in order to load it with example data and exercise its functionality. This will help drive metrics to “light up” the dashboards.

Setting it all up

Let’s walk through the details of how to deploy the example three-tier application, Azure integration on Elastic and visualize what gets ingested in Elastic’s Kibana® dashboards.

Step 0: Get an account on Elastic Cloud

Follow the instructions to get started on Elastic Cloud.

Step 1: Deploy the Microsoft Azure three-tier application

From the Azure portal, click the Cloud Shell icon at the top of the portal to open Cloud Shell…

… and when the Cloud Shell first opens, select Bash as the shell type to use.

If you’re prompted that “You have no storage mounted,” then click the Create storage button to create a file store to be used for saving and editing files from Cloud Shell.

You should now see the open Cloud Shell terminal.

Run the following command in Cloud Shell to define the environment variables that we’ll be using in the Cloud Shell commands required to deploy and view the sample application.

Be sure to specify a valid RESOURCE_GROUP from your available Resource Groups listed in the Azure portal. Also specify a new password to replace the SpecifyNewPasswordHere placeholder text before running the command. See the Microsoft password policy documentation for password requirements.

RESOURCE_GROUP="test"
APP_PASSWORD="SpecifyNewPasswordHere"

Run the following az deployment group create command, which will deploy the example three-tier web app in around five minutes.

az deployment group create --resource-group $RESOURCE_GROUP --template-uri https://raw.githubusercontent.com/MicrosoftDocs/mslearn-n-tier-architecture/master/Deployment/azuredeploy.json --parameters password=$APP_PASSWORD

After the deployment has completed, run the following command, which returns the URL for the app.

az deployment group show --output table --resource-group $RESOURCE_GROUP --name azuredeploy --query properties.outputs.webSiteUrl

Copy the web app URL and paste it into a browser to view the example “What’s for Lunch?” web app.

Step 2: Create an Azure service principal and grant access permission

Go to the Microsoft Azure Portal. Search for active directory and select Microsoft Entra ID.

Copy the Tenant ID for use in a later step in this blog post. This ID is required to configure Elastic Agent to connect to your Azure account.

In the navigation pane, select App registrations.

Then click New registration.

Type the name of your application (this tutorial uses three-tier-app-azure) and click Register (accept the default values for other settings).

Copy the Application (client) ID and save it for later. This ID is required to configure Elastic Agent to connect to your Azure account.

In the navigation pane, select Certificates & secrets , and then click New client secret to create a new security key.

Type a description of the secret and select an expiration. Click Add to create the client secret. Under Value , copy the secret value and save it (along with your client ID) for later.

After creating the Azure service principal, you need to grant it the correct permissions. In the Azure Portal, search for and select Subscriptions.

In the Subscriptions page, click the name of your subscription. On the subscription details page, copy your Subscription ID and save it for a later step.

In the navigation pane, select Access control (IAM).

Click Add and select Add role assignment.

On the Role tab, select the Monitoring Reader role and then click Next.

On the Members tab, select the option to assign access to User, group, or service principal. Click Select members , and then search for and select the principal you created earlier. For the description, enter the name of your service principal. Click Next to review the role assignment.

Click Review + assign to grant the service principal access to your subscription.

Step 3: Create an Azure VM instance

In the Azure Portal, search for and select Virtual machines.

On the Virtual machines page, click + Create and select Azure virtual machine.

On the Virtual machine creation page, enter a name like “metrics-vm” for the virtual machine name and select VM Size to be “Standard_D2s_v3 - 2 vcpus, 8 GiB memory.” Click the Next : Disks button.

On the Disks page, keep the default settings and click the Next : Networking button.

On the Networking page, demo-vnet should be selected for Virtual network and demo-biz-subnet should be selected for Subnet. These resources are created as part of the three-tier example app’s deployment that was done in Step 1.

Click the Review + create button.

On the Review page, click the Create button.

Step 4: Install the Azure Resource Metrics integration

In your Elastic Cloud deployment, navigate to the Elastic Azure integrations by selecting Integrations from the top-level menu. Search for azure resource and click the Azure Resource Metrics tile.

Click Add Azure Resource Metrics.

Click Add integration only (skip agent installation).

Enter the values that you saved previously for Client ID, Client Secret, Tenant ID, and Subscription ID.

As you can see, the Azure Resource Metrics integration will collect a significant amount of data from eight Azure services. Click Save and continue.

You’ll be presented with a confirmation dialog window. Click Add Elastic Agent to your hosts.

This will display the instructions required to install the Elastic agent. Copy the command under the Linux Tar tab.

Next you will need to use SSH to log in to the Azure VM instance and run the commands copied from Linux Tar tab. Go to Azure Virtual Machines in the Azure portal. Then click the name of the VM instance that you created in Step 3.

Click the Select button in the SSH Using Azure CLI section.

Select the “I understand …” checkbox and then click the Configure + connect button.

Once you are SSH’d inside the VM instance terminal window, run the commands copied previously from Linux Tar tab in the Install Elastic Agent on your host instructions. When the installation completes, you’ll see a confirmation message in the Install Elastic Agent on your host form.

Super! The Elastic agent is sending data to Elastic Cloud. Now let’s observe some metrics.

Step 5: Run traffic against the application

While getting the application running is fairly easy, there is nothing to monitor or observe with Elastic unless you add a load on the application.

Here is a simple script you can also run using Playwright to add traffic and exercise the functionality of the Azure three-tier application:

import { test, expect } from "@playwright/test";

test("homepage for Microsoft Azure three tier app", async ({ page }) => {
  // Load web app
  await page.goto("http://20.172.198.231/");
  // Add lunch suggestions
  await page.fill("id=txtAdd", "tacos");
  await page.keyboard.press("Enter");
  await page.waitForTimeout(1000);
  await page.fill("id=txtAdd", "sushi");
  await page.keyboard.press("Enter");
  await page.waitForTimeout(1000);
  await page.fill("id=txtAdd", "pizza");
  await page.keyboard.press("Enter");
  await page.waitForTimeout(1000);
  await page.fill("id=txtAdd", "burgers");
  await page.keyboard.press("Enter");
  await page.waitForTimeout(1000);
  await page.fill("id=txtAdd", "salad");
  await page.keyboard.press("Enter");
  await page.waitForTimeout(1000);
  await page.fill("id=txtAdd", "sandwiches");
  await page.keyboard.press("Enter");
  await page.waitForTimeout(1000);
  // Click vote buttons
  await page.getByRole("button").nth(1).click();
  await page.getByRole("button").nth(3).click();
  await page.getByRole("button").nth(5).click();
  await page.getByRole("button").nth(7).click();
  await page.getByRole("button").nth(9).click();
  await page.getByRole("button").nth(11).click();
  // Click remove buttons
  await page.getByRole("button").nth(12).click();
  await page.getByRole("button").nth(10).click();
  await page.getByRole("button").nth(8).click();
  await page.getByRole("button").nth(6).click();
  await page.getByRole("button").nth(4).click();
  await page.getByRole("button").nth(2).click();
});

Step 6: View Azure dashboards in Elastic

With Elastic Agent running, you can go to Elastic Dashboards to view what’s being ingested. Simply search for “dashboard” in Elastic and choose Dashboard.

This will open the Elastic Dashboards page. In the Dashboards search box, search for azure vm and click the [Azure Metrics] Compute VMs Overview dashboard, one of the many out-of-the-box dashboards available.

You will see a Dashboard populated with your deployed application’s VM metrics.

On the Azure Compute VM dashboard, we can see the following sampling of some of the many available metrics:

• CPU utilization
• Available memory
• Network sent and received bytes
• Disk writes and reads metrics

For metrics not covered by out-of-the-box dashboards, custom dashboards can be easily created to visualize metrics that are important to you.

Congratulations, you have now started monitoring metrics from Microsoft Azure services for your application!

Analyze your data with Elastic AI Assistant

Once metrics and logs (or either one) are in Elastic, start analyzing your data with context-aware insights using the Elastic AI Assistant for Observability.

Conclusion: Monitoring Microsoft Azure service metrics with Elastic Observability is easy!

We hope you’ve gotten an appreciation for how Elastic Observability can help you monitor Azure service metrics. Here’s a quick recap of what you learned:

• Elastic Observability supports ingest and analysis of Azure service metrics.
• It’s easy to set up ingest from Azure services via the Elastic Agent.
• Elastic Observability has multiple out-of-the-box Azure service dashboards you can use to preliminarily review information and then modify for your needs.

Try it out for yourself by signing up via Microsoft Azure Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on Microsoft Azure around the world. Your Azure Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with Microsoft Azure.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Elastic is committed to supporting Google Kubernetes Engine (GKE) in all of its delivery modes. In October, during the Google Cloud Next ‘22 event, we announced our intention to integrate and certify Elastic Agent on Anthos, Autopilot, Google Distributed Cloud, and more.

Since that event, we have worked together with Google to get the Elastic Agent certified for use on Anthos, but we didn’t stop there.

Today we are happy to announce that we have been certified for operation on GKE Autopilot.

Hands on with Elastic and GKE Autopilot

Kubernetes observability has never been easier

To show how easy it is to get started with Autopilot and Elastic, let's walk through deploying the Elastic Agent on an Autopilot cluster. I’ll show how easy it is to set up and monitor an Autopilot cluster with the Elastic Agent and observe the cluster’s behavior with Kibana integrations.

One of the main differences between GKE and GKE Autopilot is that Autopilot protects the system namespace “kube-system.” To increase the stability and security of a cluster, Autopilot prevents user space workloads from adding or modifying system pods. The default configuration for Elastic Agent is to install itself into the system namespace. The majority of the changes we will make here are to convince the Elastic Agent to run in a different namespace.

Let’s get started with Elastic Stack!

While writing this article, I used the latest version of Elastic. The best way for you to get started with Elastic Observability is to:

1. Get an account on Elastic Cloud and look at this tutorial to help launch your first stack, or
2. Launch Elastic Cloud on your Google Account

Provisioning an Autopilot cluster and an Elastic stack

To test the agent, I first deployed the recommended, default GKE Autopilot cluster. Elastic’s GKE integration supports kube-state-metrics (KSM), which will increase the number of reported metrics available for reporting and dashboards. Like the Elastic Agent, KSM defaults to running in the system namespace, so I modified its manifest to work with Autopilot. For my testing, I also deployed a basic Elastic stack on Elastic Cloud in the same Google region as my Autopilot cluster. I used a fresh cluster deployed on Elastic’s managed service (ESS), but the process is the same if you are using an Elastic Cloud subscription purchased through the Google marketplace.

Adding Elastic Observability to GKE Autopilot

Because this is a brand new deployment, Elastic suggests adding integrations to it. Let’s add the Kubernetes integration into the new deployment:

Elastic offers hundreds of integrations; filter the list by typing “kub” into the search bar (1) and then click the Kubernetes integration (2).

The Kubernetes integration page gives you an overview of the integration and lets you manage the Kubernetes clusters you want to observe. We haven’t added a cluster yet, so I clicked “Add Kubernetes” to add the first integration.

I changed the integration name to reflect the Kubernetes offering type and then clicked “Save and continue” to accept the integration defaults.

At this point, an Agent policy has been created. Now it’s time to install the agent. I clicked on the “Kubernetes” integration.

Then I selected the “integration policies” tab (1) and clicked “Add agent” (2).

Finally, I downloaded the full manifest for a standard GKE environment.

We won’t be using this manifest directly, but it contains many of the values that we will need to deploy the agent on Autopilot in the next section.

The Elastic stack is ready and waiting for the Autopilot logs, metrics, and events. It’s time to connect Autopilot to this deployment using the Elastic Agent for GKE.

Connect Autopilot to Elastic

From the Google cloud terminal, I downloaded and edited the Elastic Agent manifest for GKE Autopilot.

$ curl -o elastic-agent-managed-gke-autopilot.yaml \
https://github.com/elastic/elastic-agent/blob/autopilotdocumentaton/docs/manifests/elastic-agent-managed-gke-autopilot.yaml

I used the cloud shell editor to configure the manifest for my Autopilot and Elastic clusters. For example, I updated the following:

containers:
  - name: elastic-agent
    image: docker.elastic.co/beats/elastic-agent:8.19.13

I also changed the agent to the version of Elastic that I installed (8.6.0).

From the Integration manifest I downloaded earlier, I copied the values for FLEET_URL and FLEET_ENROLLMENT_TOKEN into this YAML file.

Now it’s time to apply the updated manifest to the Autopilot instance.

Before I commit, I always like to see what’s going to be created (and check for syntax errors) with a dry run.

$ clear
$ kubectl apply --dry-run="client" -f elastic-agent-managed-gke-autopilot.yaml

Everything looks good, so I’ll do it for real this time.

$ clear
$ kubectl apply -f elastic-agent-managed-gke-autopilot.yaml

After several minutes, metrics will start flowing from the Autopilot cluster directly into the Elastic deployment.

Adding a workload to the Autopilot cluster

Observing an Autopilot cluster without a workload is boring, so I deployed a modified version of Google’s Hipster Shop (which includes OpenTelemetry reporting):

$ git clone https://github.com/bshetti/opentelemetry-microservices-demo
$ cd opentelemetry-microservices-demo
$ nano ./deploy-with-collector-k8s/otelcollector.yaml

To get the application’s telemetry talking to our Elastic stack, I replaced all instances of the exporter type from HTTP (otlphttp/elastic) to gRPC (otlp/elastic). I then replaced OTEL_EXPORTER_OTLP_ENDPOINT with my APM endpoint and I replaced OTEL_EXPORTER_OTLP_HEADERS with my APM OTEL Bearer and Token.

Then I deployed the Hipster Shop.

$ kubectl create -f ./deploy-with-collector-k8s/adservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/redis.yaml
$ kubectl create -f ./deploy-with-collector-k8s/cartservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/checkoutservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/currencyservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/emailservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/frontend.yaml
$ kubectl create -f ./deploy-with-collector-k8s/paymentservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/productcatalogservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/recommendationservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/shippingservice.yaml
$ kubectl create -f ./deploy-with-collector-k8s/loadgenerator.yaml

Once all of the shop’s pods were running, I deployed the OpenTelemetry collector.

$ kubectl create -f ./deploy-with-collector-k8s/otelcollector.yaml

Observe and visualize Autopilot’s metrics

Now that we have added the Elastic Agent to our Autopilot cluster and added a workload, let's take a look at some of the Kubernetes visualizations the integration provides out of the box.

The “[Metrics Kubernetes] Overview” is a great place to start. It provides a high-level view of the resources used by the cluster and allows me to drill into more specific dashboards that I find interesting:

For example, the “[Metrics Kubernetes] Pods” gives me a high-level view of the pods deployed in the cluster:

The “[Metrics Kubernetes] Volumes” gives me an in-depth view to how storage is allocated and used in the Autopilot cluster:

Creating an alert

From here, I can easily discover patterns in my cluster’s behavior and even create Alerts. Here is an example of an alert to notify me if the the main storage volume (called “volume”) exceeds 80% of its allocated space:

With a little work, I created this view from the standard dashboard:

Conclusion

Today I have shown how easy it is to monitor, observe, and generate alerts on a GKE Autopilot cluster. To get more information on what is possible, see the official Elastic documentation for Autopilot observability with Elastic Agent.

Next steps

If you don’t have Elastic yet, you can get started for free with an Elastic Trial today. Get more from Elastic and Google together with a Marketplace subscription. Elastic does more than just integrate with GKE — check out the almost 300 integrations that Elastic provides.

The new PROMQL command in ES|QL lets you query time series data in Elasticsearch with PromQL, whether it came from Prometheus Remote Write, OpenTelemetry, or another source.

Metrics, logs, and traces - all in one place, ready to explore in Kibana.

The PROMQL source command

PROMQL is a source command in ES|QL, similar to FROM or TS. It takes standard PromQL parameters and a PromQL expression, executes the query, and returns the results as regular ES|QL columns that you can continue to process with other commands.

Here is the general syntax:

PROMQL [index=<pattern>] [step=<duration>] [start=<timestamp>] [end=<timestamp>]
  [<value_column_name>=](<PromQL expression>)

The parameters mirror the Prometheus HTTP API query parameters (step, start, end), so they should feel familiar if you have used the Prometheus query API before.

A basic range query

This query calculates the per-second rate of HTTP requests over a sliding 5-minute window, grouped by instance:

PROMQL index=metrics-*
  step=1m
  start="2026-04-01T00:00:00Z"
  end="2026-04-01T01:00:00Z"
  sum by (instance) (rate(http_requests_total[5m]))

The result contains three columns:

When the PromQL expression includes a cross-series aggregation like sum by (instance), each grouping label becomes its own output column. When there is no cross-series aggregation, all labels are returned in a single _timeseries column as a JSON string.

Naming the value column

By default, the value column name is the PromQL expression itself. You can assign a custom name to make it easier to reference in downstream commands:

PROMQL index=metrics-*
  step=1m
  start="2026-04-01T00:00:00Z"
  end="2026-04-01T01:00:00Z"
  http_rate=(sum by (instance) (rate(http_requests_total[5m])))
| SORT http_rate DESC

This works the same way as naming aggregations in STATS, for example STATS avg_cpu = avg(system.cpu.usage).

Index patterns

The index parameter accepts the same patterns as FROM and TS, including wildcards and comma-separated lists. If omitted, it defaults to *, which queries all indices configured with index.mode: time_series. In production, specifying an explicit index pattern avoids scanning unrelated data.

How it works under the hood

The PROMQL command does not run a separate query engine. Instead, PROMQL commands execute inside the ES|QL compute engine, using the same logic as time-series aggregations through the TS source command.

Consider this PromQL query:

PROMQL index=metrics-*
  step=1m
  start="2026-04-01T00:00:00Z"
  end="2026-04-01T01:00:00Z"
  sum by (host.name) (rate(http_requests_total[5m]))

Internally, the PROMQL command translates this into an equivalent ES|QL query using the TS source:

TS metrics-*
| WHERE TRANGE("2026-04-01T00:00:00Z", "2026-04-01T01:00:00Z")
| STATS SUM(RATE(http_requests_total, 5m)) BY TBUCKET(1m), host.name

Both queries produce the same result. The PROMQL command parses the PromQL syntax, resolves functions to their ES|QL equivalents (rate to RATE, sum to SUM, avg_over_time to AVG_OVER_TIME, and so on), and constructs a logical plan that the ES|QL engine executes.

This translation approach has a practical benefit: PromQL queries automatically benefit from all the optimizations in the ES|QL engine, including segment-level parallelism and time series-aware data access patterns.

There are currently 19 time series functions available, covering rates, deltas, derivatives, and various *_over_time aggregations.

Smart defaults that simplify queries

In Prometheus, a PromQL query requires explicit start, end, and step parameters. In Kibana, those are usually determined by the date picker and panel size. The PROMQL command has three features that make queries adapt automatically.

Auto-step

If you omit the step parameter, the command derives it automatically based on the time range and a target bucket count (default: 100). You can also set the target explicitly with buckets=<n>.

PROMQL index=metrics-*
  start="2026-04-01T00:00:00Z"
  end="2026-04-01T01:00:00Z"
  sum by (instance) (rate(http_requests_total[5m]))

With a 1-hour range and the default target of 100 buckets, the step would be 1m, resulting in 60 buckets. This uses the same date-rounding logic as the ES|QL BUCKET function.

Inferred start and end

Kibana adds a time range filter to every ES|QL request via a Query DSL range filter on @timestamp. The PROMQL command extracts those bounds and uses them as start and end when they are not specified in the query. The command picks up the date picker range from the request context without any additional configuration.

Implicit range selectors

In standard PromQL, functions like rate require a range selector: rate(http_requests_total[5m]). The PROMQL command allows omitting the range selector entirely:

PROMQL sum by (instance) (rate(http_requests_total))

When the range selector is absent, the window is determined automatically as max(step, scrape_interval). The scrape_interval defaults to 1m and can be overridden with the scrape_interval parameter if your data has a different collection interval, for example: PROMQL scrape_interval=15s sum(rate(http_requests_total)).

The result

Combining all three defaults, a fully adaptive query in Kibana looks like this:

PROMQL sum(rate(http_requests_total))

This query responds to the date picker, adjusts the step size to the selected time range, and sizes the range selector window accordingly. No manual tuning needed.

Post-processing with ES|QL

Because PROMQL is an ES|QL source command, its output flows into the rest of the ES|QL pipeline. You can filter, sort, enrich, and transform PromQL results using any ES|QL command.

Filter results

PROMQL index=metrics-*
  http_rate=(sum by (instance) (rate(http_requests_total[5m])))
| WHERE http_rate > 100

Sort and limit

PROMQL index=metrics-*
  http_rate=(sum by (instance) (rate(http_requests_total[5m])))
| SORT http_rate DESC
| LIMIT 10

Enrich with a lookup

PROMQL index=metrics-*
  http_rate=(sum by (instance) (rate(http_requests_total[5m])))
| LOOKUP JOIN instance_metadata ON instance

This is something you cannot do in Prometheus. PromQL results are self-contained; there is no way to join them with external data or apply arbitrary post-processing. In Elasticsearch, the PromQL output is just the first stage of a query that can continue with any ES|QL operation.

Current coverage and what's next

In 9.4, the PROMQL command will be available as a tech preview with over 80% query coverage benchmarked against popular Grafana open source dashboards.

The most notable gaps in the current tech preview:

• Group modifiers like on(chip) group_left(chip_name) are not yet supported.
• Binary set operators (or, and, unless) are not yet available.
• Some functions are still missing, including histogram_quantile, predict_linear, and label_join.

These are all planned for upcoming releases. The roadmap includes broader PromQL function and operator coverage, Prometheus-aligned step semantics, and support for native histograms.

Try it

PromQL support is available as a tech preview on Elasticsearch Serverless with no additional configuration. For self-managed clusters, it is available starting with version 9.4.

To try it in Kibana:

1. Go to Dashboards, create a new panel, and select ES|QL as the query type.
2. Enter a PROMQL query, for example: PROMQL index=metrics-* sum by (host.name) (rate(http_requests_total)).
3. The command automatically infers the time range from the Kibana date picker, so no additional parameters are needed.

You can also run PromQL queries in the ES|QL mode of Discover, which shows results in a table and an XY chart. Stay tuned for a full walkthrough of using PromQL in Kibana Dashboards, Discover, and Alerting in a dedicated Kibana blog post.

For the full command reference, including all options and examples, see the PROMQL command documentation.

If you want to try it with a self-managed cluster, check out start-local to get up and running quickly.

If you run into issues or have feedback, open an issue on the Elasticsearch repository.

Mid 2021, the Elasticsearch team embarked on making Elasticsearch a much better fit for metrics. The team created Time Series Data Streams (TSDS), which were released in 8.7 as generally available (GA).

This blog post dives into how TSDS works and how we use it in Elastic Observability, as well as how you can use it for your own metrics.

A quick introduction to TSDS

Time Series Data Streams (TSDS) are built on top of data streams in Elasticsearch that are optimized for time series. To create a data stream for metrics, an additional setting on the data stream is needed. As we are using data streams, first an Index Template has to be created:

PUT _index_template/metrics-laptop
{
  "index_patterns": [
    "metrics-laptop-*"
  ],
  "data_stream": {},
  "priority": 200,
  "template": {
    "settings": {
      "index.mode": "time_series"
    },
    "mappings": {
      "properties": {
        "host.name": {
          "type": "keyword",
          "time_series_dimension": true
        },
        "packages.sent": {
          "type": "integer",
          "time_series_metric": "counter"
        },
        "memory.usage": {
          "type": "double",
          "time_series_metric": "gauge"
        }
      }
    }
  }
}

Let's have a closer look at this template. On the top part, we mark the index pattern with metrics-laptop-*. Any pattern can be selected, but it is recommended to use the data stream naming scheme for all your metrics. The next section sets the "index.mode": "time_series" in combination with making sure it is a data_stream: "data_stream": {}.

Dimensions

Each time series data stream needs at least one dimension. In the example above, host.name is set as a dimension field with "time_series_dimension": true. You can have up to 16 dimensions by default. Not every dimension must show up in each document. The dimensions define the time series. The general rule is to pick fields as dimensions that uniquely identify your time series. Often this is a unique description of the host/container, but for some metrics like disk metrics, the disk id is needed in addition. If you are curious about default recommended dimensions, have a look at this ECS contribution with dimension properties.

Reduced storage and increased query speed

At this point, you already have a functioning time series data stream. Setting the index mode to time series automatically turns on synthetic source. By default, Elasticsearch typically duplicates data three times:

• row-oriented storage (_source field)
• column-oriented storage (doc_values: true for aggregations)
• indices (index: true for filtering and search)

With synthetic source, the _source field is not persisted; instead, it is reconstructed from the doc values. Especially in the metrics use case, there are little benefits to keeping the source.

Not storing it means a significant reduction in storage. Time series data streams sort the data based on the dimensions and the time stamp. This means data that is usually queried together is stored together, which speeds up query times. It also means that the data points for a single time series are stored alongside each other on disk. This enables further compression of the data as the rate at which a counter increases is often relatively constant.

Metric types

But to benefit from all the advantages of TSDS, the field properties of the metrics fields must be extended with the time_series_metric: {type}. Several types are supported — as an example, gauge and counter were used above. Giving Elasticsearch knowledge about the metric type allows Elasticsearch to offer more optimized queries for the different types and reduce storage usage further.

When you create your own templates for data streams under the data stream naming scheme, it is important that you set "priority": 200 or higher, as otherwise the built-in default template will apply.

Ingest a document

Ingesting a document into a TSDS isn't in any way different from ingesting documents into Elasticsearch. You can use the following commands in Dev Tools to add a document, and then search for it and also check out the mappings. Note: You have to adjust the @timestamp field to be close to your current date and time.

# Add a document with `host.name` as the dimension
POST metrics-laptop-default/_doc
{
  # This timestamp neesd to be adjusted to be current
  "@timestamp": "2023-03-30T12:26:23+00:00",
  "host.name": "ruflin.com",
  "packages.sent": 1000,
  "memory.usage": 0.8
}

# Search for the added doc, _source will show up but is reconstructed
GET metrics-laptop-default/_search

# Check out the mappings
GET metrics-laptop-default

If you do search, it still shows _source but this is reconstructed from the doc values. The additional field added above is @timestamp. This is important as it is a required field for any data stream.

Why is this all important for Observability?

One of the advantages of the Elastic Observability solution is that in a single storage engine, all signals are brought together in a single place. Users can query logs, metrics, and traces together without having to jump from one system to another. Because of this, having a great storage and query engine not only for logs but also metrics is key for us.

Usage of TSDS in integrations

With integrations, we give our users an out of the box experience to integrate with their infrastructure and services. If you are using our integrations, eventually you will automatically get all the benefits of TSDS for your metrics assuming you are on version 8.7 or newer.

Currently we are working through the list of our integration packages, add the dimensions, metric type fields and then turn on TSDS for the metrics data streams. What this means is as soon as the package has all properties enabled, the only thing you have to do is upgrade the integration and everything else will happen automatically in the background.

To visualize your time series in Kibana, use Lens, which has native support built in for TSDS.

Learn more

If you switch over to TSDS, you will automatically benefit from all the future improvements Elasticsearch is making for metrics time series, be it more efficient storage, query performance, or new aggregation capabilities. If you want to learn more about how TSDS works under the hood and all available config options, check out the TSDS documentation. What Elasticsearch supports in 8.7 is only the first iteration of the metrics time series in Elasticsearch.

TSDS can be used since 8.7 and will be in more and more of our integrations automatically when integrations are upgraded. All you will notice is lower storage usage and faster queries. Enjoy!

Why do we need alerts?

Logs, metrics, and traces are just the base to build a complete monitoring solution for Kubernetes clusters. Their main goal is to provide debugging information and historical evidence for the infrastructure.

While out-of-the-box dashboards, infrastructure topology, and logs exploration through Kibana are already quite handy to perform ad-hoc analyses, adding notifications and active monitoring of infrastructure allows users to deal with problems detected as early as possible and even proactively take actions to prevent their Kubernetes environments from facing even more serious issues.

How can this be achieved?

By building alerts on top of their infrastructure, users can leverage the data and effectively correlate it to a specific notification, creating a wide range of possibilities to dynamically monitor and observe their Kubernetes cluster.

In this blog post, we will explore how users can leverage Elasticsearch’s search powers to define alerting rules in order to be notified when a specific condition occurs.

SLIs, alerts, and SLOs: Why are they important for SREs?

For site reliability engineers (SREs), the incident response time is tightly coupled with the success of everyday work. Monitoring, alerting, and actions will help to discover, resolve, or prevent issues in their systems.

• An SLA (Service Level Agreement) is an agreement you create with your users to specify the level of service they can expect.
• An SLO (Service Level Objective) is an agreement within an SLA about a specific metric like uptime or response time.
• An SLI (Service Level Indicator) measures compliance with an SLO.

SREs’ day-to-day tasks and projects are driven by SLOs. By ensuring that SLOs are defended in the short term and that they can be maintained in the medium to long term, we lay the basis of a stable working infrastructure.

Having said this, identifying the high-level categories of SLOs is crucial in order to organize the work of an SRE. Then in each category of SLOs, SREs will need the corresponding SLIs that can cover the most important cases of their system under observation. Therefore, the decision of which SLIs we will need demands additional knowledge of the underlying system infrastructure.

One widely used approach to categorize SLIs and SLOs is the Four Golden Signals method. The categories defined are Latency, Traffic, Errors, and Saturation.

A more specific approach is the The RED method developed by Tom Wilkie, who was an SRE at Google and used the Four Golden Signals. The RED method drops the saturation category because this one is mainly used for more advanced cases — and people remember better things that come in threes.

Focusing on Kubernetes infrastructure operators, we will consider the following groups of infrastructure SLIs/SLOs:

• Group 1: Latency of control plane (apiserver,
• Group 2: Resource utilization of the nodes/pods (how much cpu, memory, etc. is consumed)
• Group 3: Errors (errors on logs or events or error count from components, network, etc.)

Creating alerts for a Kubernetes cluster

Now that we have a complete outline of our goal to define alerts based on SLIs/SLOs, we will dive into defining the proper alerting. Alerts can be built using Kibana.

See Elastic documentation.

In this blog, we will define more complex alerts based on complex Elasticsearch queries provided by Watcher’s functionality. Read more about Watcher and how to properly use it in addition to the examples in this blog.

Latency alerts

For this kind of alert, we want to define the basic SLOs for a Kubernetes control plane, which will ensure that the basic control plane components can service the end users without an issue. For instance, facing high latencies in queries against the Kubernetes API Server is enough of a signal that action needs to be taken.

Resource saturation

The next group of alerting will be resource utilization. Node’s CPU utilization or changes in Node’s condition is something critical for a cluster to ensure the smooth servicing of the workloads provisioned to run the applications that end users will interact with.

Error detection

Last but not least, we will define alerts based on specific errors like the network error rate or Pods’ failures like the OOMKilled situation. It’s a very useful indicator for SRE teams to either detect issues on the infrastructure level or just be able to notify developer teams about problematic workloads. One example that we will examine later is having an application running as a Pod and constantly getting restarted because it hits its memory limit. In that case, the owners of this application will need to get notified to act properly.

From Kubernetes data to Elasticsearch queries

Having a solid plan about the alerts that we want to implement, it's time to explore the data we have collected from the Kubernetes cluster and stored in Elasticsearch. For this we will consult the list of the available data fields that are ingested using the Elastic Agent Kubernetes integration (the full list of fields can be found here). Using these fields we can create various alerts like:

• Node CPU utilization
• Node Memory utilization
• BW utilization
• Pod restarts
• Pod CPU/memory utilization

CPU utilization alert

Our first example will use the CPU utilization fields to calculate the Node’s CPU utilization and create an alert. For this alert, we leverage the metrics:

kubernetes.node.cpu.usage.nanocores
kubernetes.node.cpu.capacity.cores.

The following calculation (nodeUsage / 1000000000 ) /nodeCap grouped by node name will give us the CPU utilization of our cluster’s nodes.

The Watcher definition that implements this query can be created with the following API call to Elasticsearch:

curl -X PUT "https://elastic:changeme@localhost:9200/_watcher/watch/Node-CPU-Usage?pretty" -k -H 'Content-Type: application/json' -d'
{
  "trigger": {
    "schedule": {
      "interval": "10m"
    }
  },
  "input": {
    "search": {
      "request": {
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "must": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-10m",
                      "lte": "now",
                      "format": "strict_date_optional_time"
                    }
                  }
                },
                {
                  "bool": {
                    "must": [
                      {
                        "query_string": {
                          "query": "data_stream.dataset: kubernetes.node OR data_stream.dataset: kubernetes.state_node",
                          "analyze_wildcard": true
                        }
                      }
                    ],
                    "filter": [],
                    "should": [],
                    "must_not": []
                  }
                }
              ],
              "filter": [],
              "should": [],
              "must_not": []
            }
          },
          "aggs": {
            "nodes": {
              "terms": {
                "field": "kubernetes.node.name",
                "size": "10000",
                "order": {
                  "_key": "asc"
                }
              },
              "aggs": {
                "nodeUsage": {
                  "max": {
                    "field": "kubernetes.node.cpu.usage.nanocores"
                  }
                },
                "nodeCap": {
                  "max": {
                    "field": "kubernetes.node.cpu.capacity.cores"
                  }
                },
                "nodeCPUUsagePCT": {
                  "bucket_script": {
                    "buckets_path": {
                      "nodeUsage": "nodeUsage",
                      "nodeCap": "nodeCap"
                    },
                    "script": {
                      "source": "( params.nodeUsage / 1000000000 ) / params.nodeCap",
                      "lang": "painless",
                      "params": {
                        "_interval": 10000
                      }
                    },
                    "gap_policy": "skip"
                  }
                }
              }
            }
          }
        },
        "indices": [
          "metrics-kubernetes*"
        ]
      }
    }
  },
  "condition": {
    "array_compare": {
      "ctx.payload.aggregations.nodes.buckets": {
        "path": "nodeCPUUsagePCT.value",
        "gte": {
          "value": 80
        }
      }
    }
  },
  "actions": {
    "log_hits": {
      "foreach": "ctx.payload.aggregations.nodes.buckets",
      "max_iterations": 500,
      "logging": {
        "text": "Kubernetes node found with high CPU usage: {{ctx.payload.key}} -> {{ctx.payload.nodeCPUUsagePCT.value}}"
      }
    }
  },
  "metadata": {
    "xpack": {
      "type": "json"
    },
    "name": "Node CPU Usage"
  }
}

OOMKilled Pods detection and alerting

Another Watcher that we will explore is the one that detects Pods that have been restarted due to an OOMKilled error. This error is quite common in Kubernetes workloads and is useful to detect this early on to inform the team that owns this workload, so they can either investigate issues that could cause memory leaks or just consider increasing the required resources for the workload itself.

This information can be retrieved from a query like the following:

kubernetes.container.status.last_terminated_reason: OOMKilled

Here is how we can create the respective Watcher with an API call:

curl -X PUT "https://elastic:changeme@localhost:9200/_watcher/watch/Pod-Terminated-OOMKilled?pretty" -k -H 'Content-Type: application/json' -d'
{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "must": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-1m",
                      "lte": "now",
                      "format": "strict_date_optional_time"
                    }
                  }
                },
                {
                  "bool": {
                    "must": [
                      {
                        "query_string": {
                          "query": "data_stream.dataset: kubernetes.state_container",
                          "analyze_wildcard": true
                        }
                      },
                      {
                        "exists": {
                          "field": "kubernetes.container.status.last_terminated_reason"
                        }
                      },
                      {
                        "query_string": {
                          "query": "kubernetes.container.status.last_terminated_reason: OOMKilled",
                          "analyze_wildcard": true
                        }
                      }
                    ],
                    "filter": [],
                    "should": [],
                    "must_not": []
                  }
                }
              ],
              "filter": [],
              "should": [],
              "must_not": []
            }
          },
          "aggs": {
            "pods": {
              "terms": {
                "field": "kubernetes.pod.name",
                "order": {
                  "_key": "asc"
                }
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "array_compare": {
      "ctx.payload.aggregations.pods.buckets": {
        "path": "doc_count",
        "gte": {
          "value": 1,
          "quantifier": "some"
        }
      }
    }
  },
  "actions": {
    "ping_slack": {
      "foreach": "ctx.payload.aggregations.pods.buckets",
      "max_iterations": 500,
      "webhook": {
        "method": "POST",
        "url": "https://hooks.slack.com/services/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",
        "body": "{\"channel\": \"#k8s-alerts\", \"username\": \"k8s-cluster-alerting\", \"text\": \"Pod {{ctx.payload.key}} was terminated with status OOMKilled.\"}"
      }
    }
  },
  "metadata": {
    "xpack": {
      "type": "json"
    },
    "name": "Pod Terminated OOMKilled"
  }
}

From Kubernetes data to alerts summary

So far we saw how we can start from plain Kubernetes fields, use them in ES queries, and build Watchers and alerts on top of them.

One can explore more possible data combinations and build queries and alerts following the examples we provided here. A full list of alerts is available, as well as a basic scripted way of installing them.

Of course, these examples come with simple actions defined that only log messages into the Elasticsearch logs. However, one can use more advanced and useful outputs like Slack’s webhooks:

"actions": {
    "ping_slack": {
      "foreach": "ctx.payload.aggregations.pods.buckets",
      "max_iterations": 500,
      "webhook": {
        "method": "POST",
        "url": "https://hooks.slack.com/services/T04SW3JHXasdfasdfasdfasdfasdf",
        "body": "{\"channel\": \"#k8s-alerts\", \"username\": \"k8s-cluster-alerting\", \"text\": \"Pod {{ctx.payload.key}} was terminated with status OOMKilled.\"}"
      }
    }
  }

The result would be a Slack message like the following:

Next steps

In our next steps, we would like to make these alerts part of our Kubernetes integration, which would mean that the predefined alerts would be installed when users install or enable the Kubernetes integration. At the same time, we plan to implement some of these as Kibana’s native SLIs, providing the option to our users to quickly define SLOs on top of the SLIs through a nice user interface. If you’re interested to learn more about these, follow the public GitHub issues for more information and feel free to provide your feedback:

• https://github.com/elastic/package-spec/issues/484
• https://github.com/elastic/kibana/issues/150050

For those who are eager to start using Kubernetes alerting today, here is what you need to do:

1. Make sure that you have an Elastic cluster up and running. The fastest way to deploy your cluster is to spin up a free trial of Elasticsearch Service.
2. Install the latest Elastic Agent on your Kubernetes cluster following the respective documentation.
3. Install our provided alerts that can be found at https://github.com/elastic/integrations/tree/main/packages/kubernetes/docs or at https://github.com/elastic/k8s-integration-infra/tree/main/scripts/alerting.

Of course, if you have any questions, remember that we are always happy to help on the Discuss forums.

Whether you've added a new integration, set up an OpenTelemetry pipeline, or configured a custom agent for your infrastructure, you need to see what's landing in the cluster before you build dashboards, alerts, or SLOs on top of it. Discover gives you that view: the metrics in a time series stream, each rendered as a time series chart for your desired time range. No dashboard to build, no exploratory queries to write. Just the raw picture of what you have.

Discover your data streams

In the left navigation under Observability, open Streams. That page lists every data stream in your cluster, wherever it comes from: integrations, OpenTelemetry pipelines, custom agents, and similar sources. Each source you monitor (Docker, Kubernetes, Nginx, and so on) produces one or more data streams. Here you can see exactly what streams exist and what you can build on.

Open a stream to see its detail page.

On the top left, a "Time series" badge means the stream is a time series stream (optimized for metrics and more efficient); if the badge isn't there, the stream is regular. Click View in Discover in the top right to open Discover with the right query for that stream. The query depends on the stream type:

• TS (time series): TS is an ES|QL source command that selects a time series data stream and enables time series aggregation functions (such as RATE or AVG_OVER_TIME). When Discover recognizes metrics data from time series metrics data streams (for example streams whose names match metrics-*), it shows each metric as a chart. See the ES|QL TS command documentation for the full reference.
• FROM (regular, document-based streams): use for document-style queries. Discover shows documents in a table rather than the per-metric chart grid you get with time series metrics streams.

Because our example is a time series stream, Discover opens with:

TS metrics-docker.cpu-default

See all your metrics, automatically visualized

This is where it gets useful. Instead of a table of documents, Discover shows you the metrics in that stream, each rendered as a time series chart for the selected time range. No configuration needed. This capability, metrics in Discover, is currently in technical preview.

Each metric (docker.cpu.total.pct, docker.cpu.system.pct, docker.cpu.user.pct, and others) appears with a chart that shows its behavior over time. Discover recognizes different metric types and renders them accordingly: gauges as averages, counters as rates, and histograms as P95 distributions. You get an instant, at-a-glance view of what's being collected and whether the values look reasonable.

When you're onboarding a new source, that removes the guesswork: which metrics are active, which have data, what the values look like. You can confirm coverage and sanity-check the pipeline before you rely on that data for dashboards or alerting.

Iterate quickly

From here, you can adjust to get the view you need:

Change the time range. The default 15-minute window might catch a quiet period and make healthy data look flat. Expanding to 1 hour or more reveals patterns you care about: periodic spikes from batch jobs, daily traffic curves, or the ramp-up after a new deployment. Picking the right window matters when you're validating that a new pipeline or integration is behaving as expected.

Switch data streams. You don't need to go back to the Streams page to explore another data source. Update the query to a different data stream, or use a pattern like metrics-docker.* to see metrics across all your Docker data streams at once: CPU, memory, network, disk I/O, all in one view.

Search for specific metrics. With many metrics in a stream, the search on the top right of the grid lets you filter by name. Need to confirm that memory limits or request rates are present? Type the metric name and you either find it or confirm it's missing, so you can fix the pipeline or agent before you depend on that metric elsewhere.

Validate at a glance

The automatic visualizations also serve as a health check for data ingestion:

• Data is flowing: charts show recent, continuous values, not gaps or stale data.
• Values are reasonable: CPU in expected ranges, memory tracking activity, network I/O reflecting traffic.
• Coverage is what you expect: if you enabled Docker monitoring but don't see network I/O metrics, the agent policy or module likely needs a change.

This kind of quick validation replaces manual doc checks, mapping inspection, and one-off exploratory queries. You get a clear picture of what's in the stream before you wire it into dashboards, alerts, or SLOs. Once you've confirmed the data looks healthy, you can add panels to dashboards or use it for alerting and SLOs.

In this blog post, we are excited to share the significant strides we have made in improving the UI performance of Elastic APM. Maintaining a snappy user interface can be a challenge when interactively summarizing the massive amounts of data needed to provide an overview of the performance for an entire enterprise-scale service inventory. We want to assure our customers that we have listened, taken action, and made notable architectural changes to elevate the scalability and maturity of our solution.

Architectural enhancements

Our journey began back in the 7.x series where we noticed that doing ad-hoc aggregations on raw transaction data put Elasticsearch^® under a lot of pressure in large-scale environments. Since then, we’ve begun to pre-aggregate the transactions into transaction metrics during ingestion. This has helped to keep the performance of the UI relatively stable. Regardless of how busy the monitored application is and how many transaction events it is creating, we’re just querying pre-aggregated metrics that are stored at a constant rate. We’ve enabled the metrics-powered UI by default in 7.15.

However, when showing an inventory of a large number of services over large time ranges, the number of metric data points that need to be aggregated can still be large enough to cause performance issues. We also create a time series for each distinct set of dimensions. The dimensions include metadata, such as the transaction name and the host name. Our documentation includes a full list of all available dimensions. If there’s a very high number of unique transaction names, which could be a result of improper instrumentation (see docs for more details), this will create a lot of individual time series that will need to be aggregated when requesting a summary of the service’s overall performance. Global labels that are added to the APM Agent configuration are also added as dimensions to these metrics, and therefore they can also impact the number of time series. Refer to the FAQs section below for more details.

Within the 8.7 and 8.8 releases, we’ve addressed these challenges with the following architectural enhancements that aim to reduce the number of documents Elasticsearch needs to search and aggregate on-the-fly, resulting in faster response times:

• Pre-aggregation of transaction metrics into service metrics. Instead of aggregating all distinct time series that are created for each individual transaction name on-the-fly for every user request, we’re already pre-aggregating a summary time series for each service during data ingestion. Depending on how many unique transaction names the services have, this reduces the number of documents Elasticsearch needs to look up and aggregate by a factor of typically 10–100. This is particularly useful for the service inventory and the service overview pages.
• Pre-aggregation of all metrics into different levels of granularity. The APM UI chooses the most appropriate level of granularity, depending on the selected time range. In addition to the metrics that are stored at a 1-minute granularity, we’re also summarizing and storing metrics at a 10-minute and 60-minute granularity level. For example, when looking at a 7-day period, the 60-minute data stream is queried instead of the 1-minute one, resulting in 60x fewer documents for Elasticsearch to examine. This makes sure that all graphs are rendered quickly, even when looking at larger time ranges.
• Safeguards on the number of unique transactions per service for which we are aggregating metrics. Our agents are designed to keep the cardinality of the transaction name low. But in the wild, we’ve seen some services that have a huge amount of unique transaction names. This used to cause performance problems in the UI because APM Server would create many time series that the UI needed to aggregate at query time. In order to protect APM Server from running out of memory when aggregating a large number of time series for each unique transaction name, metrics were published without aggregating when limits for the number of time series were reached. This resulted in a lot of individual metric documents that needed to be aggregated at query time. To address the problem, we've introduced a system where we aggregate metrics in a dedicated overflow bucket for each service when limits are reached. Refer to our documentation for more details.

The exact factor of the document count reduction depends on various conditions. But to get a feeling for a typical scenario, if your services, on average, have 10 instances, no instance-specific global labels, 100 unique transaction names each, and you’re looking at time ranges that can leverage the 60m granularity, you’d see a reduction of documents that Elasticsearch needs to aggregate by a factor of 180,000 (10 instances x 100 transaction names x 60m x 3 because we’re also collapsing the event.outcome dimension). While the response times of Elasticsearch aggregations isn’t exactly scaling linearly with the number of documents, there is a strong correlation.

FAQs

When upgrading to the latest version, will my old data also load faster?

Updating to 8.8 doesn’t immediately make the UI faster. Because the improvements are powered by pre-aggregations that APM Server is doing during ingestion, only new data will benefit from it. For that reason, you should also make sure to update APM Server as well. The UI can still display data that was ingested using an older version of the stack.

If the UI is based on metrics, can I still slice and dice using custom labels?

High cardinality analysis is a big strength of Elastic Observability, and this focus on pre-aggregated metrics does not compromise that in any way.

The UI implements a sophisticated fallback mechanism that uses service metrics, transaction metrics, or raw transaction events, depending on which filters are applied. We’re not creating metrics for each user.id, for example. But you can still filter the data by user.id and the UI will then use raw transaction events. Chances are that you’re looking at a narrow slice of data when filtering by a dimension that is not available on the pre-aggregated metrics, therefore aggregations on the raw data are typically very fast.

Note that all global labels that are added to the APM agent configuration are part of the dimension of the pre-aggregated metrics, with the exception of RUM (see more details in this issue).

Can I use the pre-aggregated metrics in custom dashboards?

Yes! If you use Lens and select the "APM" data view, you can filter on either metricset.name:service_transaction or metricset.name:transaction, depending on the level of detail you need. Transaction latency is captured in transaction.duration.histogram, and successful outcomes and failed outcomes are stored in event.success_count. If you don't need a distribution of values, you can also select the transaction.duration.summary field for your metric aggregations, which should be faster. If you want to calculate the failure rate, here's a Lens formula: 1 - (sum(event.success_count) / count(event.success_count)). Note that the only granularity supported here is 1m.

Do the additional metrics have an impact on the storage?

While we’re storing more metrics than before, and we’re storing all metrics in different levels of granularity, we were able to offset that by enabling synthetic source for all metric data streams. We’ve even increased the default retention for the metrics in the coarse-grained granularity levels, so that the 60m rollup data streams are now stored for 390 days. Please consult our documentation for more information about the different metric data streams.

Are there limits on the amount of time series that APM Server can aggregate?

APM Server performs pre-aggregations in memory, which is fast, but consumes a considerable amount of memory. There are limits in place to protect APM Server from running out of memory, and from 8.7, most of them scale with available memory by default, meaning that allocating more memory to APM Server will allow it to handle more unique pre-aggregation groups like services and transactions. These limits are described in APM Server Data Model docs.

On the APM Server roadmap, we have plans to move to a LSM-based approach where pre-aggregations are performed with the help of disks in order to reduce memory usage. This will enable APM Server to scale better with the input size and cardinality.

A common pitfall when working with pre-aggregations is to add instance-specific global labels to APM agents. This may exhaust the aggregation limits and cause metrics to be aggregated under the overflow bucket instead of the corresponding service. Therefore, make sure to follow the best practice of only adding a limited set of global labels to a particular service.

Validation

To validate the effectiveness of the new architecture, and to ensure that the accuracy of the data is not negatively affected, we prepared a test environment where we generated 35K+ transactions per minute in a timespan of 14 days resulting in approximately 850 million documents.

We’ve tested the queries that power our service inventory, the service overview, and the transaction details using different time ranges (1d, 7d, 14d). Across the board, we’ve seen orders of magnitude improvements. Particularly, queries across larger time ranges that benefit from using the coarse-grained metrics in addition to the pre-aggregated service metrics saw incredible reductions of the response time.

We’ve also validated that there’s no loss in accuracy when using the more coarse-grained metrics for larger time ranges.

Every environment will behave a bit differently, but we’re confident that the impressive improvements in response time will translate well to setups of even bigger scale.

Planned improvements

As mentioned in the FAQs section, the number of time series for transaction metrics can grow quickly, as it is the product of multiple dimensions. For example, given a service that runs on 100 hosts and has 100 transaction names that each have 4 transaction results, APM Server needs to track 40,000 (100 x 100 x 4) different time series for that service. This would even exceed the maximum per-service limit of 32,000 for APM Servers with 64GB of main memory.

As a result, the UI will show an entry for “Remaining Transactions” in the Service overview page. This tracks the transaction metrics for a service once it hits the limit. As a result, you may not see all transaction names of your service. It may also be that all distinct transaction names are listed, but that the transaction metrics for some of the instances of that service are combined in the “Remaining Transactions” category.

We’re currently considering restructuring the dimensions for the metrics to avoid that the combination of the dimensions for transaction name and service instance-specific dimensions (such as the host name) lead to an explosion of time series. Stay tuned for more details.

Conclusion

The architectural improvements we’ve delivered in the past releases provide a step-function in terms of the scalability and responsiveness of our UI. Instead of having to aggregate massive amounts of data on-the-fly as users are navigating through the user interface, we pre-aggregate the results for the most common queries as data is coming in. This ensures we have the answers ready before users have even asked their most frequently asked questions, while still being able to answer ad-hoc questions.

We are excited to continue supporting our community members as they push boundaries on their growth journey, providing them with a powerful and mature platform that can effortlessly handle the demands of the largest workloads. Elastic is committed to its mission to enable everyone to find the answers that matter. From all data. In real time. At scale.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Elastic donated Elastic Common Schema (ECS) to OpenTelemetry and is actively working to converge it with semantic conventions. In the meantime, we are dedicated to support our users by ensuring they don’t have to navigate different standards. Our goal is to provide a seamless end-to-end experience while using OpenTelemetry with our application and infrastructure monitoring solutions. This commitment allows users to benefit from the best of both worlds without any friction.

In this blog, we explore how to use the OpenTelemetry (OTel) collector to capture core system metrics from various sources such as AWS EC2, Google Compute, Kubernetes clusters, and individual systems running Linux or MacOS.

Powering Infrastructure UIs with Two Ingest Paths

Elastic users who wish to have OpenTelemetry as their data collection mechanism can now monitor the health of the hosts where the OpenTelemetry collector is deployed using the Hosts and Inventory UIs available in Elastic Observability.

Elastic offers two distinct ingest paths to power Infrastructure UIs: the ElasticsearchExporter Ingest Path and the OTLP Exporter Ingest Path.

ElasticsearchExporter Ingest Path:

The hostmetrics receiver in OpenTelemetry collects system-level metrics such as CPU, memory, and disk usage from the host machine in OTel Schema. The ElasticsearchExporter ingest path leverages the Hostmetrics Receiver to generate host metrics in the OTel schema. We've developed the ElasticInfraMetricsProcessor, which utilizes the opentelemetry-lib to convert these metrics into a format that Elastic UIs understand.

For example, the system.network.io OTel metric includes a direction attribute with values receive or transmit. These correspond to system.network.in.bytes and system.network.out.bytes, respectively, within Elastic.

The processor then forwards these metrics to the Elasticsearch Exporter, now enhanced to support exporting metrics in ECS mode. The exporter sends the metrics to an Elasticsearch endpoint, lighting up the Infrastructure UIs with insightful data.

To utilize this path, you can deploy the collector from the Elastic Collector Distro, available here.

An example collector config for this Ingest Path:

receivers:
  hostmetrics:
    collection_interval: 10s
    scrapers:
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
          system.cpu.logical.count:
            enabled: true
      memory:
        metrics:
          system.memory.utilization:
            enabled: true
      process:
        metrics:
          process.open_file_descriptors:
            enabled: true
          process.memory.utilization:
            enabled: true
          process.disk.operations:
            enabled: true
      network:
      processes:
      load:
      disk:
      filesystem:

processors:
  resourcedetection/system:
    detectors: ["system", "ec2"]
  elasticinframetrics:

exporters:  
  logging:
    verbosity: detailed
  elasticsearch/metrics: 
    endpoints: <elasticsearch_endpoint>
    api_key: <api_key>
    mapping:
      mode: ecs

service:
  pipelines:
    metrics/host:
      receivers: [hostmetrics]
      processors: [resourcedetection/system, elasticinframetrics]
      exporters: [logging, elasticsearch/ metrics]

The Elastic exporter path is ideal for users who would prefer using the custom Elastic Collector Distro. This path includes the ElasticInfraMetricsProcessor, which sends data to Elasticsearch via Elasticsearch exporter.

OTLP Exporter Ingest Path:

In the OTLP Exporter Ingest path, the hostmetrics receiver collects system-level metrics such as CPU, memory, and disk usage from the host machine in OTel Schema. These metrics are sent to the OTLP Exporter, which forwards them to the APM Server endpoint. The APM Server, using the same opentelemetry-lib, converts these metrics into a format compatible with Elastic UIs. Subsequently, the APM Server pushes the metrics to Elasticsearch, powering the Infrastructure UIs.

An example collector configuration for the APM Ingest Path

receivers:
  hostmetrics:
    collection_interval: 10s
    scrapers:
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
          system.cpu.logical.count:
            enabled: true
      memory:
        metrics:
          system.memory.utilization:
            enabled: true
      process:
        metrics:
          process.open_file_descriptors:
            enabled: true
          process.memory.utilization:
            enabled: true
          process.disk.operations:
            enabled: true
      network:
      processes:
      load:
      disk:
      filesystem:

processors:
  resourcedetection/system:
    detectors: ["system"]
    system:
      hostname_sources: ["os"]

exporters:
  otlphttp:
    endpoint: <mis_endpoint>
    tls:
      insecure: false
    headers:
      Authorization: <api_key_>
  logging:
    verbosity: detailed

service:
  pipelines:
    metrics/host:
      receivers: [hostmetrics]
      processors: [resourcedetection/system]
      exporters: [logging, otlphttp]

The OTLP Exporter Ingest path can help existing users who are already using Elastic APM and want to see the Infrastructure UIs populated as well. These users can use the default OpenTelemetry Collector.

A glimpse of the Infrastructure UIs

The Infrastructure UIs showcase both Host and Kubernetes level views. Below are some of the glimpses of the UIs

The Hosts Overview UI

The Hosts Inventory UI

The Process-related Details of the Host

The Kubernetes Inventory UI

Pod level Metrics

Our next step is to create Infrastructure UIs powered by native OTel data, with dedicated OTel dashboards that run on this native data.

Conclusion

Elastic's integration with OpenTelemetry simplifies the observability landscape and while we are diligently working to align ECS with OpenTelemetry’s semantic conventions, our immediate priority is to support our users by simplifying their experience. With this added support, we aim to deliver a seamless, end-to-end experience for those using OpenTelemetry with our application and infrastructure monitoring solutions. We are excited to see how our users will leverage these capabilities to gain deeper insights into their systems.

Prometheus is primarily built for short-term metric storage, typically retaining data in-memory or on local disk storage, with a focus on real-time monitoring and alerting rather than historical analysis. While it offers valuable insights into current metric values and trends, it may pose economic challenges and fall short of the robust functionalities and capabilities necessary for in-depth historical analysis, long-term trend detection, and forecasting. This is particularly evident in large environments with a substantial number of targets or high data ingestion rates, where metric data accumulates rapidly.

Numerous organizations assess their unique needs and explore avenues to augment their Prometheus monitoring and observability capabilities. One effective approach is integrating Prometheus with Elastic®. In this blog post, we will showcase the integration of Prometheus with Elastic, emphasizing how Elastic elevates metrics monitoring through extensive historical analytics, anomaly detection, and forecasting, all in a cost-effective manner.

Integrate Prometheus with Elastic seamlessly

Organizations that have configured their cloud-native applications to expose metrics in Prometheus format can seamlessly transmit the metrics to Elastic by using Prometheus integration. Elastic enables organizations to monitor their metrics in conjunction with all other data gathered through Elastic's extensive integrations.

Go to Integrations and find the Prometheus integration.

To gather metrics from Prometheus servers, the Elastic Agent is employed, with central management of Elastic agents handled through the Fleet server.

After enrolling the Elastic Agent in the Fleet, users can choose from the following methods to ingest Prometheus metrics into Elastic.

1. Prometheus collectors

The Prometheus collectors connect to the Prometheus server and pull metrics or scrape metrics from a Prometheus exporter.

2. Prometheus queries

The Prometheus queries execute specific Prometheus queries against Prometheus Query API.

3. Prometheus remote-write

The Prometheus remote_write can receive metrics from a Prometheus server that has configured the remote_write setting.

After your Prometheus metrics are ingested, you have the option to visualize your data graphically within the Metrics Explorer and further segment it based on labels, such as hosts, containers, and more.

You can also query your metrics data in Discover and explore the fields of your individual documents within the details panel.

Storing historical metrics with Elastic’s data tiering mechanism

By exporting Prometheus metrics to Elasticsearch, organizations can extend the retention period and gain the ability to analyze metrics historically. Elastic optimizes data storage and access based on the frequency of data usage and the performance requirements of different data sets. The goal is to efficiently manage and store data, ensuring that it remains accessible when needed while keeping storage costs in check.

After ingesting Prometheus metrics data, you have various retention options. You can set the duration for data to reside in the hot tier, which utilizes high IO hardware (SSD) and is more expensive. Alternatively, you can move the Prometheus metrics to the warm tier, employing cost-effective hardware like spinning disks (HDD) while maintaining consistent and efficient search performance. The cold tier mirrors the infrastructure of the warm tier for primary data but utilizes S3 for replica storage. Elastic automatically recovers replica indices from S3 in case of node or disk failure, ensuring search performance comparable to the warm tier while reducing disk cost.

The frozen tier allows direct searching of data stored in S3 or an object store, without the need for rehydration. The purpose is to further reduce storage costs for Prometheus metrics data that is less frequently accessed. By moving historical data into the frozen tier, organizations can optimize their storage infrastructure, ensuring that the recent, critical data remains in higher-performance tiers while less frequently accessed data is stored economically in the frozen tier. This way, organizations can perform historical analysis and trend detection, identify patterns and make informed decisions, and maintain compliance with regulatory standards in a cost-effective manner.

An alternative way to store your cloud-native metrics more efficiently is to use Elastic Time Series Data Stream (TSDS). TSDS can store your metrics data more efficiently with ~70% less disk space than a regular data stream. The downsampling functionality will further reduce the storage required by rolling up metrics within a fixed time interval into a single summary metric. This not only assists organizations in cutting down on storage expenses for metric data but also simplifies the metric infrastructure, making it easier for users to correlate metrics with logs and traces through a unified interface.

Advanced analytics

Besides Metrics Explorer and Discover, Elasticsearch® provides more advanced analytics capabilities and empowers organizations to gain deeper, more valuable insights into their Prometheus metrics data.

Out of the box, Prometheus integration provides a default overview dashboard.

From Metrics Explorer or Discover, users can also easily edit their Prometheus metrics visualization in Elastic Lens or create new visualizations from Lens.

Elastic Lens enables users to explore and visualize data intuitively through dynamic visualizations. This user-friendly interface eliminates the need for complex query languages, making data analysis accessible to a broader audience. Elasticsearch also offers other powerful visualization methods with aggregations and filters, enabling users to perform advanced analytics on their Prometheus metrics data, including short-term and historical data. To learn more, check out the how-to series: Kibana.

Anomaly detection and forecasting

When analyzing data, maintaining a constant watch on the screen is simply not feasible, especially when dealing with millions of time series of Prometheus metrics. Engineers frequently encounter the challenge of differentiating normal from abnormal data points, which involves analyzing historical data patterns — a process that can be exceedingly time consuming and often exceeds human capabilities. Thus, there is a pressing need for a more intelligent approach to detect anomalies efficiently.

Setting up alerts may seem like an obvious solution, but relying solely on rule-based alerts with static thresholds can be problematic. What's normal on a Wednesday at 9:00 a.m. might be entirely different from a Sunday at 2:00 a.m. This often leads to complex and hard-to-maintain rules or wide alert ranges that end up missing crucial issues. Moreover, as your business, infrastructure, users, and products evolve, these fixed rules don't keep up, resulting in lots of false positives or, even worse, important issues slipping through the cracks without detection. A more intelligent and adaptable approach is needed to ensure accurate and timely anomaly detection.

Elastic's machine learning anomaly detection excels in such scenarios. It automatically models the normal behavior of your Prometheus data, learning trends, and identifying anomalies, thereby reducing false positives and improving mean time to resolution (MTTR). With over 13 years of development experience in this field, Elastic has emerged as a trusted industry leader.

The key advantage of Elastic's machine learning anomaly detection lies in its unsupervised learning approach. By continuously observing real-time data, it acquires an understanding of the data's behavior over time. This includes grasping daily and weekly patterns, enabling it to establish a normalcy range of expected behavior. Behind the scenes, it constructs statistical models that allow accurate predictions, promptly identifying any unexpected variations. In cases where emerging data exhibits unusual trends, you can seamlessly integrate with alerting systems, operationalizing this valuable insight.

Machine learning's ability to project into the future, forecasting data trends one day, a week, or even a month ahead, equips engineers not only with reporting capabilities but also with pattern recognition and failure prediction based on historical Prometheus data. This plays a crucial role in maintaining mission-critical workloads, offering organizations a proactive monitoring approach. By foreseeing and addressing issues before they escalate, organizations can avert downtime, cut costs, optimize resource utilization, and ensure uninterrupted availability of their vital applications and services.

Creating a machine learning job for your Prometheus data is a straightforward task with a few simple steps. Simply specify the data index and set the desired time range in the single metric view. The machine learning job will then automatically process the historical data, building statistical models behind the scenes. These models will enable the system to predict trends and identify anomalies effectively, providing valuable and actionable insights for your monitoring needs.

In essence, Elastic machine learning empowers us to harness the capabilities of data scientists and effectively apply them in monitoring Prometheus metrics. By seamlessly detecting anomalies and predicting potential issues in advance, Elastic machine learning bridges the gap and enables IT professionals to benefit from the insights derived from advanced data analysis. This practical and accessible approach to anomaly detection equips organizations with a proactive stance toward maintaining the reliability of their systems.

Try it out

Start a free trial on Elastic Cloud and ingest your Prometheus metrics into Elastic. Enhance your Prometheus monitoring with Elastic Observability. Stay ahead of potential issues with advanced AI/ML anomaly detection and prediction capabilities. Eliminate data silos, reduce costs, and enhance overall response efficiency.

Elevate your monitoring capabilities with Elastic today!

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

OpenTelemetry recently published a blog on how to do Autodiscovery based on Kubernetes Pods' annotations with the OpenTelemetry Collector.

In this blog post, we will talk about how to use this Kubernetes-related feature of the OpenTelemetry Collector, which is already available with the Elastic Distribution of the OpenTelemetry (EDOT) Collector.

In addition to this feature, at Elastic, we heavily invest in making OpenTelemetry the best, standardized ingest solution for Observability. You might already have seen us focusing on:

• Semantic Conventions standardization

• significant log collection improvements

• various other topics around instrumentation

• profiling

Let's walk you through a hands-on journey using the EDOT Collector covering various use cases you might encounter in the real world, highlighting the capabilities of this powerful feature.

Configuring EDOT Collector

The Collector’s configuration is not our main focus here, since based on the nature of this feature it is minimal, letting workloads define how they should be monitored.

To illustrate the point, here is the Collector configuration snippet that enables the feature for both logs and metrics:

receivers:
    receiver_creator/metrics:
      watch_observers: [k8s_observer]
      discovery:
        enabled: true
      receivers:

    receiver_creator/logs:
      watch_observers: [k8s_observer]
      discovery:
        enabled: true
      receivers:

You can include the above in the EDOT’s Collector configuration, specifically the receivers’ section.

Since logs collection in our examples will happen from the discovery feature make sure that the static filelog receiver configuration block is removed and its preset is disabled (i.e. set to false) to avoid having log duplication.

Make sure that the receiver creator is properly added in the pipelines for logs (in addition to removing the filelog receiver completely) and metrics respectively.

Ensure that k8sobserver is enabled as part of the extensions:

extensions:
  k8s_observer:
    observe_nodes: true
    observe_services: true
    observe_ingresses: true

// ...

service:
  extensions: [k8s_observer]

Last but not least, ensure the log files' volume is mounted properly:

volumeMounts:
 - name: varlogpods
   mountPath: /var/log/pods
   readOnly: true

volumes:
  - name: varlogpods
    hostPath:
      path: /var/log/pods

Once the configuration is ready follow the Kubernetes quickstart guides on how to deploy the EDOT Collector. Make sure to replace the values.yaml file linked in the quickstart guide with the file that includes the above-described modifications.

Collecting Metrics from Moving Targets Based on Their Annotations

In this example, we have a Deployment with a Pod spec that consists of two different containers. One container runs a Redis server, while the other runs an NGINX server. Consequently, we want to provide different hints for each of these target containers.

The annotation-based discovery feature supports this, allowing us to specify metrics annotations per exposed container port.

Here is how the complete spec file looks:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-conf
data:
  nginx.conf: |
    user  nginx;
    worker_processes  1;
    error_log  /dev/stderr warn;
    pid        /var/run/nginx.pid;
    events {
      worker_connections  1024;
    }
    http {
      include       /etc/nginx/mime.types;
      default_type  application/octet-stream;

      log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for"';
      access_log  /dev/stdout main;
      server {
          listen 80;
          server_name localhost;

          location /nginx_status {
              stub_status on;
          }
      }
      include /etc/nginx/conf.d/*;
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-deployment
  labels:
    app: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
      annotations:
        # redis container port hints
        io.opentelemetry.discovery.metrics.6379/enabled: "true"
        io.opentelemetry.discovery.metrics.6379/scraper: redis
        io.opentelemetry.discovery.metrics.6379/config: |
          collection_interval: "20s"
          timeout: "10s"

        # nginx container port hints
        io.opentelemetry.discovery.metrics.80/enabled: "true"
        io.opentelemetry.discovery.metrics.80/scraper: nginx
        io.opentelemetry.discovery.metrics.80/config: |
          endpoint: "http://`endpoint`/nginx_status"
          collection_interval: "30s"
          timeout: "20s"
    spec:
      volumes:
      - name: nginx-conf
        configMap:
          name: nginx-conf
          items:
            - key: nginx.conf
              path: nginx.conf
      containers:
        - name: webserver
          image: nginx:latest
          ports:
            - containerPort: 80
              name: webserver
          volumeMounts:
            - mountPath: /etc/nginx/nginx.conf
              readOnly: true
              subPath: nginx.conf
              name: nginx-conf
        - image: redis
          imagePullPolicy: IfNotPresent
          name: redis
          ports:
            - name: redis
              containerPort: 6379
              protocol: TCP

When this workload is deployed, the Collector will automatically discover it and identify the specific annotations. After this, two different receivers will be started, each one responsible for each of the target containers.

Collecting Logs from Multiple Target Containers

The annotation-based discovery feature also supports log collection based on the provided annotations. In the example below, we again have a Deployment with a Pod consisting of two different containers, where we want to apply different log collection configurations. We can specify annotations that are scoped to individual container names:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox-logs-deployment
  labels:
    app: busybox
spec:
  replicas: 1
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
      annotations:
        io.opentelemetry.discovery.logs.lazybox/enabled: "true"
        io.opentelemetry.discovery.logs.lazybox/config: |
          operators:
            - id: container-parser
              type: container
            - id: some
              type: add
              field: attributes.tag
              value: hints-lazybox
        io.opentelemetry.discovery.logs.busybox/enabled: "true"
        io.opentelemetry.discovery.logs.busybox/config: |
          operators:
            - id: container-parser
              type: container
            - id: some
              type: add
              field: attributes.tag
              value: hints-busybox
    spec:
      containers:
        - name: busybox
          image: busybox
          args:
            - /bin/sh
            - -c
            - while true; do echo "otel logs from busybox at $(date +%H:%M:%S)" && sleep 5s; done
        - name: lazybox
          image: busybox
          args:
            - /bin/sh
            - -c
            - while true; do echo "otel logs from lazybox at $(date +%H:%M:%S)" && sleep 25s; done

The above configuration enables two different filelog receiver instances, each applying a unique parsing configuration. This is handy when we know how to parse specific technology logs, such as Apache server access logs.

Combining Both Metrics and Logs Collection

In our third example, we illustrate how to define both metrics and log annotations on the same workload. This allows us to collect both signals from the discovered workload. Below is a Deployment with a Pod consisting of a Redis server and a BusyBox container that performs dummy log writing. We can target annotations to the port and container levels to collect metrics from the Redis server using the Redis receiver, and logs from the BusyBox using the filelog receiver. Here’s how:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-deployment
  labels:
    app: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
      annotations:
        io.opentelemetry.discovery.metrics.6379/enabled: "true"
        io.opentelemetry.discovery.metrics.6379/scraper: redis
        io.opentelemetry.discovery.metrics.6379/config: |
          collection_interval: "20s"
          timeout: "10s"

        io.opentelemetry.discovery.logs.busybox/enabled: "true"
        io.opentelemetry.discovery.logs.busybox/config: |
          operators:
            - id: container-parser
              type: container
            - id: some
              type: add
              field: attributes.tag
              value: hints
    spec:
      containers:
        - image: redis
          imagePullPolicy: IfNotPresent
          name: redis
          ports:
            - name: redis
              containerPort: 6379
              protocol: TCP
        - name: busybox
          image: busybox
          args:
            - /bin/sh
            - -c
            - while true; do echo "otel logs at $(date +%H:%M:%S)" && sleep 15s; done

Explore and analyse data coming from dynamic targets in Elastic

Once the target Pods are discovered and the Collector has started collecting telemetry data from them, we can then explore this data in Elastic. In Discover we can search for Redis and NGINX metrics as well as logs collected from the Busybox container. Here is how it looks like:

Summary

The examples above showcase how users of our OpenTelemetry Collector can take advantage of this new feature — one we played a major role in developing.

For this, we leveraged our years of experience with similar features already supported in Metricbeat, Filebeat, and Elastic-Agent. This makes us extremely happy and confident, as it closes the feature gap between Elastic's specific monitoring agents and the OpenTelemetry Collector — making it even better.

Interested in learning more? Visit the documentation and give it a try by following our EDOT quickstart guide.

In this blog, we’ll show how these ES|QL variable controls turn dashboards into interactive investigation tools, and how to set them up to uncover problems that averages were hiding. By selecting a value in a control, every panel using that variable adapts.

The dashboard

This is a custom "Infrastructure Overview" dashboard monitoring 10 hosts across 3 AWS regions using OpenTelemetry host metrics. Four line charts (CPU, Memory, Disk, Load average) and ES|QL variable controls at the top.

With the default dashboard controls (AVG aggregation, region breakdown, 15-minute buckets, all hosts selected), everything looks healthy. Smooth diurnal cycles across all three regions.

But there is a problem hiding in this view.

The problem with fixed queries

A fixed chart query hardcodes decisions that need to change during an investigation:

• The aggregation function (AVG, MAX, MIN, MEDIAN)
• The dimension used to slice the data (host, region, availability zone)
• Which hosts are included or excluded
• The time bucket interval (1m, 5m, 15m, 1h)

With those baked in, every change means editing queries across multiple panels.

ES|QL variable controls

ES|QL variable controls inject user-selected values into queries at runtime. Two types:

• Value controls (?variable): replace a value in the query, such as a time interval or a list of hostnames
• Structure controls (??variable): replace a function name or field name, such as the aggregation function or the dimension used to slice data

One query pattern, reused across all panels.

The query

The original static CPU query looks like this:

TS metrics-hostmetricsreceiver.otel-default
| WHERE system.cpu.utilization IS NOT NULL
  AND attributes.state != "idle"
| STATS AVG(system.cpu.utilization)
  BY BUCKET(@timestamp, 1 minute), resource.attributes.host.name

To adapt this query to use variable controls, each hardcoded part has to be replaced with a variable. The aggregation function, the time bucket, and the breakdown dimension are straightforward replacements. The hostname filter requires one extra step because we want the control to allow selecting multiple hosts at once, and filtering by a single value only matches one host at a time. MV_CONTAINS checks whether a value exists inside a multi-value list, so MV_CONTAINS(?hostname, resource.attributes.host.name) returns true if the field contains any of the selected values in the control.

After replacing each part, the query becomes:

TS metrics-hostmetricsreceiver.otel-default
| WHERE system.cpu.utilization IS NOT NULL
  AND attributes.state != "idle"
  AND MV_CONTAINS(?hostname, resource.attributes.host.name)
| STATS ??aggregation(system.cpu.utilization)
  BY BUCKET(@timestamp, ?interval), ??breakdown

The same pattern applies to all four panels (CPU, Memory, Disk, Load). Changing any control updates every panel at once.

The controls

• Hostname (?hostname): Filters to the hosts selected in the control. Configured as "Values from a query" with multi-select enabled. It runs an ES|QL query that returns available host names, and MV_CONTAINS in the chart queries enables selecting more than one.

  

• Aggregation (??aggregation): Swaps the aggregation function. Static values control with AVG, MAX, MIN, MEDIAN.

• Time interval (?interval): Controls the time bucket size. Static values control with 1 minute, 5 minutes, 15 minutes, 1 hour.

• Breakdown (??breakdown): Swaps the dimension used to slice the data. Static values control with resource.attributes.host.name, resource.attributes.cloud.region, resource.attributes.cloud.availability_zone.

The investigation

The dashboard opens with AVG aggregation, region breakdown, 15-minute buckets, and all hosts selected. Nothing looks wrong. The first change is switching the aggregation from AVG to MAX and the time interval to 1 minute. A bump immediately appears in us-east-1 around March 7, roughly 68% where normal peak sits around 57%. The average was hiding this because one host's intermittent spikes get averaged across five hosts in the region.

Next, switching the breakdown from region to host makes it clear. db-01 stands out with spikes to 65-70% while its normal baseline sits around 24%. Every other host follows its expected pattern.

Setting the hostname control to db-01 only isolates the incident. Intermittent CPU bursts, not sustained saturation. Memory climbs from 85% to 93%, Load from 2.4 to 3.0, Disk from 67% to 73%. All four panels corroborate a 4-hour event window.

Why structure your queries with variable controls

A dashboard built with variable controls supports investigation paths that did not exist when the dashboard was built. Without them, every dashboard is a frozen perspective chosen at build time. When an incident does not match that perspective, someone has to edit queries or build a new dashboard under pressure. With controls, the panels adapt.

Value controls like ?hostname and ?interval handle what you filter and define the granularity of the data. Structure controls like ??aggregation and ??breakdown handle how you aggregate and how you slice. Panels sharing one query pattern means a fix or improvement applies everywhere, and a new investigation path is a single value added to a control. Together they turn a static dashboard into an investigation surface.

The day-to-day operations of understanding the status and health of your Kubernetes clusters and applications running on them, through the logs, metrics, and traces they generate, will likely be your biggest challenge. But as an operations engineer you will need all of that important data to help prevent, predict, and remediate issues. And you certainly don’t need that volume of metrics, logs and traces spread across multiple tools when you need to visualize and analyze Kubernetes telemetry data for troubleshooting and support.

Elastic Observability helps manage the sprawl of Kubernetes metrics and logs by providing extensive and centralized observability capabilities beyond just the logging that we are known for. Elastic Observability provides you with granular insights and context into the behavior of your Kubernetes clusters along with the applications running on them by unifying all of your metrics, log, and trace data through OpenTelemetry and APM agents.

Regardless of the cluster location (EKS, GKE, AKS, self-managed) or application, Kubernetes monitoring is made simple with Elastic Observability. All of the node, pod, container, application, and infrastructure (AWS, GCP, Azure) metrics, infrastructure and application logs, along with application traces are available in Elastic Observability.

In this blog we will show:

• How Elastic Cloud can aggregate and ingest metrics and log data through the Elastic Agent (easily deployed on your cluster as a DaemonSet) to retrieve logs and metrics from the host (system metrics, container stats) along with logs from all services running on top of Kubernetes.
• How Elastic Observability can bring a unified telemetry experience (logs, metrics,traces) across all your Kubernetes cluster components (pods, nodes, services, namespaces, and more).

Prerequisites and config

If you plan on following this blog, here are some of the components and details we used to set up this demonstration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).
• While we used GKE, you can use any location for your Kubernetes cluster.
• We used a variant of the ever so popular HipsterShop demo application. It was originally written by Google to showcase Kubernetes across a multitude of variants available such as the OpenTelemetry Demo App. To use the app, please go here and follow the instructions to deploy. You don’t need to deploy otelcollector for Kubernetes metrics to flow — we will cover this below.
• Elastic supports native ingest from Prometheus and FluentD, but in this blog, we are showing a direct ingest from Kubernetes cluster via Elastic Agent. There will be a follow-up blog showing how Elastic can also pull in telemetry from Prometheus or FluentD/bit.

What can you observe and analyze with Elastic?

Before we walk through the steps on getting Elastic set up to ingest and visualize Kubernetes cluster metrics and logs, let’s take a sneak peek at Elastic’s helpful dashboards.

As we noted, we ran a variant of HipsterShop on GKE and deployed Elastic Agents with Kubernetes integration as a DaemonSet on the GKE cluster. Upon deployment of the agents, Elastic starts ingesting metrics from the Kubernetes cluster (specifically from kube-state-metrics) and additionally Elastic will pull all log information from the cluster.

Visualizing Kubernetes metrics on Elastic Observability

Here are a few Kubernetes dashboards that will be available out of the box (OOTB) on Elastic Observability.

In addition to the cluster overview dashboard and pod dashboard, Elastic has several useful OOTB dashboards:

• Kubernetes overview dashboard (see above)
• Kubernetes pod dashboard (see above)
• Kubernetes nodes dashboard
• Kubernetes deployments dashboard
• Kubernetes DaemonSets dashboard
• Kubernetes StatefulSets dashboards
• Kubernetes CronJob & Jobs dashboards
• Kubernetes services dashboards
• More being added regularly

Additionally, you can either customize these dashboards or build out your own.

Working with logs on Elastic Observability

As you can see from the screens above, not only can I get Kubernetes cluster metrics, but also all the Kubernetes logs simply by using the Elastic Agent in my Kubernetes cluster.

Prevent, predict, and remediate issues

In addition to helping manage metrics and logs, Elastic can help you detect and predict anomalies across your cluster telemetry. Simply turn on Machine Learning in Elastic against your data and watch it help you enhance your analysis work. As you can see below, Elastic is not only a unified observability location for your Kubernetes cluster logs and metrics, but it also provides extensive true machine learning capabilities to enhance your analysis and management.

In the top graph, you see anomaly detection across logs and it shows something potentially wrong in the September 21 to 23 time period. Dig into the details on the bottom chart by analyzing a single kubernetes.pod.cpu.usage.node metric showing cpu issues early in September and again, later on in the month. You can do more complicated analyses on your cluster telemetry with Machine Learning using multi-metric analysis (versus the single metric issue I am showing above) along with population analysis.

Elastic gives you better machine learning capabilities to enhance your analysis of Kubernetes cluster telemetry. In the next section, let’s walk through how easy it is to get your telemetry data into Elastic.

Setting it all up

Let’s walk through the details of how to get metrics, logs, and traces into Elastic from a HipsterShop application deployed on GKE.

First, pick your favorite version of Hipstershop — as we noted above, we used a variant of the OpenTelemetry-Demo because it already has OTel. We slimmed it down for this blog, however (fewer services with some varied languages).

Step 0: Get an account on Elastic Cloud

Follow the instructions to get started on Elastic Cloud.

Step 1: Get a Kubernetes cluster and load your Kubernetes app into your cluster

Get your app on a Kubernetes cluster in your Cloud service of choice or local Kubernetes platform. Once your app is up on Kubernetes, you should have the following pods (or some variant) running on the default namespace.

NAME                                    READY   STATUS    RESTARTS   AGE
adservice-8694798b7b-jbfxt              1/1     Running   0          4d3h
cartservice-67b598697c-hfsxv            1/1     Running   0          4d3h
checkoutservice-994ddc4c4-p9p2s         1/1     Running   0          4d3h
currencyservice-574f65d7f8-zc4bn        1/1     Running   0          4d3h
emailservice-6db78645b5-ppmdk           1/1     Running   0          4d3h
frontend-5778bfc56d-jjfxg               1/1     Running   0          4d3h
jaeger-686c775fbd-7d45d                 1/1     Running   0          4d3h
loadgenerator-c8f76d8db-gvrp7           1/1     Running   0          4d3h
otelcollector-5b87f4f484-4wbwn          1/1     Running   0          4d3h
paymentservice-6888bb469c-nblqj         1/1     Running   0          4d3h
productcatalogservice-66478c4b4-ff5qm   1/1     Running   0          4d3h
recommendationservice-648978746-8bzxc   1/1     Running   0          4d3h
redis-cart-96d48485f-gpgxd              1/1     Running   0          4d3h
shippingservice-67fddb767f-cq97d        1/1     Running   0          4d3h

Step 2: Turn on <a href="https://github.com/kubernetes/kube-state-metrics" target="_self">kube-state-metrics</a>

Next you will need to turn on kube-state-metrics.

First:

git clone https://github.com/kubernetes/kube-state-metrics.git

Next, in the kube-state-metrics directory under the examples directory, just apply the standard config.

kubectl apply -f ./standard

This will turn on kube-state-metrics, and you should see a pod similar to this running in kube-system namespace.

kube-state-metrics-5f9dc77c66-qjprz                    1/1     Running   0          4d4h

Step 3: Install the Elastic Agent with Kubernetes integration

Add Kubernetes Integration:

1. 
2. In Elastic, go to integrations and select the Kubernetes Integration, and select to Add Kubernetes.
3. Select a name for the Kubernetes integration.
4. Turn on kube-state-metrics in the configuration screen.
5. Give the configuration a name in the new-agent-policy-name text box.
6. Save the configuration. The integration with a policy is now created.

You can read up on the agent policies and how they are used on the Elastic Agent here.

1. Add Kubernetes integration.
2. Select the policy you just created in the second.
3. In the third step of Add Agent instructions, copy and paste or download the manifest.
4. Add manifest to the shell where you have kubectl running, save it as elastic-agent-managed-kubernetes.yaml, and run the following command.

kubectl apply -f elastic-agent-managed-kubernetes.yaml

You should see a number of agents come up as part of a DaemonSet in kube-system namespace.

NAME                                                   READY   STATUS    RESTARTS   AGE
elastic-agent-qr6hj                                    1/1     Running   0          4d7h
elastic-agent-sctmz                                    1/1     Running   0          4d7h
elastic-agent-x6zkw                                    1/1     Running   0          4d7h
elastic-agent-zc64h                                    1/1     Running   0          4d7h

In my cluster, I have four nodes and four elastic-agents started as part of the DaemonSet.

Step 4: Look at Elastic out of the box dashboards (OOTB) for Kubernetes metrics and start discovering Kubernetes logs

That is it. You should see metrics flowing into all the dashboards. To view logs for specific pods, simply go into Discover in Kibana and search for a specific pod name.

Additionally, you can browse all the pod logs directly in Elastic.

In the above example, I searched for frontendService and cartService logs.

Step 5: Bonus!

Because we were using an OTel based application, Elastic can even pull in the application traces. But that is a discussion for another blog.

Here is a quick peek at what Hipster Shop’s traces for a front end transaction look like in Elastic Observability.

Conclusion: Elastic Observability rocks for Kubernetes monitoring

I hope you’ve gotten an appreciation for how Elastic Observability can help you manage Kubernetes clusters along with the complexity of the metrics, log, and trace data it generates for even a simple deployment.

A quick recap of lessons and more specifically learned:

• How Elastic Cloud can aggregate and ingest telemetry data through the Elastic Agent, which is easily deployed on your cluster as a DaemonSet and retrieves metrics from the host, such as system metrics, container stats, and metrics from all services running on top of Kubernetes
• Show what Elastic brings from a unified telemetry experience (Kubernenetes logs, metrics, traces) across all your Kubernetes cluster components (pods, nodes, services, any namespace, and more).
• Interest in exploring Elastic’s ML capabilities which will reduce your MTTHH (mean time to happy hour)

Ready to get started? Register and try out the features and capabilities I’ve outlined above.

Kubernetes observability with Elastic is built for the operator who gets paged at 3 AM. That operator is often in a terminal, a chat tool, or an IDE. They need an answer that is grounded in what is happening in the cluster right now.

The new Elastic Kubernetes integration is built for that operator. It includes dashboards with drilldowns, alert rule templates, and ML anomaly detection jobs. Additionally Elastic also offers Agentic Investigations, that drives investigations automatically.

This blog will cover the foundational observability components (dashboards, drilldowns, alert templates, etc), while a part 2 covering the agentic investigations will cover workflows, agent skills, and MCP tools and views

The new Kubernetes integration content in this post is generally available across Elastic Cloud Hosted, Serverless, and self-managed deployments.

Dashboards designed for drill-down, not just display

The new Kubernetes dashboards are organized around a three-tier design: a cluster Overview that surfaces what needs attention at a glance, object summary pages for clusters, nodes, namespaces, workloads, and pods, and object detail pages that give you the full picture for any single entity.

Every layer connects to the next: click any entity in a summary table and choose: apply it as a filter on the current view, or open its dedicated detail page.

Here's what that looks like when something's actually wrong:

Following a restart cascade from overview to container

Overview: The Overview surfaces what needs attention across your cluster. You can see top pods by CPU, top namespaces by container restarts, and top nodes by memory utilization in one screen. When the "container restarts" panel starts climbing, you know where to look.

Namespaces Overview: Click into the flagged namespace with 1232 restarts and CPU limit utilization at 116%. The detail view plots CPU and memory against requests and limits over time. This shows both the size and duration of the overage.

Namespace Details: We can get more info on the various pods in this namespace here. Click the pod driving the restarts.

Pod Details: The pod detail dashboard is organized into capacity, metrics, and containers sections. Container restarts are flagged in red at the top of the page. Most panels are metric-driven, and the dashboard also links to correlated pod logs in Discover.

It takes four clicks to move from the Cluster Overview to container logs that explain the failure. These dashboards are starting points for your team. You can copy and customize them with ESQL visualizations.

Alert rules that fire on day one

The integration ships with pre-built alerting rule templates for states that are wrong by definition. No historical baseline or warmup period is required. Enable them during setup and they work immediately.

These rules do not ask, "Is this abnormal for this service?" They ask, "Is this a known bad state in Kubernetes?" A pod in CrashLoopBackOff is always a problem. A container killed by the kernel for exceeding its memory limit is always a problem.

Like the Kubernetes dashboards, these alerts are built on ES|QL queries. You can see that in the CrashLoopBackOff definition below. If you are new to ES|QL, you can start with the ES|QL docs.

The alert templates cover:

• CrashLoopBackOff detection - Fires when a pod's restart count exceeds a configurable threshold within a rolling window. The default catches a real restart cycle without triggering on routine restarts during a rolling deployment.
• Container OOMKilled - Surfaces kernel-level container terminations due to memory limits. These events are easy to miss in dashboards and often precede wider failures. This rule fires on any occurrence.
• Deployment below desired replicas - Fires when a deployment runs fewer replicas than declared for longer than a grace period. This catches scaling failures and partially failed rollouts.
• Pod stuck in Pending - Fires when a pod cannot be scheduled past a configurable time threshold. This surfaces node capacity problems, missing resources, and affinity failures before availability drops.
• Node disk pressure - Fires immediately when the Kubernetes DiskPressure node condition is True. A node condition is a direct state signal, not a statistical threshold.
• Persistent volume near capacity - Alerts when storage utilization crosses a configurable threshold before writes start failing.

Each template is parameterized. Adjust thresholds in the ES|QL query to match your environment. Connect notifications to PagerDuty, Slack, or another destination in your runbook.

Anomaly detection jobs with ML baselines

Alert rules catch what is definitively wrong. ML anomaly detection catches patterns that often precede failures. If you are new to this area, see the Elastic anomaly detection overview.

A pod that always runs at 85% memory utilization might be healthy. A pod that grew from 40% to 85% over twelve hours is usually not healthy. A static threshold often catches this only after an OOM kill. The ML module should catch the trajectory earlier.

The integration ships with ML module configurations that learn workload baselines and flag meaningful deviations. These jobs need 24 to 48 hours of data before results become useful. Results become more reliable as jobs continue to run.

The included modules

1. Pod memory growth anomalies

• What it learns: per-pod memory consumption pattern over time
• What it flags: Growth trajectories that are inconsistent with baseline behavior, such as a slow leak that never crosses the hard limit.
• Why ML (not alert rule): The alert rule catches the OOMKill after the fact. The ML job catches the trajectory that leads there.

2. Network I/O anomalies

• What it learns: per-pod network transmit/receive byte rate patterns
• What it flags: Unusual spikes or drops relative to the pod baseline. A spike can indicate a runaway process or unexpected load. A drop can indicate a network partition that causes the pod to go idle.
• Why ML (not alert rule): Normal network traffic varies by time of day and workload type. A batch job pod at high throughput during its normal window is expected. The same throughput outside that window can be anomalous.

3. Pod restart frequency

• What it learns: Per-workload restart rate patterns during deployments, scaling events, and routine operations.
• What it flags: Restart patterns that are anomalous relative to each workload's own history. This is distinct from the CrashLoopBackOff alert rule, which fires on a fixed threshold regardless of context.
• Why ML (not alert rule): A deployment that restarts twice during every rollout can be healthy. The same deployment restarting twice on a Tuesday afternoon may be unhealthy. The alert rule cannot distinguish these cases without workload history.

Here's our Single Metric Viewer showing anomalies triggered against a specific pod, for the memory growth job:

And here's the multi-series Anomaly Explorer view of the same job, showing detections firing across a variety of pods:

Try it yourself: the OTel Astronomy Shop

If you do not have a Kubernetes cluster ready, you can use the OpenTelemetry Astronomy Shop demo environment. It uses the same commands as Getting Started Step 2, Path A, but points to demo services. Create the namespace and secret, then run the Helm install. All 16 services, Kafka, and PostgreSQL start flowing into Elastic without instrumentation changes.

The demo ships with a built-in feature flag service, flagd, that lets you activate failure scenarios. Enable cartServiceFailure and watch the checkout-service restart cascade unfold in real time. The CrashLoopBackOff alert rule fires. The ML modules begin establishing baselines. If you have the investigation workflow enabled, it runs automatically when the alert fires.

Getting started

Step 1 - Install the Kubernetes integration. Dashboards are available immediately. No additional configuration is required.

Step 2 - Deploy data collection. There are two supported paths, both based on Helm. Choose the one that fits your deployment model.

Path A - OpenTelemetry (EDOT collector): This path uses the opentelemetry-kube-stack Helm chart with the Elastic Distribution of OpenTelemetry (EDOT) collector. Create a namespace and a secret with your endpoint and API key, then install:

kubectl create namespace opentelemetry-operator-system

kubectl create secret generic elastic-secret-otel \
  --namespace opentelemetry-operator-system \
  --from-literal=elastic_otlp_endpoint='https://<your-endpoint>.elastic.cloud:443' \
  --from-literal=elastic_api_key='<your-api-key>'

helm upgrade --install opentelemetry-kube-stack open-telemetry/opentelemetry-kube-stack \
  --namespace opentelemetry-operator-system \
  --values 'https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v9.3.2/deploy/helm/edot-collector/kube-stack/managed_otlp/values.yaml' \
  --version '0.12.4'

Path B - Elastic Agent (standalone): This path uses the elastic/elastic-agent Helm chart. The default manifest includes resource limits that may not be appropriate for production. Review the Scaling Elastic Agent on Kubernetes guide before deploying.

helm repo add elastic https://helm.elastic.co/ && \
helm install elastic-agent elastic/elastic-agent \
  --version 9.3.2 \
  -n kube-system \
  --set outputs.default.url=https://<your-endpoint>.es.elastic.cloud:443 \
  --set outputs.default.type=ESPlainAuthAPI \
  --set outputs.default.api_key=$(echo "<your-base64-api-key>" | base64 -d) \
  --set kubernetes.enabled=true

Step 3 - Enable the alert rule templates. Go to Observability > Alerts in Kibana. The Kubernetes templates are in the rule library. Enable the templates relevant to your environment, set thresholds, and connect your notification channel.

Step 4 - Let the ML modules warm up. After 24 to 48 hours, anomaly detection modules establish baselines and begin surfacing pattern-based deviations. Longer running jobs usually produce better baselines. Find results in the ML Anomaly Explorer, linked from the Kubernetes dashboards.

Steps 5, 6, and 7 - Agentic content will be covered in Part 2 (forthcoming), Kubernetes observability with Elastic: Agentic Investigations.

What's next

The next step is the layer that runs investigation workflows when an alert fires. That includes skills that encode investigation logic, tools that expose facts like ML state and topology, and MCP apps that render outputs in places like Claude Desktop or VS Code. These technical preview capabilities are available today and will be covered in Part 2 (forthcoming), Kubernetes observability with Elastic: Agentic Investigations.

If you are running Kubernetes on Elastic today, tell us which investigation steps you repeat manually on every incident. Tell us which remediations you would trust a workflow to propose. You can join the Elastic Community Discussion here.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

As a core pillar of Observability, metrics offer a highly structured, quantitative view of system performance and health. They provide a crucial symptomatic perspective—revealing what is happening, such as high application latency, increasing service errors, or spiking container CPU utilization, which is essential for initiating alerting and triaging efforts. This capability for effective monitoring, alerting, and triaging is paramount to ensuring robust service delivery and achieving successful business outcomes.

Elastic Observability provides a comprehensive, end-to-end experience for metrics data. Elastic ensures that metrics data can be collected from numerous sources, enriched as needed and shipped to the Elastic Stack. Elastic efficiently stores this time series data, including high-cardinality metrics, utilizing the TSDS index mode (Time Series Data Stream), introduced in prior versions and used across Elastic time series integrations. This foundation ensures comprehensive observability through out-of-the-box dashboards, alerts, SLOs, and streamlined data management.

Elastic Observability 9.2 provides enhancements to metrics exploration and analysis through powerful query language extensions and expanded UI capabilities. These enhancements focus on making analysis on TSDS data via counter rates and common aggregations over time easier and faster than ever before.

The main metrics enhancements center on these key features, offered as Tech Preview:

1. Metrics analytics with TSDS and ES|QL
2. Interactive metrics exploration in Discover
3. OTLP endpoint for metrics

Metrics analytics with TSDS and ES|QL

The introduction of the new TS source command in ES|QL (Elasticsearch Query Language) on TSDS metrics dramatically simplifies time series analysis.

The TS command is specifically designed to target only time series indices, differentiating it from the general FROM command. Its core power lies in enabling a dedicated suite of time series aggregation functions within the STATS command.

This mechanism utilizes a dual aggregation paradigm, which is standard for time series querying. These queries involve two aggregation functions:

• Inner (Time Series) function: Applied implicitly per time series, often over bucketed time intervals.

• Outer (Regular) function: Used to aggregate the results of the inner function across groups. For instance, if you use STATS SUM(RATE(search_requests)) BY TBUCKET(1 hour), host, the RATE() function is the inner function applied per time series in hourly buckets, and SUM() is the outer function, summing these rates for each host and hourly bucket.

If an ES|QL query using the TS command is missing an inner (time series) aggregation function, LAST_OVER_TIME() is implicitly assumed and used. For example, TS metrics | STATS AVG(memory_usage) is equivalent to TS metrics | STATS AVG(LAST_OVER_TIME(memory_usage)).

Key time series aggregation functions available in ES|QL via TS command

These functions allow for powerful analysis on time-series data:

These functions, available with the TS command, allow SREs and Ops teams to easily perform rate calculations and other common aggregations, enabling efficient metrics analysis as a routine part of observability workflows. And it’s much faster, too! Internal performance testing has revealed that TS commands outperform other ways of querying metrics data by an order of magnitude or more, and consistently! 

Interactive metrics exploration in Discover

The 9.2 release introduces the capability to explore and analyze metrics directly and interactively within the Discover interface. In addition to exploring and analyzing logs and raw events, Discover now provides a dedicated environment for metrics exploration:

• Easy start: Begin exploration simply by querying metrics ingested via TS metrics-*.

• Grid view and pre-applied aggregations: This command displays all metrics in a grid format at a glance, immediately applying the appropriate aggregations based on the metric type, such as rate versus avg.

• Search and group-by: Quickly search for specific metrics by name. Also easily group and analyze metrics by dimensions (labels) and specific values. This allows narrowing down to metrics and dimensions of choice for targeted analysis.

• Quick access to details: Furthermore, the interface provides access to crucial details, including query and response details, the underlying ES|QL commands, the metric field type, and applicable dimensions, for each metric.

• Easy tweaking and dashboarding: The system automatically populates ES|QL queries, aiding in making easy tweaks, slicing, and dicing the data. Once analyzed, metrics and resulting analyses can be added to new or existing dashboards with ease.

OTLP endpoint for metrics

We are also introducing a native OpenTelemetry Protocol (OTLP) endpoint specifically for metrics ingest directly into Elasticsearch. The endpoint especially benefits self-managed customers, and will be integrated into our Elastic Cloud Managed OTLP Endpoint for Elastic-managed offerings. The native endpoint and related updates improve ingest performance and scalability of OTel metrics, providing up to 60% higher throughput via _otlp, and up to 25% higher throughput when using classic _bulk methods. 

In Conclusion

By merging the power of ES|QL's new time series aggregations with the familiar interactive experience of Discover, Elastic 9.2 enables a potent set of metrics analytics tools. The tools significantly boost the exploration and analysis phase of any observability workflow. And we’re just getting started on unleashing the full power of metrics in Elastic Observability!

We welcome you to try the new features today!

Also learn more about how we provide metrics analytics for AWS, Azure, GCP, Kubernetes, and LLMs on Observability Labs

Migrations covered by the Observability Migration Platform

The current scope covers Datadog and Grafana. The platform can work from exported assets or live APIs, and it focuses on dashboards and alerting content on the Datadog and Grafana paths it currently covers.

Support is not identical across the two sources. Datadog has end-to-end extraction, validation, compile, upload, smoke, and verification workflows, but it currently covers a narrower slice of widgets and monitors. Grafana coverage is broader. The platform provides a practical translation pipeline for the supported paths.

The screenshots below show examples of dashboards after migration.

How the Observability Migration Platform works

At a high level, the workflow has two halves: source-aware translation on the way in and target-aware validation and delivery on the way out. That split matters because Grafana and Datadog differ not only in JSON shape, but also in query languages, panel types, controls, and alerting models.

A run starts with exported assets or live source APIs. From there, the workflow normalizes source-specific objects, chooses a translation path for each supported dashboard, panel, and alerting artifact, and emits Kibana-native output. This is where most of the source-specific logic lives: translating queries or Datadog formulas, mapping panel semantics, carrying forward controls and links where possible, and deciding when an exact translation is not the right answer.

The second half is target-aware. The emitted output can be validated against an Elastic target, compiled, and uploaded to Kibana through the shared runtime. In the happy path, that yields a working translated dashboard. In rougher cases, validation may show that a panel cannot run safely as emitted. When that happens, the workflow is designed to fail conservatively: it can mark the panel for manual review or replace it with an upload-safe placeholder instead of shipping a broken runtime panel.

Just as important, the outcome is not simply "a dashboard showed up in Kibana." The workflow also produces reviewer-facing evidence such as a migration report, manifest, verification packets, and rollout plan so you can see what translated cleanly, what was downgraded or manualized, and what still needs human judgment. Those artifacts are what make the process operationally credible: they give teams something concrete to inspect, compare, and act on.

Running the migration

The platform is CLI-driven, and a good fit for migration work that needs to be repeatable, reviewable, and easy to automate. Users can start with a representative slice of dashboards and alerting content from Grafana or Datadog, point the workflow at an Elastic target, and use that first run to understand translation quality, validation results, and how much follow-up review is required.

To run the full path against Elastic, create an Elastic Observability Serverless project, generate a Serverless project API key, and point the CLI at your Elasticsearch and Kibana endpoints:

obs-migrate migrate \
  --source grafana \
  --input-mode files \
  --input-dir ./grafana_exports \
  --output-dir ./migration_output \
  --assets all \
  --native-promql \
  --data-view "metrics-*" \
  --validate \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --es-api-key "$KEY" \
  --kibana-url "$KIBANA_ENDPOINT" \
  --kibana-api-key "$KEY" \
  --upload

The run validates the emitted queries against Elastic, compiles the generated dashboards, uploads them to Kibana, and produces the standard migration artifacts for review.

A typical run looks like this:

1. Start with exported assets or live source APIs from Grafana or Datadog.
2. Choose the asset scope with --assets dashboards, --assets alerts, or --assets all.
3. Translate the supported dashboards, queries, controls, and alerting artifacts into Kibana-native output.
4. Validate the emitted content against an Elastic target (if configured), then compile and upload the translated dashboards for dashboard-capable runs.
5. Review the migration evidence, including migration_report.json, verification_packets.json, run_summary.json, etc., to understand what translated cleanly, where semantic gaps remain, and which dashboards, panels, or alert rules still require human review.
6. If alert rule creation is enabled, review the migrated rules (which are disabled by default) in Kibana before deciding which ones to enable or redesign.

What's next

The platform is still evolving, and will continue to gain depth and self-service capabilities. The biggest open areas are stronger measured source-to-target semantic verification, further coverage for Datadog, deeper coverage for harder query families and non-dashboard surfaces, and cleaner shared runtime contracts across the workflow.

It is also built to grow over time. The source and target boundaries are explicit by design, which gives the platform room to expand coverage and support additional source paths in the future.

In conclusion

If you are planning a move into Elastic, a good starting point is to create an Elastic Observability Serverless project. That gives you the target environment where translated dashboards and alerting content can be validated and reviewed.

To learn more about the migration workflow, talk to your Elastic representative about current access, supported coverage, and how it can help with your migration needs.

In five years working with Enterprise accounts at Elastic, I have heard the same challenge again and again:

"We have several network monitoring tools, and we would love to correlate all of them into one platform."

For many organizations, the barrier to true correlation isn't a lack of data, but where that data lives. Frequently, we see SNMP metrics, flow data, and logs isolated in purpose-built silos or dashboards. Without a unified data store and a proper correlation engine, piecing together the full narrative — from a topology change to a performance degradation — becomes a manual, time-consuming puzzle.

When an incident happens, engineers become human correlation engines — manually jumping between systems, copying timestamps, cross-referencing device names, and trying to piece together what actually happened. A simple question like "Did this interface failure impact application performance?" requires querying multiple tools and mentally correlating the results.

The real cost isn't the tool licenses — it's the time lost during critical incidents.

This lab is my answer to a fundamental question: Can Elastic become the unified foundation that actually correlates network data?

More importantly, it demonstrates that Elastic is fully ready for network operations — capable of ingesting diverse telemetry and using AI to correlate relationships, identify root causes, and resolve issues in seconds instead of hours.

The Problem: Network Observability is Broken

Let me paint a typical scenario I encounter with enterprise network teams:

The Fragmented Reality:

• No single source of truth
• Manual correlation during incidents (15-30 minutes per event)
• Fragmented teams (network vs. platform engineers)
• Limited automation capabilities
• No AI-powered analysis

When a link goes down at 2 AM:

• Notice the alert - 2 minutes
• Log into monitoring tool to see the metric - 3 minutes
• Switch to traffic analyzer to check impact - 5 minutes
• Open log management to search for related messages - 10 minutes
• Manually correlate timestamps across systems - 8 minutes
• Create a ticket and copy context from multiple tools - 8 minutes

Time to initial diagnosis: 36 minutes

This workflow is expensive, error-prone, and doesn't scale.

The Vision: Elastic as a Unified Network Observability Platform

What if you could:

• Collect SNMP metrics, NetFlow, traps, and topology data in one platform
• Correlate network events with application performance automatically
• Generate executive dashboards without separate BI tools
• Use AI to analyze incidents in seconds, not hours
• Trigger alerting from network events

This is what this lab aims to demonstrate.

What I Built: A Production-Grade Network Simulation

To demonstrate how Elastic unifies network data, I needed a realistic environment that generates real-world telemetry. Enter Containerlab — a Docker-based solution that enables us to create a network simulation framework.

Lab Architecture

I simulated a Service Provider core network with:

• 7 FRR routers forming an OSPF Area 0 mesh
• 2 Ubuntu hosts for additional use cases
• 2 Layer 2 switches for access layer segmentation
• 3 telemetry collectors feeding Elastic Cloud

Total containers: 14

Deployment time: 12-15 minutes (fully automated)

Full deployment instructions and topology details are available in the GitHub repository README.

The Three Telemetry Pipelines: Proving Multi-Source Correlation

What makes this lab production-ready is its hybrid observability approach — proving that Elastic can unify disparate network data sources.

This unified architecture proves Elastic can replace multiple specialized network monitoring tools with a single platform.

The Power of Correlation: One Platform, One Query

When a network incident occurs, you need to answer questions like:

• Which interface failed? (SNMP metrics)
• What traffic was affected? (NetFlow)
• What was the sequence of events? (SNMP traps)
• Which devices are downstream? (LLDP topology)

The Problem: modern tools offer separate modules glued together, forcing users to navigate different spaces for different sets of data.

The Reality: You still have to pivot. You see a spike in the Metrics module, but to see why, you have to open the Logs module and manually align the time picker. The data lives in different tables or backends, making true correlation impossible without human intervention.

The Elastic Difference: One Store, One Language, One AI

Elastic makes it simple. Whether it's an SNMP counter (metric), a NetFlow record (flow), or a Syslog message (log), it is all stored in a unified datastore powered by the Elasticsearch engine. This allows users to easily search across multiple datasets in a single query.

FROM logs-*
| WHERE host.name == "csr23" AND interface.name == "eth1"

Time required: 3 seconds

Furthermore, as you will see later, the exact location of the data becomes agnostic to the user when leveraging the AI Assistant.

Data Transformation: From Cryptic OIDs to Actionable Intelligence

Raw SNMP traps are notoriously difficult to interpret at a glance. In our current lab setup, the data arrives looking like this:

OID: 1.3.6.1.6.3.1.1.5.3
ifIndex: 2
ifDescr: eth1

While traditional Network Management Platforms (NMPs) handle OID translation natively, bringing that clarity into Elastic requires a specific configuration.

In this initial lab, we are intentionally working with this raw data to demonstrate how AI assistants can interpret these events even without pre-existing context.

However, the strategy for the next phase of this project is to implement Elasticsearch Ingest Pipelines. This will allow us to map raw OIDs to human-readable names. This step is crucial for bridging the gap between Network tools and Application Observability platforms, allowing network events to be instantly correlated with application errors and infrastructure logs.

The Target State

Once the pipeline is implemented in the next lab, we will transform that raw trap into searchable, meaningful data:

{
  "event.action": "interface-down",
  "host.name": "csr23",
  "interface.name": "eth1",
  "interface.oper_status_text": "Link Down"
}

The result:

• Human-readable fields
• Searchable dimensions for filtering
• Context for automation rules and dashboards
• Correlation keys for joining with metrics and flows

In our next blog post, we will walk through building the ingest pipeline that performs this transformation — step by step.

Intelligent Alerting: From Noise to Actionable Intelligence

Traditional network monitoring relies on simple threshold alerts — "interface down," "high CPU." These alerts flood your inbox but provide zero context about root cause, impact, or remediation.

The Lab's Approach: ES|QL + AI Assistant

1. Semantic Detection with ES|QL

Instead of generic threshold alerts, the lab uses ES|QL to detect specific event patterns:

FROM logs-snmp.trap-prod
| WHERE snmp.trap_oid == "1.3.6.1.6.3.1.1.5.3"
| KEEP @timestamp, host.name, interface.name, message

2. Automatic AI-Powered Investigation

When the alert triggers, it invokes the Observability AI Assistant with a structured investigation prompt that:

• Performs immediate triage (which device, which interface, when)
• Assesses OSPF impact and traffic rerouting
• Correlates with other recent failures
• Generates severity assessment and recommended actions

The Transformation

Accelerating Incident Response with the Elastic AI Assistant

This is where the Elastic AI Assistant demonstrates its operational value — moving beyond passive data collection to actively interpret and explain network events in real-time

When an engineer views a trap document in Discover and asks:

"Explain this log message"

The AI Assistant provides comprehensive analysis including:

• What happened: Plain-language explanation of the SNMP trap
• Device context: Router role, interface purpose, network position
• Impact analysis: OSPF neighbor status, traffic rerouting assessment
• Root cause possibilities: Physical layer, link layer, administrative causes
• Recommended actions: Immediate steps, investigation queries, validation checks
• Severity assessment: Business and technical impact rating

Manual Triage vs. AI-Assisted Investigation

The Value Proposition: One Platform, One Data Model, One AI

What This Lab Demonstrates

Elastic provides:

• One unified platform for metrics, logs, flows
• One data model (SemConv) for consistent correlation
• One search interface (Kibana) for all network data
• One AI assistant that understands all your network telemetry
• AI-powered alerting with automated investigation

Business Impact

Efficiency Gains:

• 85% reduction in MTTR (36 min → 5 min for initial diagnosis)
• 90% reduction in manual correlation time
• Junior engineers gain access to AI-powered expert analysis

Operational Benefits:

• Network engineers focus on strategy, not tool-switching
• Cross-functional collaboration in one platform
• Reduced tool sprawl and management overhead

Lessons Learned

After building this lab, several key insights emerged regarding how network data fits into the broader observability ecosystem:

1. Extending Observability to the Network

Elastic is already the gold standard for high-volume logs and application traces. This lab demonstrates that the same engine seamlessly handles network telemetry without needing a separate, siloed tool.

• Scale: The same architecture that ingests petabytes of application logs easily handles millions of interface counters.
• Structure: Native support for complex nested documents allows for rich SNMP trap data (variable bindings) without flattening or losing context.
• Speed: Real-time search applies equally to network events, enabling sub-second troubleshooting.

2. OpenTelemetry Semantic Conventions (SemConv) as the Universal Translator

The power isn't just in storing the data, but in standardizing it. By mapping SNMP and NetFlow to the OpenTelemetry Semantic Conventions (SemConv), network data finally speaks the same language as the rest of the stack.

• Unified Search: Query across firewall logs, server metrics, and switch telemetry in a single search bar.
• Instant Visualization: Pre-built dashboards work immediately because the field names are standardized.
• Cross-Domain Correlation: Easily correlates a spike in application latency with a specific interface saturation event.

3. AI Assistants Thrive on Context

While the AI in this lab was powerful on its own, the experiment highlighted a critical realization: an AI Assistant becomes exponentially more effective when coupled with a specific Knowledge Base.

Context is King: The AI delivers better root cause analysis when provided with rich metadata, such as device roles and topology maps. Without it, the advice remains generic.

Pro Tip (and What’s Next):

To get organization-specific advice rather than generic suggestions, you need to feed the AI your documentation.

• The Goal: Create a Knowledge Base containing device roles, network topology diagrams, and troubleshooting procedures.
• The Next Step: In my next blog post, I will demonstrate exactly how to do this — connecting a Knowledge Base to the AI Assistant to enable fully context-aware troubleshooting.

Conclusion: Completing the Observability Picture

Elastic is already widely recognized as the standard for Application and Security observability. The goal of this lab wasn't to ask if Elastic can handle networking, but to demonstrate the immense value of bringing network data into that existing ecosystem.

The verdict is clear: Elastic acts as that unified foundation. It effectively breaks down the silo between Network Engineering and the rest of IT.

This isn't just about consolidating dashboards or replacing legacy tools. It is about establishing the Elasticsearch AI Platform as the single source of truth where network telemetry sits right alongside application and infrastructure data.

By treating network data as a first-class citizen in the observability stack, we unlock automated correlation, AI-assisted investigation, and the speed required to resolve incidents before they impact the business. The capabilities are in place, and the foundation is solid — Elastic is ready to unify your network with the rest of your digital business.

Ready to Try It Yourself?

Check out github.com/DeBaker1974/Containerlab-OSPF

The repository includes:

• Complete deployment scripts (12-15 minute automated setup)
• Pre-configured telemetry pipelines
• Kibana dashboards
• Alert rules with AI Assistant integration
• Detailed README

Not ready to build? Try Elastic Serverless: Start a free 14-day trial and explore AI-powered observability with your own data.

Special thanks to the Containerlab and FRRouting communities for their incredible open-source tools, and to Sheriff Lawal (CCIE, CISSP), Sr. Manager, Solutions Architecture at Elastic, for mentoring on this project.

In this blog, we dive into how to enable and use time series data streams by reviewing what a time series metrics document is and the mapping used for enabling time series. In particular, we will showcase this by using Elastic Observability’s Nginx integration. As Elastic^® time series data stream (TSDS) metrics capabilities evolve, some of the scenarios below will change.

Elastic TSDS stores metrics in indices optimized for a time series database (TSDB), which is used to store time series metrics. Elastic’s TSDB also got a significant optimization in 8.7 by reducing storage costs by upward of 70%.

What is an Elastic time series data stream?

A time series data stream (TSDS) models timestamped metrics data as one or more time series. In a TSDS, each Elasticsearch document represents an observation or data point in a specific time series. Although a TSDS can contain multiple time series, a document can only belong to one time series. A time series can’t span multiple data streams.

A regular data stream can have different usages including logs. For metrics usage, however, a time series data stream is recommended. A time series data stream is different from a regular data stream in multiple ways. A TSDS contains more than one predefined dimension and multiple metrics.

Nginx metrics as an example

Integrations provide an easy way to ingest observability metrics for a large number of services and systems. We use the Nginx integration metrics data set as an example here. This is one of the integrations, on which time series has been recently enabled.

Process of enabling TSDS on a package

Time series is enabled on a metrics data stream of an integration package, after adding the relevant time series metrics and dimension mappings. Existing integrations with metrics data streams will come with time series metrics enabled, so that users can use them as-is without any additional configuration.

The image below captures a high-level summary of a time series data stream, the corresponding index template, the time series indices and a single document. We will shortly dive into the details of each of the fields in the document.

TSDS metric document

Below we provide a snippet of an ingested Elastic document with time series metrics and dimension together.

{
  "@timestamp": "2023-06-29T03:58:12.772Z",

  "nginx": {
    "stubstatus": {
      "accepts": 202,
      "active": 2,
      "current": 3,
      "dropped": 0,
      "handled": 202,
      "hostname": "host.docker.internal:80",
      "reading": 0,
      "requests": 10217,
      "waiting": 1,
      "writing": 1
    }
  }
}

Multiple metrics per document:
An ingested document has a collection of fields, including metrics fields. Multiple related metrics fields can be part of a single document. A document is part of a single data stream, and typically all the metrics it contains are related. All the metrics in a document are part of the same time series.

Metric type and dimensions as mapping:
While the document contains the metrics details, the metric types and dimension details are defined as part of the field mapping. All the time series relevant field mappings are defined collectively for a given datastream, as part of the package development. All the integrations released with time series data stream, contain all the relevant time series field mappings, as part of the package release. There are two additional mappings needed in particular: time_series_metric mapping and time_series_dimension mapping.

Metrics types fields

A document contains the metric type fields (as shown above). The mappings for the metric type fields is done using time_series_metric mapping in the index templates as given below:

"nginx": {
    "properties": {
       "stubstatus": {
           "properties": {
                "accepts": {
                  "type": "long",
                  "time_series_metric": "counter"
                },
                "active": {
                  "type": "long",
                  "time_series_metric": "gauge"
                },
                "current": {
                  "type": "long",
                  "time_series_metric": "gauge"
                },
                "dropped": {
                  "type": "long",
                  "time_series_metric": "counter"
                },
                "handled": {
                  "type": "long",
                  "time_series_metric": "counter"
                },
                "reading": {
                  "type": "long",
                  "time_series_metric": "gauge"
                },
                "requests": {
                  "type": "long",
                  "time_series_metric": "counter"
                },
                "waiting": {
                  "type": "long",
                  "time_series_metric": "gauge"
                },
                "writing": {
                  "type": "long",
                  "time_series_metric": "gauge"
                }
           }
       }
    }
}

Dimension fields

Dimensions are field names and values that, in combination, identify a document’s time series.

In Elastic time series, there are some additional considerations for dimensions:

• Dimension fields need to be defined for each time series. There will be no time series with zero dimension fields.
• Keyword (or similar) type fields can be defined as dimensions.
• There is a current limit on the number of dimensions that can be defined in a data stream. The limit restrictions will likely be lifted going forward.

Dimension is common for all the metrics in a single document, as part of a data stream. Each time series data stream of a package (example: Nginx) already comes with a predefined set of dimension fields as below.

The document would contain more than one dimension field. In the case of Nginx, agend.id and nginx.stubstatus.hostname are some of the dimension fields. The mappings for the dimension fields is done using time_series_dimension mapping as below:

"agent": {
   "properties": {
      "id": {
         "type": "keyword",
         "time_series_dimension": true
       }
    }
 },

"nginx": {
   "properties": {
       "stubstatus": {
            "properties": {
                "hostname": {
                  "type": "keyword",
                  "time_series_dimension": true
                },
            }
       }
    }
}

Meta fields

Documents ingested also have additional meta fields apart from the metric and dimension fields explained above. These additional fields provide richer query capabilities for the metrics.

Example Elastic meta fields

"data_stream": {
      "dataset": "nginx.stubstatus",
      "namespace": "default",
      "type": "metrics"
 }

Discover and visualization in Kibana

Elastic provides comprehensive search and visualization for the time series metrics. Time series metrics can be searched as-is in Discover. In the search below, the counter and gauges metrics are captured as different icons. Below we also provide examples of visualization for the time series metrics using Lens and OOTB dashboard included as part of the Nginx integration package.

Try it out!

We have provided a detailed example of a time series document ingested by the Elastic Nginx integration. We have walked through how time series metrics are modeled in Elastic and the additional time series mappings with examples. We provided details of dimension requirements for Elastic time series, as well as brief examples of search/visualization/dashboard of TSDS metrics in Kibana^®.

Don’t have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out the auto-instrumentation capabilities that I discussed above. I would be interested in getting your feedback about your experience in gaining visibility into your application stack with Elastic.

• How to use Elasticsearch and Time Series Data Streams for observability metrics
• Time Series Data Stream in Elastic documentation 
• Efficient storage with Elastic Time Series DatabaseElastic integrations catalog
• Elastic integrations catalog

First, we need an example application that allows us to monitor infrastructure changes effectively. We use a Python Flask application with the Elastic Python APM agent. The application is a simple calculator taking the numbers as a REST request. We utilize Locust — a simple load-testing tool to evaluate performance under varying workloads.

The next step includes obtaining the pricing information associated with the cloud services. Every cloud provider is different. Most of them offer an option to retrieve pricing through an API. But today, we will focus on Google Cloud and will leverage their pricing calculator to retrieve relevant cost information.

The calculator and Google Cloud pricing

To perform a cost analysis, we need to know the cost of the machines in use. Google provides a billing API and Client Library to fetch the necessary data programmatically. In this blog, we are not covering the API approach. Instead, the Google Cloud Pricing Calculator is enough. Select the machine type and region in the calculator and set the count 1 instance. It will then report the total estimated cost for this machine. Doing this for an e2-standard-4 machine type results in 107.7071784 US$ for a runtime of 730 hours.

Now, let’s go to our Kibana® where we will create a new index inside Dev Tools. Since we don’t want to analyze text, we will tell Elasticsearch® to treat every text as a keyword. The index name is cloud-billing. I might want to do the same for Azure and AWS, then I can append it to the same index.

PUT cloud-billing
{
  "mappings": {
    "dynamic_templates": [
      {
        "stringsaskeywords": {
          "match": "*",
          "match_mapping_type": "string",
          "mapping": {
            "type": "keyword"
          }
        }
      }
    ]
  }
}

Next up is crafting our billing document:

POST cloud-billing/_doc/e2-standard-4_europe-west4
{
  "machine": {
    "enrichment": "e2-standard-4_europe-west4"
  },
  "cloud": {
    "machine": {
       "type": "e2-standard-4"
    },
    "region": "europe-west4",
    "provider": "google"
  },
  "stats": {
    "cpu": 4,
    "memory": 8
  },
  "price": {
    "minute": 0.002459068,
    "hour": 0.14754408,
    "month": 107.7071784
  }
}

We create a document and set a custom ID. This ID matches the instance name and the region since the machines' costs may differ in each region. Automatic IDs could be problematic because I might want to update what a machine costs regularly. I could use a timestamped index for that and only ever use the latest document matching. But this way, I can update and don’t have to worry about it. I calculated the price down to minute and hour prices as well. The most important thing is the machine.enrichment field, which is the same as the ID. The same instance type can exist in multiple regions, but our enrichment processor is limited to match or range. We create a matching name that can explicitly match as in e2-standard-4_europe-west4. It’s up to you to decide whether you want the cloud provider in there and make it google_e2-standard-4_europ-west-4.

Calculating the cost

There are multiple ways of achieving this in the Elastic Stack. In this case, we will use an enrich policy, ingest pipeline, and transform.

The enrich policy is rather easy to setup:

PUT _enrich/policy/cloud-billing
{
  "match": {
    "indices": "cloud-billing",
    "match_field": "machine.enrichment",
    "enrich_fields": ["price.minute", "price.hour", "price.month"]
  }
}

POST _enrich/policy/cloud-billing/_execute

Don’t forget to run the _execute at the end of it. This is necessary to make the internal indices used by the enrichment in the ingest pipeline. The ingest pipeline is rather minimalistic — it calls the enrichment and renames a field. This is where our machine.enrichment field comes in. One caveat around enrichment is that when you add new documents to the cloud-billing index, you need to rerun the _execute statement. The last bit calculates the total cost with the count of unique machines seen.

PUT _ingest/pipeline/cloud-billing
{
  "processors": [
    {
      "set": {
        "field": "_temp.machine_type",
        "value": "{{cloud.machine.type}}_{{cloud.region}}"
      }
    },
    {
      "enrich": {
        "policy_name": "cloud-billing",
        "field": "_temp.machine_type",
        "target_field": "enrichment"
      }
    },
    {
      "rename": {
        "field": "enrichment.price",
        "target_field": "price"
      }
    },
    {
      "remove": {
        "field": [
          "_temp",
          "enrichment"
        ]
      }
    },
    {
      "script": {
        "source": "ctx.total_price=ctx.count_machines*ctx.price.hour"
      }
    }
  ]
}

Since this is all configured now, we are ready for our Transform. For this, we need a data view that matches the APM data_streams. This is traces-apm*, metrics-apm.*, logs-apm.*. For the Transform, go to the Transform UI in Kibana and configure it in the following way:

We are doing an hourly breakdown, therefore, I get a document per service, per hour, per machine type. The interesting bit is the aggregations. I want to see the average CPU usage and the 75,95,99 percentile, to view the CPU usage on an hourly basis. Allowing me to identify the CPU usage across an hour. At the bottom, give the transform a name and select an index cloud-costs and select the cloud-billing ingest pipeline.

Here is the entire transform as a JSON document:

PUT _transform/cloud-billing
{
  "source": {
    "index": [
      "traces-apm*",
      "metrics-apm.*",
      "logs-apm.*"
    ],
    "query": {
      "bool": {
        "filter": [
          {
            "bool": {
              "should": [
                {
                  "exists": {
                    "field": "cloud.provider"
                  }
                }
              ],
              "minimum_should_match": 1
            }
          }
        ]
      }
    }
  },
  "pivot": {
    "group_by": {
      "@timestamp": {
        "date_histogram": {
          "field": "@timestamp",
          "calendar_interval": "1h"
        }
      },
      "cloud.provider": {
        "terms": {
          "field": "cloud.provider"
        }
      },
      "cloud.region": {
        "terms": {
          "field": "cloud.region"
        }
      },
      "cloud.machine.type": {
        "terms": {
          "field": "cloud.machine.type"
        }
      },
      "service.name": {
        "terms": {
          "field": "service.name"
        }
      }
    },
    "aggregations": {
      "avg_cpu": {
        "avg": {
          "field": "system.cpu.total.norm.pct"
        }
      },
      "percentiles_cpu": {
        "percentiles": {
          "field": "system.cpu.total.norm.pct",
          "percents": [
            75,
            95,
            99
          ]
        }
      },
      "avg_transaction_duration": {
        "avg": {
          "field": "transaction.duration.us"
        }
      },
      "percentiles_transaction_duration": {
        "percentiles": {
          "field": "transaction.duration.us",
          "percents": [
            75,
            95,
            99
          ]
        }
      },
      "count_machines": {
        "cardinality": {
          "field": "cloud.instance.id"
        }
      }
    }
  },
  "dest": {
    "index": "cloud-costs",
    "pipeline": "cloud-costs"
  },
  "sync": {
    "time": {
      "delay": "120s",
      "field": "@timestamp"
    }
  },
  "settings": {
    "max_page_search_size": 1000
  }
}

Once the transform is created and running, we need a Kibana Data View for the index: cloud-costs. For the transaction, use the custom formatter inside Kibana and set its format to “Duration” in “microseconds.”

With that, everything is arranged and ready to go.

Observing infrastructure changes

Below I created a dashboard that allows us to identify:

• How much costs a certain service creates
• CPU usage
• Memory usage
• Transaction duration
• Identify cost-saving potential

From left to right, we want to focus on the very first chart. We have the bars representing the CPU as average in green and 95th percentile in blue on top. It goes from 0 to 100% and is normalized, meaning that even with 8 CPU cores, it will still read 100% usage and not 800%. The line graph represents the transaction duration, the average being in red, and the 95th percentile in purple. Last, we have the orange area at the bottom, which is the average memory usage on that host.

We immediately realize that our calculator does not need a lot of memory. Hovering over the graph reveals 2.89% memory usage. The e2-standard-8 machine that we are using has 32 GB of memory. We occasionally spike to 100% CPU in the 95th percentile. When this happens, we see that the average transaction duration spikes to 2.5 milliseconds. However, every hour this machine costs us a rounded 30 cents. Using this information, we can now downsize to a better fit. The average CPU usage is around 11-13%, and the 95th percentile is not that far away.

Because we are using 8 CPUs, one could now say that 12.5% represents a full core, but that is just an assumption on a piece of paper. Nonetheless, we know there is a lot of headroom, and we can downscale quite a bit. In this case, I decided to go to 2 CPUs and 2 GB of RAM, known as e2-highcpu2. This should fit my calculator application better. We barely touched the RAM, 2.89% out of 32GB are roughly 1GB of use. After the change and reboot of the calculator machine, I started the same Locust test to identify my CPU usage and, more importantly, if my transactions get slower, and if so, by how much. Ultimately, I want to decide whether 1 millisecond more latency is worth 10 more cents per hour. I added the change as an annotation in Lens.

After letting it run for a bit, we can now identify the smaller hosts' impact. In this case, we can see that the average did not change. However, the 95th percentile — as in 95% of all transactions are below this value — did spike up. Again, it looks bad at first, but checking in, it went from ~1.5 milliseconds to ~2.10 milliseconds, a ~0.6 millisecond increase. Now, you can decide whether that 0.6 millisecond increase is worth paying ~180$ more per month or if the current latency is good enough.

Conclusion

Observability is more than just collecting logs, metrics, and traces. Linking user experience to cloud costs allows your business to identify areas where you can save money. Having the right tools at your disposal will help you generate those insights quickly. Making informed decisions about how to optimize your cloud cost and ultimately improve the user experience is the bottom-line goal.

The dashboard and data view can be found in my GitHub repository. You can download the .ndjson file and import it using the Saved Objects inside Stack Management in Kibana.

Caveats

Pricing is only for base machines without any disk information, static public IP addresses, and any other additional cost, such as licenses for operating systems. Furthermore, it excludes spot pricing, discounts, or free credits. Additionally, data transfer costs between services are also not included. We only calculate it based on the minute rate of the service running — we are not checking billing intervals from Google Cloud. In our case, we would bill per minute, regardless of what Google Cloud has. Using the count for unique instance.ids work as intended. However, if a machine is only running for one minute, we calculate it based on the hourly rate. So, a machine running for one minute, will cost the same as running for 50 minutes — at least how we calculate it. The transform uses calendar hour intervals; therefore, it's 8 am-9 am, 9 am-10 am, and so on.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

This post looks at what happens inside Elasticsearch when a Remote Write request arrives.

If you want to understand the implementation, evaluate how Elasticsearch compares to other Prometheus-compatible backends, or contribute, this is the post for you. A companion post, Ship Prometheus Metrics to Elasticsearch with Remote Write, covers the setup and configuration side.

Request lifecycle: from HTTP to indexed documents

A quick note on the Prometheus data model before we dive in: Prometheus stores all metric values as 64-bit floats and treats the metric name as just another label (__name__). The storage engine itself is agnostic of whether a value is a counter or a gauge. Keep this in mind as we walk through how Elasticsearch maps these concepts.

Here is the full path of a Remote Write request through Elasticsearch:

1. HTTP layer — The endpoint receives a compressed protobuf payload, checks indexing pressure, decompresses with Snappy, and parses the protobuf WriteRequest.
2. Document construction — Each sample in each time series becomes an Elasticsearch document with @timestamp, labels.*, and metrics.* fields.
3. Bulk indexing — All documents from a single request are written to the target data stream via a single bulk call.

The sections below walk through each stage in detail.

HTTP layer

The endpoint accepts application/x-protobuf POST requests. The incoming request body is tracked against the same indexing pressure limits that protect the bulk indexing API. If the cluster is already under heavy indexing load, the request gets rejected with a 429 before any parsing happens.

Prometheus compresses Remote Write payloads with Snappy. Elasticsearch decompresses the body in a streaming fashion without materializing it into a single contiguous allocation, and validates the declared uncompressed size against a configurable maximum to guard against decompression bombs.

The decompressed body is then deserialized as a protobuf WriteRequest. Each WriteRequest contains a list of TimeSeries entries, and each TimeSeries contains a set of labels (key-value pairs) and a list of samples (timestamp + float64 value).

Document construction

For each sample in each time series, Elasticsearch builds an index request. Here is what a single document looks like:

{
  "@timestamp": "2026-04-01T12:00:00.000Z",
  "data_stream": {
    "type": "metrics",
    "dataset": "generic.prometheus",
    "namespace": "default"
  },
  "labels": {
    "__name__": "http_requests_total",
    "job": "prometheus",
    "instance": "localhost:9090",
    "method": "GET",
    "status": "200"
  },
  "metrics": {
    "http_requests_total": 1027.0
  }
}

All labels from the Prometheus time series (including __name__) end up in the labels.* fields. The metric value goes into metrics.<metric_name>, where <metric_name> is the value of the __name__ label.

Time series without a __name__ label are dropped entirely, and the samples are counted as failures. Non-finite values (NaN, Infinity, negative Infinity) are silently skipped. This includes Prometheus staleness markers, which use a special NaN bit pattern (0x7ff0000000000002) to signal that a series has disappeared.

One sample, one document

You might wonder whether storing each individual sample as its own document creates significant storage overhead, especially for labels. A common pattern to reduce that overhead was to group all metrics sharing the same labels and timestamp into a single document.

With recent TSDB improvements, that optimization is no longer necessary. Elasticsearch has trimmed the per-document storage overhead to the point where there is negligible difference between packing many metrics in a single document and writing each sample separately. A dedicated post covering these TSDB storage improvements in detail is coming soon.

Bulk indexing

All documents from a single Remote Write request are sent to Elasticsearch via a single bulk request. Each document targets the data stream metrics-{dataset}.prometheus-{namespace} and is indexed as an append-only create operation.

Metric type inference

Remote Write v1 does not reliably transmit metric types alongside samples. Prometheus sends metadata (type, help text, unit) in separate requests roughly once per minute, and those requests may land on a different node than the samples. Buffering samples until metadata arrives is not practical in a distributed system, so Elasticsearch infers the type from naming conventions instead.

Metric names ending in _total, _sum, _count, or _bucket are mapped as counters. Everything else defaults to gauge. This is a well-established convention that other Prometheus-compatible backends use as well.

http_requests_total             → counter
request_duration_seconds_sum    → counter
request_duration_seconds_count  → counter
request_duration_seconds_bucket → counter
process_resident_memory_bytes   → gauge
go_goroutines                   → gauge

The heuristic can be wrong. A metric like temperature_total (if someone named a gauge that way) would be misclassified as a counter. The main consequence today is that some ES|QL functions like rate() require the metric type to be a counter and will reject a misclassified gauge. For PromQL, we plan to lift this restriction so that rate() works regardless of the declared type, which will make incorrect inference less consequential.

You can override the inference by creating a metrics-prometheus@custom component template with custom dynamic templates. For example, to treat all *_counter fields as counters:

PUT /_component_template/metrics-prometheus@custom
{
  "template": {
    "mappings": {
      "dynamic_templates": [
        {
          "counter": {
            "path_match": "metrics.*_counter",
            "mapping": {
              "type": "double",
              "time_series_metric": "counter"
            }
          }
        }
      ]
    }
  }
}

Custom dynamic templates are merged with the built-in ones, so the default naming-convention rules still apply for metrics you don't explicitly override.

The index template

Elasticsearch installs a built-in index template that matches metrics-*.prometheus-*. This template is what makes field type inference work without manual mapping configuration.

TSDS mode is enabled, which gives you time-based partitioning, optimized storage, deduplication, and the ability to downsample data as it ages.

Passthrough object fields are used for both the labels and metrics namespaces. This serves three purposes:

1. Namespace isolation: Labels and metrics live in separate object namespaces (labels.* and metrics.*), so a label named status and a metric named status cannot conflict with each other.

2. Dimension identification: The labels passthrough object is configured with time_series_dimension: true, which means every field under labels.* is automatically treated as a TSDS dimension. When Prometheus sends a time series with a label you have never seen before, it becomes a dimension without any explicit field mapping.

3. Transparent queries: You don't need to write the labels. or metrics. prefix in ES|QL or PromQL. A query can reference job instead of labels.job, or http_requests_total instead of metrics.http_requests_total. The passthrough mapping handles the resolution.

Dynamic inference for metrics applies the naming-convention heuristics described above. When a new metric name appears for the first time, its field mapping is created automatically under metrics.* with the correct time_series_metric annotation.

Failure store is enabled. Documents that fail indexing (for example, due to a mapping conflict where the same metric name appears with incompatible types) are routed to a separate failure store instead of being dropped silently.

Data stream routing

The three URL patterns map directly to data stream names:

This lets you separate metrics from different Prometheus instances or environments into different data streams. That separation is useful for a few reasons.

Lifecycle isolation: you can apply different retention policies per data stream. Production metrics might be kept for 90 days, while dev metrics might expire after 7 days.

Access control: you can scope API keys to specific data streams. A team's Prometheus instance writes to metrics-teamA.prometheus-prod, and their API key only has access to that stream.

Query performance: PromQL queries and Grafana dashboards can be scoped to a specific index pattern, avoiding scans of unrelated data.

Error handling and the Remote Write spec

The Remote Write spec defines two response classes: retryable (5xx, 429) and non-retryable (4xx). Prometheus uses this distinction to decide whether to retry or drop a failed request.

Elasticsearch returns 429 (Too Many Requests) if any sample in the bulk request was rejected due to indexing pressure. This signals Prometheus to back off and retry with exponential backoff.

For partial failures (some samples indexed, others rejected), the response includes a summary. It reports how many samples failed, grouped by target index and status code, along with a sample error message from each group.

Time series without a __name__ label result in a 400 error for those samples. Non-finite values (NaN, Infinity) are silently dropped: Prometheus receives a success response and will not retry.

NaN appears most commonly for summary quantiles when no observations have been recorded (for example, a p99 latency metric before any requests arrive) and for staleness markers. The practical impact of dropping these is limited today: for most queries, a missing sample behaves similarly to a NaN one, since PromQL's lookback window fills the gap with the last known value either way. The more significant gap is staleness markers, which are covered below.

What's next: Remote Write v2 and beyond

Remote Write v2 is still experimental, which is why the current implementation starts with v1. But v2 addresses several of v1's shortcomings.

Metadata alongside samples: v2 sends metric type, unit, and description with each time series in the same request. This eliminates the need for naming-convention heuristics entirely.

Native histograms: v2 supports Prometheus native histograms, which map naturally to Elasticsearch's exponential_histogram field type. Classic histograms (one counter per bucket boundary) are verbose and lose precision at query time. Native histograms are more compact and more accurate.

Dictionary encoding: v2 replaces repeated label strings with integer references, reducing payload size significantly for high-cardinality label sets.

Created timestamps: counters in v2 include a "created" timestamp that marks when the counter was initialized. This allows backends to detect counter resets more accurately than the current heuristic (value decreased since last sample).

Beyond v2, there are two other items in consideration for future enhancements.

Staleness marker support: currently, staleness markers (the special NaN that Prometheus writes when a scrape target disappears) are dropped. Supporting them would allow correct PromQL lookback behavior and avoid the 5-minute "trailing data" artifact where a disappeared series still appears in query results.

Shared metric field: the current layout creates a separate field for each metric name (metrics.http_requests_total, metrics.go_goroutines, etc.). This works, but it means the number of field mappings grows with the number of distinct metric names, which is why the field limit is set to 10,000 for Prometheus data streams. A different approach we're considering is to store the metric name only in the __name__ label and write the metric value to a single shared field. This eliminates the field explosion problem entirely and more closely matches how Prometheus stores data internally. This direction is part of the broader effort to make Elasticsearch's metrics storage more efficient and more compatible with Prometheus conventions.

Availability

The Prometheus Remote Write endpoint is available now on Elasticsearch Serverless with no additional configuration.

For self-managed clusters, check out start-local to get up and running quickly.

If you run into issues or have feedback, open an issue on the Elasticsearch repository.

This lets you bring your Prometheus metrics into the same cluster which can also store logs, traces, and other data. One storage backend, one set of access controls, one place to query.

Why store Prometheus metrics in Elasticsearch?

Prometheus local storage is designed for short retention, typically 15 to 30 days. For anything beyond that, you need a remote storage backend.

Elasticsearch's time series data streams (TSDS) are built for highly efficient long term metrics storage: automatic rollover, time-based partitioning, compression via index sorting, and downsampling to reduce storage costs as data ages. Your Prometheus scrape configs stay the same.

Recent Elasticsearch releases have significantly reduced the storage footprint for metrics. A dedicated post with the numbers is coming soon.

On the query side, ES|QL embraces PromQL: a built-in PROMQL function lets your existing queries run unchanged, while the rest of ES|QL is available when you want joins, aggregations, or transformations that span multiple datasets.

And because metrics land in the same store as your logs, traces, and profiling data, correlating signals across types becomes a single query rather than a cross-system investigation.

How it works

For a detailed look at what happens inside Elasticsearch when a Remote Write request arrives — protobuf parsing, metric type inference, TSDS mapping, and data stream routing — see How Prometheus Remote Write Ingestion Works in Elasticsearch.

Prometheus sends metrics to Elasticsearch via the standard Remote Write protocol (v1). The endpoint accepts protobuf-encoded, snappy-compressed WriteRequest payloads.

Each sample becomes an Elasticsearch document in a pre-defined time series data stream. Prometheus labels become TSDS dimensions. The metric value is stored in a typed field under metrics.<metric_name>.

Elasticsearch infers the metric type (counter vs gauge) from naming conventions. Names ending in _total, _sum, _count, or _bucket are treated as counters. Everything else is treated as a gauge.

Setting it up

Step 1: Get an Elasticsearch endpoint

You need an Elasticsearch cluster with the Prometheus endpoints enabled. The simplest option is Elastic Cloud Serverless, where this works out of the box.

For serverless: sign in to cloud.elastic.co, create an Observability project, and copy the Elasticsearch endpoint from the project settings page. The endpoint looks like https://<project-id>.es.<region>.<provider>.elastic.cloud.

Step 2: Create an API key

Create an API key scoped to writing metrics data streams only. In your Elastic Cloud Serverless project, go to Admin and settings (the gear icon at the bottom left of the side nav), then API keys.

Use the following role descriptor in the Control security privileges section:

{
  "ingest": {
    "indices": [
      {
        "names": ["metrics-*"],
        "privileges": ["auto_configure", "create_doc"]
      }
    ]
  }
}

Copy the key value before closing the dialog. You will not be able to retrieve it again.

Step 3: Configure Prometheus

Add the following remote_write block to your prometheus.yml:

remote_write:
  - url: "https://YOUR_ES_ENDPOINT/_prometheus/api/v1/write"
    authorization:
      type: ApiKey
      credentials: YOUR_API_KEY

That's it. Prometheus will start shipping metrics to Elasticsearch on the next scrape interval.

If you use Grafana Alloy instead of Prometheus, the equivalent configuration is:

prometheus.remote_write "elasticsearch" {
  endpoint {
    url = "https://YOUR_ES_ENDPOINT/_prometheus/api/v1/write"
    headers = {"Authorization" = "ApiKey YOUR_API_KEY"}
  }
}

Routing metrics to separate data streams

By default, all metrics land in metrics-generic.prometheus-default. You can route metrics from different environments or teams into separate data streams using the dataset and namespace path segments in the URL.

The three URL patterns are:

• /_prometheus/api/v1/write routes to metrics-generic.prometheus-default
• /_prometheus/metrics/{dataset}/api/v1/write routes to metrics-{dataset}.prometheus-default
• /_prometheus/metrics/{dataset}/{namespace}/api/v1/write routes to metrics-{dataset}.prometheus-{namespace}

For example, using /_prometheus/metrics/infrastructure/production/api/v1/write routes data to metrics-infrastructure.prometheus-production.

This is useful for separating production from staging metrics, or giving different teams their own data streams with independent lifecycle policies.

What gets stored

Here is what a sample document looks like in Elasticsearch:

{
  "@timestamp": "2026-04-02T10:30:00.000Z",
  "data_stream": {
    "type": "metrics",
    "dataset": "generic.prometheus",
    "namespace": "default"
  },
  "labels": {
    "__name__": "prometheus_http_requests_total",
    "handler": "/api/v1/query",
    "code": "200",
    "instance": "localhost:9090",
    "job": "prometheus"
  },
  "metrics": {
    "prometheus_http_requests_total": 42
  }
}

Labels map to keyword fields that serve as TSDS dimensions. The metric value is stored under metrics.<metric_name> with the inferred time_series_metric type (counter or gauge).

Elasticsearch installs a built-in index template matching metrics-*.prometheus-* that configures TSDS mode, passthrough dimension container objects, and a 10,000 field limit. The field limit is configurable via a custom component template (see the custom metric type inference section below for how to use one). You do not need to create any templates or mappings yourself.

Custom metric type inference

Metric type inference is based on naming conventions. Metrics that don't follow Prometheus naming best practices may be classified incorrectly. You can override the defaults by creating a metrics-prometheus@custom component template with your own dynamic templates. For example, to mark all *_counter metrics as counters:

{
  "template": {
    "mappings": {
      "dynamic_templates": [
        {
          "counter": {
            "path_match": "metrics.*_counter",
            "mapping": {
              "type": "double",
              "time_series_metric": "counter"
            }
          }
        }
      ]
    }
  }
}

Custom rules are merged with the built-in patterns, so the defaults still apply for metrics you don't override.

Current limitations

Only Remote Write v1 is supported. v2, which brings native histograms and exemplars, is planned.

Staleness markers (special NaN values Prometheus uses to signal a series has disappeared) are not yet stored or respected in queries.

Non-finite values (NaN, Infinity) are silently dropped.

Get started

The Prometheus Remote Write endpoint is available now on Elasticsearch Serverless with no configuration needed. To get started with a local cluster, start-local gets you a single-node cluster in minutes.

Once metrics are flowing, you can query them with ES|QL using the built-in PROMQL function for PromQL compatibility, or write native ES|QL queries to join metrics with logs and traces in the same store.

Running PromQL queries in Kibana

In the ES|QL editor in Kibana, enter the PROMQL command, and type your PromQL in that block. PROMQL marks that segment so Elasticsearch parses it as PromQL inside the wider ES|QL request Kibana sends.

What you can query

Here are a few patterns to get started.

Raw metric

PROMQL container.cpu.usage

Average across all containers

PROMQL avg(container.cpu.usage)

rate() on a counter

PROMQL rate(docker.network.inbound.bytes)

Aggregated rate

PROMQL sum(rate(docker.network.inbound.bytes))

Group by a label

PROMQL sum by (agent.id) (rate(docker.network.inbound.bytes))

You may notice that none of these examples include start, end, step, or a lookback window on every rate(). Those parameters are optional: the time picker and Kibana defaults handle most of it for you.

Optionally, you can include the data stream name using the index= parameter. For example: PROMQL index=metrics-docker.cpu-default container.cpu.usage. Adding the parameter helps narrow down the scope of what data the query scans.

The current release of PromQL tech preview has over 80% query coverage benchmarked against top Grafana dashboards. Advanced modifiers and specific functions are in consideration for future releases.

Find your streams and metric names

If you have existing PromQL queries, you can use them directly in the PROMQL command without changes. If you are writing a query from scratch and need to find the exact field names, run TS metrics-* in Discover to see every metrics data stream. Each metric appears as a small chart so you can tell at a glance what is active. Hover over a metric and click the "View details" action to see the field name and the data stream it belongs to.

For a deeper walkthrough, see Explore metrics data with Discover in Kibana.

Time picker and query time handling

The time picker in Kibana sets the time window for the query. Dashboard panels and Alerting rules work the same way using their own time range, so you do not need to write start= or end= in the query itself.

Step is the gap between two consecutive data points on the chart. A smaller step means more data points across the same span. If you do not set step= or buckets=, the default is buckets=100. You can set step= to a fixed width such as 1m, or set buckets= to a different target maximum number of data points.

Discover and Dashboards

In Discover, switch to ES|QL mode and run your PROMQL query so you can see how the metric behaves over the range you pick, as a time-series chart. When you want to save that visualization, choose "Save visualization to dashboard" and add it to a new or existing dashboard.

Or go to Dashboards directly: add a panel, choose ES|QL, and write your PROMQL query.

Alerting

You can create alert rules using PromQL. Go to Alerts, open Manage rules, and create a rule. Search for Elasticsearch query and select it. Choose ES|QL as the query type.

Write your PROMQL query, but assign the metric to a variable so you can use it in a WHERE clause for the alert condition:

PROMQL metric_value=(sum by (agent.id) (rate(docker.network.inbound.bytes)))
| WHERE metric_value >= 500

Select @timestamp for the time field and continue defining the rest of the rule configuration.

Try it

1. Open an Observability project on Elastic Cloud Serverless, or use Elastic Stack 9.4.
2. Write your query: in the ES|QL editor in Kibana, run your PromQL via PROMQL. You can also go to Dashboards, add a panel, choose ES|QL, and write the query there.
3. If you are writing from scratch and need to find metric names, run TS metrics-* in Discover (see "Find your streams and metric names" above).
4. Check the results and adapt the query if needed.

PromQL support in Elasticsearch and Kibana will continue to evolve. Follow the Observability Labs feed for follow-up posts as coverage and ergonomics improve.

With the release of an upgraded vSphere integration we now support an enhanced set of metrics and datastreams. Package version 1.15.0 onwards introduces new datastreams that significantly improve the collection of performance metrics, providing deeper insights into your vSphere environment.

This enhanced version includes a total of seven datastreams, featuring critical new metrics such as disk performance, memory utilization, and network status. Additionally, these datastreams now offer detailed visibility into associated resources like hosts, clusters, and resource pools. To make the most of these insights, we’ve also introduced prebuilt dashboards, helping teams monitor and troubleshoot their vSphere environments with ease and precision.

We have expanded the performance metrics to encompass a broader range of insights across all datastreams, while also introducing new datastreams for clusters, resource pools, and networks. This enhanced integration version now includes a total of seven datastreams, featuring critical new metrics such as disk performance, memory utilization, and network status. Additionally, these datastreams now offer detailed visibility into associated resources like hosts, clusters, and resource pools.

Each datastream also includes detailed alarm information, such as the alarm name, description, status (e.g. critical or warning), and the affected entity's name. To make the most of these insights, we’ve also introduced prebuilt dashboards, helping teams monitor and troubleshoot their vSphere environments with ease and precision.

Overview of the Datastreams

• Host Datastream: This datastream monitors the disk performance of the host, including metrics such as disk latency, average read/write bytes, uptime, and status. It also captures network metrics, such as packet information, network bandwidth, and utilization, as well as CPU and memory usage of the host. Additionally, it lists associated datastores, virtual machines, and networks within vSphere.

• Virtual Machine Datastream: This datastream tracks the used and available CPU and memory resources of virtual machines, along with the uptime and status of each VM. It includes information about the host on which the VM is running, as well as detailed snapshot metrics like the number of snapshots, creation dates, and descriptions. Additionally, it provides insights into associated hosts and datastores.

• Datastore Datastream: This datastream provides information on the total, used, and available capacity of datastores, along with their overall status. It also captures metrics such as the average read/write rate and lists the hosts and virtual machines connected to each datastore.

• Datastore Cluster: A datastore cluster in vSphere is a collection of datastores grouped together for efficient storage management. This datastream provides details on the total capacity and free space in the storage pod, along with the list of datastores within the cluster.

• Resource Pool: Resource pools in vSphere serve as logical abstractions that allow flexible allocation of CPU and memory resources. This datastream captures memory metrics, including swapped, ballooned, and shared memory, as well as CPU metrics like distributed and static CPU entitlement. It also lists the virtual machines associated with each resource pool.

• Network Datastream: This datastream captures the overall configuration and status of the network, including network types (e.g., vSS, vDS). It also lists the hosts and virtual machines connected to each network.

• Cluster Datastream: A Cluster in vSphere is a collection of ESXi hosts and their associated virtual machines that function as a unified resource pool. Clustering in vSphere allows administrators to manage multiple hosts and resources centrally, providing high availability, load balancing, and scalability to the virtual environment. This datastream includes metrics indicating whether HA or admission control is enabled and lists the hosts, networks, and datastores associated with the cluster.

Alarms support in vSphere Integration

Alarms are a vital part of the vSphere integration, providing real-time insights into critical events across your virtual environment. In the updated Elastic’s vSphere integration, alarms are now reported for all the entities. They include detailed information such as the alarm name, description, severity (e.g., critical or warning), affected entity, and triggered time. These alarms are seamlessly integrated into datastreams, helping administrators and SREs quickly identify and resolve issues like resource shortages or performance bottlenecks.

Example Alarm

"triggered_alarms": [
  {
    "description": "Default alarm to monitor host memory usage",
    "entity_name": "host_us",
    "id": "alarm-4.host-12",
    "name": "Host memory usage",
    "status": "red",
    "triggered_time": "2024-08-28T10:31:26.621Z"
  }
]

This example highlights a triggered alarm for monitoring host memory usage, indicating a critical status (red) for the host "host_us." Such alarms empower teams to act swiftly and maintain the stability of their vSphere environment.

Lets Try It Out!

The new vSphere integration in Elastic Cloud is more than just a monitoring tool; it’s a comprehensive solution that empowers you to manage and optimize your virtual environments effectively. With deeper insights and enhanced data granularity, you can ensure high availability, improved load balancing, and smarter resource allocation. Spin up an Elastic Cloud, and start monitoring your vSphere infrastructure.

• Detect Kubernetes pod restarts and OOMKill events using Elastic Agent Builder
• Analyze CPU and memory pressure using ES|QL over Kubernetes metrics
• Generate troubleshooting summaries and remediation guidance

This article explains how to use Elastic Agent Builder to automatically detect, analyze, and remediate Kubernetes pod failures caused by resource pressure (CPU and memory), with a focus on pods experiencing frequent restarts and OOMKilled events. Elastic Agent Builder lets you quickly create precise agents that utilize all your data with powerful tools (such as ES|QL queries), chat interfaces, and custom agents.

Introduction: What is the Elastic Agent Builder?

Elastic has an AI Agent embedded that you can use to get more insights from all of the logs, metrics and traces that you’ve ingested. While that’s great, you can take it one step further and streamline the process by creating tools that the agent can use.

Giving the agent tools means it spends less time ‘thinking’ and quickly gets to assessing what’s important to you. For example, if I have a Kubernetes environment that needs monitoring, and I want to keep an eye on pod restarts and memory and CPU usage without hanging out at the terminal, I can have Elastic alert me if something goes wrong. 

Having an alert is great, but how do I get the bigger picture, faster? You need to know what service is having (or creating) the issues, why, and how to fix it.

Assumptions

This guide assumes:

• A running Kubernetes cluster
• An Elastic Observability deployment
• Kubernetes metrics indexed in Elastic

Step 1: Create a New Elastic Agent

In Elastic Observability, use the top search bar to search for Agents. Create a new agent.

This agent is going to be the Kubernetes Pod Troubleshooter agent, designed to help users troubleshoot pod restarts, OOMKill terminations and evaluate CPU or memory pressure. 

The Kubernetes Pod Troubleshooter agent will:

1. Identify pods that have restarted more than once
2. Filter for pods that are not in a running state
3. Retrieve the container termination reason (e.g., OOMKilled)
4. Analyze CPU and memory utilization for affected services
5. Flag resource utilization above 60% (warning) and 80% (critical)
6. Provide remediation recommendations

The agent requires instructions to guide how the agent behaves when interacting with tools or responding to queries. This description can set tone, priorities or special behaviours. The instructions below tell the agent to execute the steps outlined above. 

You will help users troubleshoot problematic pods by searching the metrics for pods that have restarted more than once and the status is not running. Pods that have the highest number of restarts will be returned to the user.
Once the containers that are not running and have restarted multiple times are found you will use their container ID or image name to to look up the container status reason and reason for the last termination. You will return that reason to the user.
You will also begin basic troubleshooting steps, such as checking  for insufficient cluster resources (CPU or memory) from the metrics and tools available.
Any CPU or memory utilization percentages over 60%, and definitely over 80% should be flagged to the user with remediation steps.

Getting answers quickly is critical when troubleshooting high-value systems and environments. Using Tools ensures that the workflow is repeatable and that you can trust the results. You also get complete oversight of the process, as the Elastic Agent outlines every step and query that it took and you can explore the results in Discover.

You will create custom tools that the agent will run to complete the Kubernetes troubleshooting tasks that the custom instructions references such as: look up the container status reason and reason for the last termination and checking  for insufficient cluster resources (CPU or memory).

Step 2: Create Tools - Pod Restarts

The first tool takes the Kubernetes metrics and assesses if the pod has restarted and it has a last terminated reason, and if it has the agent will present that information to the user.

This pod-restarts tool uses a custom ES|QL query that interrogates the Kubernetes metrics data coming from OTel.

The ES|QL query:

1. Filters for containers that have restarted and have a reason for termination; then
2. Calculates the number of restarts; then
3. Returns the number of restarts and termination reason per service.

FROM metrics-k8sclusterreceiver.otel-default
| WHERE metrics.k8s.container.restarts > 0
| WHERE resource.attributes.k8s.container.status.last_terminated_reason IS NOT NULL
| STATS total_restarts = SUM(metrics.k8s.container.restarts),
        reasons = VALUES(resource.attributes.k8s.container.status.last_terminated_reason) 
  BY resource.attributes.service.name
| SORT total_restarts DESC

Step 3: Create Tools - Service Memory

The custom tools can take input variables, which increases speed and accuracy of the results.

Common reasons for pods not scheduling, or restarting often, is due to the cluster or nodes being under-resourced. The pod-restarts tool returns services that have many restarts and OOMKill termination reasons, which indicate memory pressure.

The eval-pod-memory tool is a custom ES|QL that:

1. Filters for metrics data that match the service name returned from the pod-restarts tool within the last 12 hours; then
2. Converts memory usage, requests, limits and utilization into megabytes; then
3. Calculates the average of each of those metrics; then
4. Groups them into 1 minute groupings and sorts them.

FROM metrics-*
| WHERE resource.attributes.service.name == ?servicename
| WHERE @timestamp >= NOW() - 12 hours
| EVAL
  memory_usage_mb = metrics.container.memory.usage / 1024 / 1024,
   memory_request_mb = metrics.k8s.container.memory_request / 1024 / 1024,
   memory_limit_mb = metrics.k8s.container.memory_limit / 1024 / 1024,
   memory_utilization_pct = metrics.k8s.container.memory_limit_utilization * 100
| STATS
   avg_memory_usage = AVG(memory_usage_mb),
   avg_memory_request = AVG(memory_request_mb),
   avg_memory_limit = AVG(memory_limit_mb),
   avg_memory_utilization = AVG(memory_utilization_pct)
   BY bucket = BUCKET(@timestamp, 1 minute)
| SORT bucket ASC

Step 4: Create Tools: Service CPU

As CPU usage is another common reason for pods to fail scheduling or be stuck in endless restart loops, the next tool will evaluate CPU usage, requests and limits.

The eval-pod-cpu tool is a custom ES|QL that:

1. Filters for metrics data that match the service name returned from the pod-restarts tool within the last 12 hours; then
2. Calculates the average for CPU usage, CPU request utilization and CPU limit utilization.

FROM metrics-kubeletstatsreceiver.otel-default
| WHERE k8s.container.name == ?servicename OR resource.attributes.k8s.container.name == ?servicename
| STATS
  avg_cpu_usage = AVG(container.cpu.usage),
  avg_cpu_request_utilization = AVG(k8s.container.cpu_request_utilization) * 100,
  avg_cpu_limit_utilization = AVG(k8s.container.cpu_limit_utilization) * 100
| LIMIT 100

Step 5: Assign Tools to Kubernetes Pod Troubleshooter Agent

Once all of the tools are built you need to assign them to the agent.

This image shows the Kubernetes Pod Troubleshooter agent with the three tools: pod-restarts, eval-pod-cpu and eval-pod-memory assigned to it and active.

Step 6: Test the Kubernetes Pod Troubleshooter Agent

To simulate memory pressure the Open Telemetry demo is running inside the cluster. Artificially lowering the memory requests and limits and increasing the service load will cause pods to restart.

To do this to the open telemetry demo in your cluster, follow these steps. 

Reduce the cart service to one replica by scaling the deployment. Once that is complete, change the resources on the deployment by lowering the memory requests and limits as shown in this command:

kubectl -n otel-demo scale deploy/cart --replicas=1
kubectl -n otel-demo set resources deploy/cart -c cart --requests=memory=50Mi --limits=memory=60Mi

The OpenTelemetry demo application comes with a load-generator. This is used to simulate requests to the demo site by modifying the users and spawn rate in the load generator deployment, as shown in this command:

kubectl -n otel-demo set env deploy/load-generator LOCUST_USERS=800 LOCUST_SPAWN_RATE=200 LOCUST_BROWSER_TRAFFIC_ENABLED=false

If you list all of your pods in the cluster or namespace, you should begin to see restarts.

You can now chat with the Kubernetes Pod Troubleshooter agent and ask “Are any of my Kubernetes pods having issues?”.

The screenshot shows the final response from the Kubernetes Pod Troubleshooter agent. It provides a problem summary of its findings from each tool, showing which services were experiencing the most restarts and memory and CPU utilization. 

The threshold interpretations were described in the initial agent instructions, where >60% utilization is a warning (sustained pressure) and >80% utilization is critical (high likelihood of restarts or throttling). This aligns with findings presented by the Kubernetes Pod Troubleshooter agent, where the services that had the highest restarts were all above 90% memory utilization. The agent needs clearly defined threshold values to correctly assess the returned memory and CPU utilization values. 

Problem summary returned by the Kubernetes Pod Troubleshooter agent:

Conclusion and Final Thoughts

Elastic Agent Builder enables fast, repeatable Kubernetes troubleshooting by combining ES|QL-driven analysis with constrained AI reasoning.

The creation of custom tools that use specific ES|QL queries combined with downstream queries that take input variables from the output of previous tools eliminates or reduces error propagation and hallucinations. In comparison to generic AI troubleshooting without purpose-built tools, you run the risk of it analyzing too many services (that aren’t relevant to the issue at hand). This will slow down the thinking process and generate longer responses, increasing the likelihood of error propagation and hallucinations. 

With the Elastic Agent Builder, you can inspect the output of every tool if you need to, to explore and verify the outputs.

Having a succinct problem summary is a game-changer, bringing your attention straight to the most affected services.

Reasoning returned by the Kubernetes Pod Troubleshooter agent:

Not only that, but the agent can go one step further and offer recommendations for remediation based on what outputs the tools delivered.

Remediation recommendation returned by the Kubernetes Pod Troubleshooter agent:

Sign up for Elastic Cloud Serverless and try this out with your Kubernetes clusters.

Frequently Asked Questions

1. When to use the Elastic Agent Builder for Troubleshooting

Use the Elastic Agent Builder for Troubleshooting that works best if:

• You need repeatable, auditable troubleshooting workflows

• You want deterministic analysis instead of free-form AI responses

• You’re investigating something that is reported in the logs or metrics (i.e. pod restarts, OOMKills, or resource pressure)

• You want to reduce mean time to resolution (MTTR)

2. Do I need OpenTelemetry to use Elastic Agent Builder for Kubernetes troubleshooting? 

No, you don’t need to use OpenTelemetry. You have two options:

• You can collect logs and metrics from Kubernetes using the Elastic Agent; or 

• You can collect logs, traces and metrics with the Elastic Distro for OTel (EDOT) Collector

When following the steps above, this would change the field names that are used in the tools above. For example, kubernetes.container.memory.usage.bytes vs metrics.container.memory.usage.

3. Can this agent be adapted for node-level failures? 

Yes, Elastic has hundreds of integrations, including AWS (for EKS), Azure (for AKS), Google Cloud (for GKE), as well as host operating system monitoring.

The queries shown above would be modified to use the correct field.

4. Can these tools be reused in automation workflows? 

Yes, Elastic Workflows can reuse the same scripted automations and AI agents you build in Elastic. An agent can handle the initial analysis and investigation (reducing manual effort), and the workflow can continue with structured steps, such as running Elasticsearch queries, transforming data, branching on conditions and calling external APIs or tools like Slack, Jira and PagerDuty. Workflows can also be exposed to Agent Builder as reusable tools, just like the tool created in this guide.

For more advanced automation from a similar scenario as described in this guide, learn how to integrate AI agents into GitHub Actions to monitor K8s health and improve deployment reliability via Observability.

5. Can these tools be triggered by alerts? 

Yes, alerts can trigger Elastic Workflows, and pass the alert context to the workflow. This workflow may be integrated with an Elastic Agent, as described above.

Additionally, Elastic Alerts allow you to publish investigation guides alongside alerts so an SRE has all of the information they need to begin investigating. Any troubleshooting or investigative agents can be linked to from the investigation guide, meaning the SRE doesn’t have to follow manual processes outlined in an investigation guide and instead let the agent handle the manual, repetitive investigations.

6. How can I get started with Agent Builder?

Sign up for Elastic Cloud Serverless, a new fully managed, stateless architecture that auto-scales no matter your data, usage, and performance needs.

Logging is an important part of observability, for which we generally think of metrics and/or tracing. However, the amount of logs an application or the underlying infrastructure output can be significantly daunting.

With Elastic Observability, there are three main mechanisms to ingest logs:

• The new Elastic Agent pulls metrics and logs from CloudWatch and S3 where logs are generally pushed from a service (for example, EC2, ELB, WAF, Route53, etc ). We reviewed Elastic agent metrics configuration for EC2, RDS (Aurora), ELB, and NAT metrics in this blog.
• Using Elastic’s Serverless Forwarder (runs on Lambda and available in AWS SAR) to send logs from Firehose, S3, CloudWatch, and other AWS services into Elastic.
• Beta feature (contact your Elastic account team): Using AWS Firehose to directly insert logs from AWS into Elastic — specifically if you are running the Elastic stack on AWS infrastructure.

In this blog we will provide an overview of the second option, Elastic’s serverless forwarder collecting VPC Flow Logs from an application deployed on EC2 instances. Here’s what we'll cover:

• A walk-through on how to analyze VPC Flow Log info with Elastic’s Discover, dashboard, and ML analysis.
• A detailed step-by-step overview and setup of the Elastic serverless forwarder on AWS as a pipeline for VPC Flow Logs into Elastic Cloud.

Elastic’s serverless forwarder on AWS Lambda

AWS users can quickly ingest logs stored in Amazon S3, CloudWatch, or Kinesis with the Elastic serverless forwarder, an AWS Lambda application, and view them in the Elastic Stack alongside other logs and metrics for centralized analytics. Once the AWS serverless forwarder is configured and deployed from AWS, Serverless Application Registry (SAR) logs will be ingested and available in Elastic for analysis. See the following links for further configuration guidance:

• Elastic’s serverless forwarder (runs Lambda and available in AWS SAR)
• Serverless forwarder GitHub repo

In our configuration we will ingest VPC Flow Logs into Elastic for the three-tier app deployed in the previous blog.

There are three different configurations with the Elastic serverless forwarder:

Logs can be directly ingested from:

• Amazon CloudWatch: Elastic serverless forwarder can pull VPC Flow Logs directly from an Amazon CloudWatch log group, which is a commonly used endpoint to store VPC Flow Logs in AWS.
• Amazon Kinesis: Elastic serverless forwarder can pull VPC Flow Logs directly from Kinesis, which is another location to publish VPC Flow Logs.
• Amazon S3: Elastic serverless forwarder can pull VPC Flow Logs from Amazon S3 via SQS event notifications, which is a common endpoint to publish VPC Flow Logs in AWS.

We will review how to utilize a common configuration, which is to send VPC Flow Logs to Amazon S3 and into Elastic Cloud in the second half of this blog.

But first let's review how to analyze VPC Flow Logs on Elastic.

Analyzing VPC Flow Logs in Elastic

Now that you have VPC Flow Logs in Elastic Cloud, how can you analyze them?

There are several analyses you can perform on the VPC Flow Log data:

1. Use Elastic’s Analytics Discover capabilities to manually analyze the data.
2. Use Elastic Observability’s anomaly feature to identify anomalies in the logs.
3. Use an out-of-the-box (OOTB) dashboard to further analyze data.

Using Elastic Discover

In Elastic analytics, you can search and filter your data, get information about the structure of the fields, and display your findings in a visualization. You can also customize and save your searches and place them on a dashboard. With Discover, you can:

• View logs in bulk, within specific time frames
• Look at individual details of each entry (document)
• Filter for specific values
• Analyze fields
• Create and save searches
• Build visualizations

For a complete understanding of Discover and all of Elastic’s analytics capabilities, look at Elastic documentation.

For VPC Flow Logs, an important stat is to understand:

• How many logs were accepted/rejected
• Where potential security violations are occur (for example, source IPs from outside the VPC)
• What port is generally being queried

I’ve filtered the logs on the following:

• Amazon S3: bshettisartest
• VPC Flow Log action: REJECT
• VPC Network Interface: Webserver 1

We want to see what IP addresses are trying to hit our web servers.

From that, we want to understand which IP addresses we are getting the most REJECTS from, and we simply find the source.ip field. Then, we can quickly get a breakdown that shows 185.242.53.156 is the most rejected for the last 3+ hours we’ve turned on VPC Flow Logs.

Additionally, I can see a visualization by selecting the “Visualize” button. We get the following, which we can add to a dashboard:

In addition to IP addresses, we want to also see what port is being hit on our web servers.
We select the destination port field, and the quick pop-up shows us a list of ports being targeted. We can see that port 23 is being targeted (this port is generally used for telnet), port 445 is being targeted (used for Microsoft Active Directory), and port 433 (used for https ssl). We also see these are all REJECT.

Anomaly detection in Elastic Observability logs

Addition to Discover, Elastic Observability provides the ability to detect anomalies on logs. In Elastic Observability -> logs -> anomalies you can turn on machine learning for:

• Log rate: automatically detects anomalous log entry rates
• Categorization: automatically categorizes log messages

For our VPC Flow Log, we turned both on. And when we look at what has been detected for anomalous log entry rates, we see:

Elastic immediately detected a spike in logs when we turned on VPC Flow Logs for our application. The rate change is being detected because we’re also ingesting VPC Flow Logs from another application for a couple of days prior to adding the application in this blog.

We can further drill down into this anomaly with machine learning and analyze further.

There is more machine learning analysis you can utilize with your logs — check out Elastic machine learning documentation.

Since we know that a spike exists, we can also use Elastic AIOps Labs Explain Log Rate Spikes capability in Machine Learning. Additionally, we’ve grouped them to see what is causing some of the spikes.

As we can see, a specific network interface is sending more VPC log flows than others. We can further drill down into this further in Discover.

VPC Flow Log dashboard on Elastic Observability

Finally, Elastic also provides an OOTB dashboard to showing the top IP addresses hitting your VPC, geographically where they are coming from, the time series of the flows, and a summary of VPC Flow Log rejects within the time frame.

This is a baseline dashboard that can be enhanced with visualizations you find in Discover, as we reviewed in option 1 (Using Elastic’s Analytics Discover capabilities) above.

Setting it all up

Let’s walk through the details of configuring Amazon Kinesis Data Firehose and Elastic Observability to ingest data.

Prerequisites and config

If you plan on following steps, here are some of the components and details we used to set up this demonstration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here) on AWS. Deploying this on AWS is required for Elastic Serverless Forwarder.
• Ensure you have an AWS account with permissions to pull the necessary data from AWS. Specifically, ensure you can configure the agent to pull data from AWS as needed. Please look at the documentation for details.
• We used AWS’s three-tier app and installed it as instructed in GitHub. (See blog on ingesting metrics from the AWS services supporting this app.)
• Configure and install Elastic’s Serverless Forwarder.
• Ensure you turn on VPC Flow Logs for the VPC where the application is deployed and send logs to AWS Firehose.

Step 0: Get an account on Elastic Cloud

Follow the instructions to get started on Elastic Cloud.

Step 1: Deploy Elastic on AWS

Once logged in to Elastic Cloud, create a deployment on AWS. It’s important to ensure that the deployment is on AWS. The Amazon Kinesis Data Firehose connects specifically to an endpoint that needs to be on AWS.

Once your deployment is created, make sure you copy the Elasticsearch endpoint.

The endpoint should be an AWS endpoint, such as:

https://aws-logs.es.us-east-1.aws.found.io

Step 2: Turn on Elastic’s AWS Integrations on AWS

In your deployment’s Elastic Integration section, go to the AWS integration and select Install AWS assets.

Step 3: Deploy your application

Follow the instructions listed out in AWS’s Three-Tier app and instructions in the workshop link on GitHub. The workshop is listed here.

Once you’ve installed the app, get credentials from AWS. This will be needed for Elastic’s AWS integration.

There are several options for credentials:

• Use access keys directly
• Use temporary security credentials
• Use a shared credentials file
• Use an IAM role Amazon Resource Name (ARN)

View more details on specifics around necessary credentials and permissions.

Step 4: Send VPC Flow Logs to Amazon S3 and set up Amazon SQS

In the VPC for the application deployed in Step 3, you will need to configure VPC Flow Logs and point them to an Amazon S3 bucket. Specifically, you will want to keep it as AWS default format.

Create the VPC Flow log.

Next:

• Set up an Amazon SQS queue
• Configure Amazon S3 event notifications

Step 5: Set up Elastic Serverless Forwarder on AWS

Follow instructions listed in Elastic’s documentation and refer to the previous blog providing an overview. The important bits during the configuration in Lambda’s application repository are to ensure you:

• Specify the S3 Bucket in ElasticServerlessForwarderS3Buckets where the VPC Flow Logs are being sent. The value is the ARN of the S3 Bucket you created in Step 4.
• Specify the configuration file path in ElasticServerlessForwarderS3ConfigFile. The value is the S3 url in the format "s3://bucket-name/config-file-name" pointing to the configuration file (sarconfig.yaml).
• Specify the S3 SQS Notifications queue used as the trigger of the Lambda function in ElasticServerlessForwarderS3SQSEvents. The value is the ARN of the SQS Queue you set up in Step 4.

Once Amazon CloudFormation finishes setting up Elastic serverless forwarder, you should see two Amazon Lambda functions:

In order to check if logs are coming in, go to the function with “ ApplicationElasticServer ” in the name, and go to monitor and look at logs. You should see the logs being pulled from S3.

Step 6: Check and ensure you have logs in Elastic

Now that steps 1–4 are complete, you can go to Elastic’s Discover capability and you should see VPC Flow Logs coming in. In the image below, we’ve filtered by Amazon S3 bucket bshettisartest.

Conclusion: Elastic Observability easily integrates with VPC Flow Logs for analytics, alerting, and insights

I hope you’ve gotten an appreciation for how Elastic Observability can help you manage AWS VPC Flow Logs. Here’s a quick recap of lessons and what you learned:

• A walk-through of how Elastic Observability provides enhanced analysis for VPC Flow Logs:
  • Using Elastic’s Analytics Discover capabilities to manually analyze the data
  • Leveraging Elastic Observability’s anomaly features to:
    • Identify anomalies in the VPC flow logs
    • Detects anomalous log entry rates
    • Automatically categorizes log messages
  • Using an OOTB dashboard to further analyze data
• A more detailed walk-through of how to set up the Elastic Serverless Forwarder

Start your own 7-day free trial by signing up via AWS Marketplace and quickly spin up a deployment in minutes on any of the Elastic Cloud regions on AWS around the world. Your AWS Marketplace purchase of Elastic will be included in your monthly consolidated billing statement and will draw against your committed spend with AWS.

Additional logging resources:

• Getting started with logging on Elastic (quickstart)
• Ingesting common known logs via integrations (compute node example)
• List of integrations
• Ingesting custom application logs into Elastic
• Enriching logs in Elastic
• Analyzing Logs with Anomaly Detection (ML) and AIOps

Common use case examples with logs:

• Nginx log management
• AWS VPC Flow log management
• Using OpenAI to analyze Kubernetes errors
• PostgreSQL issue analysis with AIOps