Skip to content

[Metricbeat] Add JetStream metricset to NATS module#43310

Merged
rdner merged 95 commits intoelastic:mainfrom
JAndritsch:nats-jetstream
May 6, 2025
Merged

[Metricbeat] Add JetStream metricset to NATS module#43310
rdner merged 95 commits intoelastic:mainfrom
JAndritsch:nats-jetstream

Conversation

@JAndritsch
Copy link
Contributor

@JAndritsch JAndritsch commented Mar 17, 2025 •
edited
Loading

Proposed commit message

feat(metricbeat): Add JetStream metricset to NATS module

  • Add ability to report metrics on JetStream stats, accounts, streams, and consumers
  • Update supported NATS versions to 2.2.6 - 2.11+
  • Add additional data points to connection metricset
  • Fix compatibility issues in existing NATS metricsets in regards to newer versions of NATS

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Those using older versions of NATS (<2.2.6) may find compatibility issues with the module. Support for v1 has been dropped.

Author's Checklist

  • Go unit tests were added for the jetstream metricset.
  • All unit tests for the NATS module pass.
  • Code coverage relating to unit tests is sufficient
  • Go/Python integration tests for the jetstream metricset were added
  • Go/Python integration tests for the NATS module pass
  • Metrics collected by the NATS module are compatible with NATS versions between 2.2.6 and 2.11.x (latest)
  • The connection metricset no longer produces errors related to schema application when ran against newer versions of NATS
  • The jetstream metricset (new) can be configured to collect metrics about JetStream stats, accounts, streams, and consumers
  • Configuration for the jetstream metricset supports filtering on account names, stream names, and consumer names. No filters configured report on everything. These filters can compound.
  • Fields added to existing metrics are documented
  • Fields introduced in new jetstream metricset are all documented
  • Events produced by the jetstream metricset are ingested properly into Elasticsearch (confirmation of valid mappings)

How to test this PR locally

  1. Clone and check out the branch
  2. cd into metricbeat
  3. Run mage update
  4. Run mage build
  5. cd into modules/nats and run docker compose up -d to deploy a NATS 2.11 cluster
  6. Deploy an Elastic Stack composed of at least Elasticsearch and Kibana
  7. Set up a metricbeat.yml config to enable all metricsets for the nats module (see the reference module config). Configure the hosts setting to match the local NATS server running in Docker (run docker ps to obtain the mapped port). Configure the output to be the local Elasticsearch server running in Docker.
  8. Confirm that metrics from the nats module eventually appear in Discover. Check to make sure there are metrics for the jetstream metricset.
  9. Confirm there are no errors logged by Metricbeat nor any error events relating to the nats module ingested into Elasticsearch

Related issues

Use cases

Scenario: Upgrading to newer versions of NATS and/or Metricbeat
Given a system using Metricbeat's NATS module to monitor a NATS server or cluster
When either Metricbeat or NATS are upgraded
Then there should be no compatibility issues in the collection of NATS metrics
And relevant metrics should be reported

Scenario: Monitoring JetStream streams and consumers
Given a system using NATS JetStream
When deploying Metricbeat to monitor the NATS server
Then Metricbeat should report metrics about JetStream stats, accounts, streams, and consumers

Screenshots

Logs
JAndritsch added 30 commits March 16, 2025 18:15
@JAndritsch
WIP: Jetstream metricsets
8c0a8db
@JAndritsch
Add license headers
9fe3dec
@JAndritsch
Apply structure to specific metricsets
9683785
@JAndritsch
WIP: fields definition
986ecfd
@JAndritsch
WIP: Stats working. Need to fix stream/consumer
75212e1
@JAndritsch
Reference proper metricset name in error
6c8123b
@JAndritsch
Add cluster info to consumers
2bed854
@JAndritsch
Apply the correct schemas
64290fd
@JAndritsch
Fix mapping bug, adjust detail levels based on metricset
361ca1d
@JAndritsch
Ensure early return if metricset was closed
a740be0
@JAndritsch
Fix race condition that causes a panic
e53d622 
Attempting to generate the moduleFields once and share them across
multiple mb.Event instances results in a fatal error: concurrent
map iteration and map write panic.

Instead, generate a unique set of module fields for each metricset
event.
@JAndritsch
WIP: config for jetstream metricsets
ad0a612 
Idea is to allow filtering on:
- stream names
- consumer names

Could potentially add more filtering options later
@JAndritsch
Add filtering for stream and consumer names
2d32b68 
Add demo yml config
@JAndritsch
Add more data
4d50434
@JAndritsch
Allow filter on account, stream, and consumer names
0ec8fe9
@JAndritsch
Allow configuring response size limits
91aafae
@JAndritsch
Fix schema mapping for subjects
4e103ee
@JAndritsch
WIP: fields.yml docs
0ffd790 
Also remove CompressOK. It's always true in JetStream
@JAndritsch
More fields
a5c2154
@JAndritsch
More fields.yml updates
77ba4c1
@JAndritsch
Change unsigned_long to long
124c723 
It seems 'mage update' doesn't recognize this type
@JAndritsch
Merge into a single metricset (jetstream)
d252a79 
Use configuration options to specify which sub-category of metrics
should be reported.
@JAndritsch
Remove max_limit config.
f65f14e 
API does not seem to respond to it.
@JAndritsch
Add reporting of account details.
3e86a30 
Update and commit anything documentation-related
@JAndritsch
Update module and reference config
73f7c05
@JAndritsch
Remove sample config file
00a29ba
@JAndritsch
Update metricbeat.reference.yml
87a1c91
@JAndritsch
Add basic unit tests for jetstream
02c40d3
@JAndritsch
WIP: Adding better test coverage
67736d2
@JAndritsch
Update docs and module.d config
bee19d9
@rdner
Copy link
Member

rdner commented May 5, 2025

/test
@JAndritsch
Copy link
Contributor Author

JAndritsch commented May 5, 2025

@rdner Are the test failures for libbeat related to this PR? I can't access the buildkite results (still getting a 404) and I don't have any test failures when running mage goIntegTest or mage pythonIntegTest.
@mauri870
Copy link
Member

mauri870 commented May 5, 2025

@rdner Are the test failures for libbeat related to this PR? I can't access the buildkite results (still getting a 404) and I don't have any test failures when running mage goIntegTest or mage pythonIntegTest.

For security reasons, the Buildkite pipelines are private. All the failures appear to be unrelated to the changes in this PR. I’ll retrigger the tests.
@rdner
Copy link
Member

rdner commented May 5, 2025

/test
@rdner rdner merged commit 6e3634f into elastic:main May 6, 2025
194 checks passed
@JAndritsch
Copy link
Contributor Author

JAndritsch commented May 6, 2025

Thanks everyone!
@JAndritsch JAndritsch deleted the nats-jetstream branch May 6, 2025 11:35
dliappis added a commit to dliappis/beats that referenced this pull request May 6, 2025
@dliappis
Remove nats Dockerfile from bump Golang action
f6473c7 
This commit removes the nats Dockerfile from the bump-golang
GH action, since after elastic#43310 it no longer references golang
docker images.
@dliappis dliappis mentioned this pull request May 6, 2025
Merged
dliappis added a commit that referenced this pull request May 7, 2025
@dliappis
Remove nats Dockerfile from bump Golang action (#44221)
0c52ff2 
This commit removes the nats Dockerfile from the bump-golang
GH action, since after #43310 it no longer references golang
docker images.
@colleenmcginnis colleenmcginnis mentioned this pull request Jul 17, 2025
Closed
colleenmcginnis pushed a commit to colleenmcginnis/beats that referenced this pull request Jul 30, 2025
add version information related to elastic#43310
5f2e2db
@colleenmcginnis colleenmcginnis mentioned this pull request Jul 30, 2025
Closed
@rdner rdner mentioned this pull request Aug 7, 2025
Merged
This was referenced Aug 7, 2025
Merged
Merged
colleenmcginnis pushed a commit to colleenmcginnis/beats that referenced this pull request Aug 7, 2025
add versioning information to fields.yml files related to elastic#43310
bacaa82
colleenmcginnis pushed a commit to colleenmcginnis/beats that referenced this pull request Aug 7, 2025
add versioning information to fields.yml files related to elastic#43310
db73894
@colleenmcginnis colleenmcginnis mentioned this pull request Aug 7, 2025
Merged
6 tasks
colleenmcginnis pushed a commit to colleenmcginnis/beats that referenced this pull request Aug 8, 2025
add versioning information to fields.yml files related to elastic#43310
88b8efd
This was referenced Aug 18, 2025
Closed
Closed
colleenmcginnis pushed a commit that referenced this pull request Aug 18, 2025
[docs] Adds applies_to to changes from #43310 (#45846)
b814855 
* add versioning information to fields.yml files related to #43310

* run update

* add tabs for example document in docs/reference/metricbeat/metricbeat-metricset-nats-connection.md

* add tabs for example document in docs/reference/metricbeat/metricbeat-metricset-nats-stats.md

* apply suggestions from code review
mergify bot pushed a commit that referenced this pull request Aug 18, 2025
@mergify
[docs] Adds applies_to to changes from #43310 (#45846)
8436404 
* add versioning information to fields.yml files related to #43310

* run update

* add tabs for example document in docs/reference/metricbeat/metricbeat-metricset-nats-connection.md

* add tabs for example document in docs/reference/metricbeat/metricbeat-metricset-nats-stats.md

* apply suggestions from code review

(cherry picked from commit b814855)
@mergify mergify bot mentioned this pull request Aug 18, 2025
Merged
6 tasks
rdner pushed a commit that referenced this pull request Aug 18, 2025
@mergify
[docs] Adds applies_to to changes from #43310 (#45846) (#46052)
3520743 
* add versioning information to fields.yml files related to #43310

* run update

* add tabs for example document in docs/reference/metricbeat/metricbeat-metricset-nats-connection.md

* add tabs for example document in docs/reference/metricbeat/metricbeat-metricset-nats-stats.md

* apply suggestions from code review

(cherry picked from commit b814855)

Co-authored-by: Colleen McGinnis <colleen.mcginnis@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team Team:obs-ds-hosted-services Label for the Observability Hosted Services team

8 participants

@JAndritsch @elasticmachine @rdner @lalit-satapathy @devamanv @mauri870 @kilfoyle @pierrehilbert