Skip to content

[axonius] Add initial integration with Adapter data stream#16171

Merged
kcreddy merged 7 commits intoelastic:feature/axonius-0.1.0from
muskan-agarwal26:datastream-adapter
Dec 31, 2025
Merged

[axonius] Add initial integration with Adapter data stream#16171
kcreddy merged 7 commits intoelastic:feature/axonius-0.1.0from
muskan-agarwal26:datastream-adapter

Conversation

@muskan-agarwal26
Copy link
Contributor

@muskan-agarwal26 muskan-agarwal26 commented Dec 1, 2025

Proposed commit message

This introduces a new Axonius integration for cybersecurity asset management, enabling collection of adapter health and performance data through the Axonius API. The integration provides visibility into adapter status, connection health, and configuration details to help teams maintain reliable data ingestion and identify problematic integrations.

Test samples were derived from live data samples, which were subsequently sanitized.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

To test the axonius package:

  • Clone integrations repo.
  • Install elastic package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/axonius directory.
  • Run the following command to run tests.

elastic-package test

elastic-package test -v
2025/12/01 12:14:03 DEBUG Enable verbose logging
2025/12/01 12:14:03 DEBUG checking latest release in Github
Run asset tests for the package
2025/12/01 12:14:03 DEBUG Connecting with Kibana host from current profile (profile: default, host: "https://127.0.0.1:5601")
2025/12/01 12:14:03 DEBUG Running tests sequentially
2025/12/01 12:14:03 DEBUG installing package...
2025/12/01 12:14:03 DEBUG Build directory: /home/devuser/integrations/build/packages/axonius/0.1.0
2025/12/01 12:14:03 DEBUG Clear target directory (path: /home/devuser/integrations/build/packages/axonius/0.1.0)
2025/12/01 12:14:03 DEBUG Copy package content (source: /home/devuser/integrations/packages/axonius)
2025/12/01 12:14:03 DEBUG Copy license file if needed
2025/12/01 12:14:03 DEBUG No license text file is included in package
2025/12/01 12:14:03 DEBUG Include linked files
2025/12/01 12:14:03 DEBUG Encode dashboards
2025/12/01 12:14:03 DEBUG Resolve external fields
2025/12/01 12:14:03 DEBUG Package has external dependencies defined
2025/12/01 12:14:04 DEBUG data_stream/adapter/fields/base-fields.yml: source file has been changed
2025/12/01 12:14:04 DEBUG data_stream/adapter/fields/ecs.yml: source file has been changed
2025/12/01 12:14:04 DEBUG Package doesn't have to import ECS mappings
2025/12/01 12:14:04 DEBUG Build zipped package
2025/12/01 12:14:04 DEBUG Compress using archives.Zip (destination: /home/devuser/integrations/build/packages/axonius-0.1.0.zip)
2025/12/01 12:14:04 DEBUG Skip validation of the built .zip package
2025/12/01 12:14:05 DEBUG removing package...
--- Test results for package: axonius - START ---
╭─────────┬─────────────┬───────────┬──────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                                                        │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼──────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ axonius │             │ asset     │ dashboard axonius-45c76e49-59d4-4f27-85f5-7f32edf6e0e5 is loaded │ PASS   │      2.462µs │
│ axonius │             │ asset     │ search axonius-18fe2e86-4290-48e4-ab15-86ae9cdc2473 is loaded    │ PASS   │        281ns │
│ axonius │             │ asset     │ search axonius-608023db-f45b-4afc-8c58-1eedc86bdc76 is loaded    │ PASS   │        225ns │
│ axonius │ adapter     │ asset     │ index_template logs-axonius.adapter is loaded                    │ PASS   │        219ns │
│ axonius │ adapter     │ asset     │ ingest_pipeline logs-axonius.adapter-0.1.0 is loaded             │ PASS   │        186ns │
╰─────────┴─────────────┴───────────┴──────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: axonius - END   ---
Done
Run pipeline tests for the package
2025/12/01 12:14:07 DEBUG Connecting with Elasticsearch host from current profile (profile: default, host: "https://127.0.0.1:9200")
2025/12/01 12:14:07 DEBUG Running tests sequentially
2025/12/01 12:14:07 DEBUG Imported ECS fields definition from external schema for validation (embedded in package: false, stack uses ecs@mappings template: true)
2025/12/01 12:14:08 DEBUG Dump Elastic stack data
2025/12/01 12:14:08 DEBUG Dump stack logs for elasticsearch
--- Test results for package: axonius - START ---
╭─────────┬─────────────┬───────────┬─────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                                   │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼─────────────────────────────────────────────┼────────┼──────────────┤
│ axonius │ adapter     │ pipeline  │ (ingest pipeline warnings test-adapter.log) │ PASS   │ 310.602981ms │
│ axonius │ adapter     │ pipeline  │ test-adapter.log                            │ PASS   │ 161.532603ms │
╰─────────┴─────────────┴───────────┴─────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: axonius - END   ---
Done
Run policy tests for the package
2025/12/01 12:14:08 DEBUG Connecting with Kibana host from current profile (profile: default, host: "https://127.0.0.1:5601")
--- Test results for package: axonius - START ---
No test results
--- Test results for package: axonius - END   ---
Done
Run script tests for the package
--- Test results for package: axonius - START ---
PKG axonius
[no test files]
--- Test results for package: axonius - END ---
Done
Run static tests for the package
2025/12/01 12:14:08 DEBUG Running tests sequentially
2025/12/01 12:14:08 DEBUG Imported ECS fields definition from external schema for validation (embedded in package: false, stack uses ecs@mappings template: true)
--- Test results for package: axonius - START ---
╭─────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ axonius │ adapter     │ static    │ Verify sample_event.json │ PASS   │ 127.738215ms │
╰─────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: axonius - END   ---
Done
Run system tests for the package
2025/12/01 12:14:08 DEBUG Connecting with Kibana host from current profile (profile: default, host: "https://127.0.0.1:5601")
2025/12/01 12:14:08 DEBUG Connecting with Elasticsearch host from current profile (profile: default, host: "https://127.0.0.1:9200")
2025/12/01 12:14:08 DEBUG Running suite...
2025/12/01 12:14:08 DEBUG Running system tests for data stream "adapter"
2025/12/01 12:14:08 DEBUG System runner: data stream "adapter" config file "test-default-config.yml" variant ""
2025/12/01 12:14:08  INFO Installing package...
2025/12/01 12:14:08 DEBUG Build directory: /home/devuser/integrations/build/packages/axonius/0.1.0
2025/12/01 12:14:08 DEBUG Clear target directory (path: /home/devuser/integrations/build/packages/axonius/0.1.0)
2025/12/01 12:14:08 DEBUG Copy package content (source: /home/devuser/integrations/packages/axonius)
2025/12/01 12:14:08 DEBUG Copy license file if needed
2025/12/01 12:14:08 DEBUG No license text file is included in package
2025/12/01 12:14:08 DEBUG Include linked files
2025/12/01 12:14:08 DEBUG Encode dashboards
2025/12/01 12:14:08 DEBUG Resolve external fields
2025/12/01 12:14:08 DEBUG Package has external dependencies defined
2025/12/01 12:14:08 DEBUG data_stream/adapter/fields/base-fields.yml: source file has been changed
2025/12/01 12:14:08 DEBUG data_stream/adapter/fields/ecs.yml: source file has been changed
2025/12/01 12:14:08 DEBUG Package doesn't have to import ECS mappings
2025/12/01 12:14:08 DEBUG Build zipped package
2025/12/01 12:14:08 DEBUG Compress using archives.Zip (destination: /home/devuser/integrations/build/packages/axonius-0.1.0.zip)
2025/12/01 12:14:09 DEBUG Skip validation of the built .zip package
2025/12/01 12:14:20 DEBUG Running tests sequentially
2025/12/01 12:14:20 DEBUG Using config: "default"
2025/12/01 12:14:20  INFO Running test for data_stream "adapter" with configuration 'default'
2025/12/01 12:14:20 DEBUG creating enroll policy...
2025/12/01 12:14:24 DEBUG creating test policy...
2025/12/01 12:14:28  INFO Setting up independent Elastic Agent...
2025/12/01 12:14:28 DEBUG setting up agent using Docker Compose agent deployer
[+] Running 2/2
 ✔ Network elastic-package-agent-axonius-adapter-22978_default            Created                                                                                 0.1s 
 ✔ Container elastic-package-agent-axonius-adapter-22978-elastic-agent-1  Started                                                                                 0.2s 
2025/12/01 12:14:29 DEBUG Wait for healthy containers: 8a89b9d6f6eb
2025/12/01 12:14:29 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:30 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:31 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:32 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:33 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:34 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:35 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:36 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:37 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:38 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:39 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:40 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:41 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:42 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:43 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:44 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:46 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:47 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:48 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:49 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:50 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:51 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:52 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:53 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:54 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:55 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:56 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:57 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:58 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:14:59 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:00 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:01 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:02 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:03 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:04 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:05 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:06 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:07 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:08 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:09 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:10 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:11 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:12 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:13 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:14 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:15 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:16 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:17 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:18 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:19 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:20 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:21 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:22 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:23 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:24 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:25 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:26 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:27 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:28 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:29 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:30 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:31 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:32 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:34 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:35 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:36 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:37 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:38 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:39 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:40 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:41 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:42 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:43 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:44 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:45 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:46 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:47 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:48 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:49 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:50 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:51 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:52 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:53 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:54 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:55 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:56 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:57 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:58 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:15:59 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:16:00 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:16:01 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:16:02 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:16:03 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:16:04 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:16:05 DEBUG Container elastic-agent (8a89b9d6f6eb) status: unhealthy
2025/12/01 12:16:06 DEBUG Container elastic-agent (8a89b9d6f6eb) status: running (health: healthy)
2025/12/01 12:16:06 DEBUG adding service container elastic-package-agent-axonius-adapter-22978-elastic-agent-1 internal ports to context
2025/12/01 12:16:06 DEBUG found 1 enrolled agent(s)
2025/12/01 12:16:06 DEBUG Selected enrolled agent "2ccc4f8f-1530-4811-9865-8de0166eef02"
2025/12/01 12:16:06  INFO Setting up service...
2025/12/01 12:16:06 DEBUG setting up service using Docker Compose service deployer
WARN[0000] /home/devuser/integrations/packages/axonius/_dev/deploy/docker/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
[+] Running 2/2
 ✔ Network elastic-package-service-89188_default      Created                                                                                                     0.1s 
 ✔ Container elastic-package-service-89188-axonius-1  Started                                                                                                     0.3s 
WARN[0000] /home/devuser/integrations/packages/axonius/_dev/deploy/docker/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
2025/12/01 12:16:07 DEBUG Wait for healthy containers: 0b1457f9d102
2025/12/01 12:16:07 DEBUG Container axonius (0b1457f9d102) status: running (no health status)
2025/12/01 12:16:07 DEBUG adding service container elastic-package-service-89188-axonius-1 internal ports to context
WARN[0000] /home/devuser/integrations/packages/axonius/_dev/deploy/docker/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
2025/12/01 12:16:07 DEBUG adding package data stream to test policy...
2025/12/01 12:16:11 DEBUG found 1 enrolled agent(s)
2025/12/01 12:16:11 DEBUG Selected enrolled agent "2ccc4f8f-1530-4811-9865-8de0166eef02"
2025/12/01 12:16:11 DEBUG Set Debug log level to agent
2025/12/01 12:16:13 DEBUG assigning package data stream to agent...
2025/12/01 12:16:13 DEBUG Wait until the policy (ID: a00b6c3b-cea8-480b-a35f-ab0fe6ebec03, revision: 2) is assigned to the agent (ID: 2ccc4f8f-1530-4811-9865-8de0166eef02)...
2025/12/01 12:16:14 DEBUG Agent 2ccc4f8f-1530-4811-9865-8de0166eef02 (Host: elastic-agent-22978): Policy ID a00b6c3b-cea8-480b-a35f-ab0fe6ebec03 LogLevel: info Status: updating
2025/12/01 12:16:15 DEBUG Agent 2ccc4f8f-1530-4811-9865-8de0166eef02 (Host: elastic-agent-22978): Policy ID a00b6c3b-cea8-480b-a35f-ab0fe6ebec03 LogLevel: info Status: updating
2025/12/01 12:16:17 DEBUG Agent 2ccc4f8f-1530-4811-9865-8de0166eef02 (Host: elastic-agent-22978): Policy ID a00b6c3b-cea8-480b-a35f-ab0fe6ebec03 LogLevel: info Status: updating
2025/12/01 12:16:19 DEBUG Agent 2ccc4f8f-1530-4811-9865-8de0166eef02 (Host: elastic-agent-22978): Policy ID a00b6c3b-cea8-480b-a35f-ab0fe6ebec03 LogLevel: info Status: online
2025/12/01 12:16:19 DEBUG Policy revision assigned to the agent (ID: 2ccc4f8f-1530-4811-9865-8de0166eef02)...
2025/12/01 12:16:19 DEBUG checking for expected data in data stream (10m0s)...
2025/12/01 12:16:19 DEBUG found 0 hits in logs-axonius.adapter-43409 data stream
2025/12/01 12:16:20 DEBUG found 0 hits in logs-axonius.adapter-43409 data stream
2025/12/01 12:16:21 DEBUG found 0 hits in logs-axonius.adapter-43409 data stream
2025/12/01 12:16:22 DEBUG found 0 hits in logs-axonius.adapter-43409 data stream
2025/12/01 12:16:23 DEBUG found 0 hits in logs-axonius.adapter-43409 data stream
2025/12/01 12:16:24 DEBUG found 2 hits in logs-axonius.adapter-43409 data stream
2025/12/01 12:16:29 DEBUG found 2 hits in logs-axonius.adapter-43409 data stream
WARN[0000] /home/devuser/integrations/packages/axonius/_dev/deploy/docker/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
2025/12/01 12:16:29 DEBUG Found 0 deprecation warnings for data stream logs-axonius.adapter-43409
2025/12/01 12:16:29 DEBUG Check whether or not synthetic source mode is enabled (data stream logs-axonius.adapter-43409)...
2025/12/01 12:16:29 DEBUG Data stream logs-axonius.adapter-43409 has synthetic source mode enabled: false
2025/12/01 12:16:29  INFO Validating test case...
2025/12/01 12:16:29 DEBUG Imported ECS fields definition from external schema for validation (embedded in package: false, stack uses ecs@mappings template: true)
2025/12/01 12:16:29 DEBUG Performing validation based on mappings
2025/12/01 12:16:29 DEBUG Get Mappings from data stream (logs-axonius.adapter-43409)
2025/12/01 12:16:29 DEBUG Simulate Index Template (logs-axonius.adapter)
2025/12/01 12:16:29 DEBUG assert hit count expected 2, observed 2
2025/12/01 12:16:29  INFO Tearing down service...
2025/12/01 12:16:29 DEBUG tearing down service using Docker Compose runner
WARN[0000] /home/devuser/integrations/packages/axonius/_dev/deploy/docker/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
[+] Stopping 1/1
 ✔ Container elastic-package-service-89188-axonius-1  Stopped                                                                                                     0.4s 
WARN[0000] /home/devuser/integrations/packages/axonius/_dev/deploy/docker/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
2025/12/01 12:16:30  INFO Write container logs to file: /home/devuser/integrations/build/container-logs/axonius-1764571590355851522.log
WARN[0000] /home/devuser/integrations/packages/axonius/_dev/deploy/docker/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
[+] Running 2/2
 ✔ Container elastic-package-service-89188-axonius-1  Removed                                                                                                     0.0s 
 ✔ Network elastic-package-service-89188_default      Removed                                                                                                     0.2s 
2025/12/01 12:16:30 DEBUG Deleting data stream for testing logs-axonius.adapter-43409
2025/12/01 12:16:30 DEBUG removing agent...
2025/12/01 12:16:32  INFO Tearing down agent...
2025/12/01 12:16:32 DEBUG tearing down agent using Docker Compose runner
2025/12/01 12:16:32  INFO Write container logs to file: /home/devuser/integrations/build/container-logs/elastic-agent-1764571592240521216.log
[+] Running 2/2
 ✔ Container elastic-package-agent-axonius-adapter-22978-elastic-agent-1  Removed                                                                                 1.5s 
 ✔ Network elastic-package-agent-axonius-adapter-22978_default            Removed                                                                                 0.3s 
2025/12/01 12:16:34 DEBUG deleting test policies...
2025/12/01 12:16:39 DEBUG Dump Elastic stack data (location: /tmp/test-system-1962115970)
2025/12/01 12:16:39 DEBUG Dump stack logs for elastic-agent
2025/12/01 12:16:39 DEBUG Dump stack logs for fleet-server
2025/12/01 12:16:40 DEBUG Dump stack logs for kibana
2025/12/01 12:16:40 DEBUG Dump stack logs for elasticsearch
2025/12/01 12:16:40 DEBUG Dump stack logs for package-registry
2025/12/01 12:16:40  INFO Uninstalling package...
--- Test results for package: axonius - START ---
╭─────────┬─────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├─────────┼─────────────┼───────────┼───────────┼────────┼───────────────┤
│ axonius │ adapter     │ system    │ default   │ PASS   │ 2m8.92928557s │
╰─────────┴─────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: axonius - END   ---
Done

Related issues

Screenshots

image (5) image (6)

Go Code for Ingest Pipeline Generation

The adapter data stream pipeline is generated using Go code built on top of the Dispear library.
Below is the code used for generating the pipeline logic:

package main

import (
	"fmt"

	. "github.com/efd6/dispear"
)

const (
	ECSVersion = "9.2.0"
	PkgRoot    = "axonius.adapter"
)
const errorFormat = "Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}"

// removeErrorHandler creates a slice of Renderer objects that will remove the specified field
// from a document. If addErrorMessage is provided and set to false, only the removal operation
// will be performed. By default, or if addErrorMessage is true, the function will also append
// a predefined error message to the 'error.message' field. This is useful in scenarios such as
// an Ingest Pipeline where a problematic field needs to be removed from the document and an
// error message should be appended to indicate the removal.
//
// Usage:
//
//	renderers := removeErrorHandler("field_name") // Default behavior: removes field and appends error message
//	renderers := removeErrorHandler("field_name", false) // Removes field without appending error message
func removeErrorHandler(f string, addErrorMessage ...bool) []Renderer {
	// Default value for addErrorMessage is true
	addError := true
	// If addErrorMessage is provided, override the default value
	if len(addErrorMessage) > 0 {
		addError = addErrorMessage[0]
	}

	// Create a slice of Renderer to remove the specified field
	renderers := []Renderer{REMOVE(f)}

	// If addError is true, append the error message Renderer
	if addError {
		renderers = append(renderers, APPEND("error.message", errorFormat))
	}

	return renderers
}

func main() {

	// Initial processors of pipeline

	DESCRIPTION("Pipeline for processing adapter logs.")

	SET("ecs.version").
		VALUE(ECSVersion).
		TAG("set ecs.version to 9.2.0")

	TERMINATE("data collection error").
		IF("ctx.error?.message != null && ctx.message == null && ctx.event?.original == null").
		DESCRIPTION("error message set and no data to process.")

	BLANK()
	BLANK().COMMENT("remove agentless metadata")

	REMOVE(
		"organization",
		"division",
		"team",
	).
		IF("ctx.organization instanceof String && ctx.division instanceof String && ctx.team instanceof String").
		IGNORE_MISSING(true).
		TAG("remove_agentless_tags").
		DESCRIPTION("Removes the fields added by Agentless as metadata, as they can collide with ECS fields.")

	BLANK()
	BLANK().COMMENT("parse the event JSON")

	RENAME("message", "event.original").
		IF("ctx.event?.original == null").
		DESCRIPTION("Renames the original `message` field to `event.original` to store a copy of the original message. The `event.original` field is not touched if the document already has one; it may happen when Logstash sends the document.").
		IGNORE_MISSING(true)

	REMOVE("message").
		TAG("remove_message").
		IF("ctx.event?.original != null").
		DESCRIPTION("The `message` field is no longer required if the document has an `event.original` field.").
		IGNORE_MISSING(true)

	JSON(PkgRoot, "event.original")

	// Add fingerprint

	BLANK()
	BLANK().COMMENT("Add fingerprint")

	FINGERPRINT("_id", "axonius.adapter.id").IGNORE_MISSING(true)

	// Setting event.* fields

	BLANK()
	BLANK().COMMENT("Set event.* fields")

	SET("event.kind").
		VALUE("event").
		TAG("set event.kind to event")

	// Use Date processors

	BLANK()
	BLANK().COMMENT("Date processors")

	// Date processors over axonius.adapter.connections array

	for _, field := range []string{
		"last_fetch_time",
		"last_successful_fetch",
	} {
		FOREACH("axonius.adapter.connections",
			DATE("_ingest._value."+field, "_ingest._value."+field, "EEE, dd MMM yyyy HH:mm:ss 'GMT'", "yyyy-MM-dd").
				ON_FAILURE(removeErrorHandler("_ingest._value."+field, false)...),
		).IF("ctx.axonius?.adapter?.connections instanceof List")
	}

	// Convert to Long

	BLANK()
	BLANK().COMMENT("Convert to Long")

	for _, field := range []string{
		"axonius.adapter.connections_count.error_count",
		"axonius.adapter.connections_count.inactive_count",
		"axonius.adapter.connections_count.success_count",
		"axonius.adapter.connections_count.total_count",
		"axonius.adapter.connections_count.warning_count",
	} {
		CONVERT("", field, "long").
			IGNORE_MISSING(true).
			ON_FAILURE(removeErrorHandler(field)...)
	}

	// Convert to boolean

	BLANK()
	BLANK().COMMENT("Convert to Boolean")

	for _, field := range []string{
		"axonius.adapter.is_master",
	} {
		CONVERT("", field, "boolean").
			IGNORE_MISSING(true).
			ON_FAILURE(removeErrorHandler(field)...)
	}

	// Convert to boolean over axonius.adapter.connections array

	for _, field := range []string{
		"active",
		"failed_connections_limit_exceeded",
		"did_notify_error",
	} {
		FOREACH("axonius.adapter.connections",
			CONVERT("", "_ingest._value."+field, "boolean").
				IGNORE_MISSING(true).
				ON_FAILURE(removeErrorHandler("_ingest._value."+field)...),
		).IF("ctx.axonius?.adapter?.connections instanceof List")
	}

	// Set ECS Mapping

	BLANK()
	BLANK().COMMENT("Map custom fields to corresponding ECS and related fields.")

	// Map ECS mapping for top-level fields

	for _, mapping := range []struct {
		ecsField, customField string
	}{
		{ecsField: "event.id", customField: "axonius.adapter.id"},
	} {
		SET(mapping.ecsField).
			COPY_FROM(mapping.customField).
			IGNORE_EMPTY(true).
			TAG(fmt.Sprintf("set %s from %s", mapping.ecsField, mapping.customField))
	}

	// Set event.outcome based on status

	SET("event.outcome").
		VALUE("success").
		IF("ctx.axonius?.adapter?.status != null && ctx.axonius.adapter.status.toLowerCase() == 'success'").
		TAG("set event.outcome to success")

	SET("event.outcome").
		VALUE("failure").
		IF("ctx.axonius?.adapter?.status != null && ctx.axonius.adapter.status.toLowerCase() == 'error'").
		TAG("set event.outcome to failure")

	// Remove Duplicate Fields.

	BLANK()
	BLANK().COMMENT("Remove duplicate custom fields if preserve_duplicate_custom_fields are not enabled")

	REMOVE(
		"axonius.adapter.id",
	).
		IF("ctx.tags == null || !ctx.tags.contains('preserve_duplicate_custom_fields')").
		TAG("remove_custom_duplicate_fields").
		IGNORE_MISSING(true)

	// Clean up script

	BLANK()
	BLANK().COMMENT("Cleanup")

	SCRIPT().
		TAG("script_to_drop_null_values").
		DESCRIPTION("This script processor iterates over the whole document to remove fields with null values.").
		LANG("painless").
		SOURCE(`
		void handleMap(Map map) {
		map.values().removeIf(v -> {
			if (v instanceof Map) {
			handleMap(v);
			} else if (v instanceof List) {
			handleList(v);
			}
			return v == null || v == '' || (v instanceof Map && v.size() == 0) || (v instanceof List && v.size() == 0)
		});
		}
		void handleList(List list) {
		list.removeIf(v -> {
			if (v instanceof Map) {
			handleMap(v);
			} else if (v instanceof List) {
			handleList(v);
			}
			return v == null || v == '' || (v instanceof Map && v.size() == 0) || (v instanceof List && v.size() == 0)
		});
		}
		handleMap(ctx);
		`)

	// Set and Append processor on last

	SET("event.kind").
		VALUE("pipeline_error").
		IF("ctx.error?.message != null").
		TAG("set event.kind to pipeline_error")

	APPEND("tags", "preserve_original_event").
		IF("ctx.error?.message != null").
		ALLOW_DUPLICATES(false)

	// Global on failure processor

	ON_FAILURE(
		APPEND("error.message", errorFormat),
		SET("event.kind").VALUE("pipeline_error").TAG("set event.kind to pipeline_error"),
		APPEND("tags", "preserve_original_event").
			ALLOW_DUPLICATES(false),
	)

	// Generate the pipeline

	Generate()
}
@muskan-agarwal26 muskan-agarwal26 requested a review from a team as a code owner December 1, 2025 08:26
@muskan-agarwal26 muskan-agarwal26 changed the base branch from main to feature/axonius-0.1.0 December 1, 2025 08:26
@andrewkroh andrewkroh added documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:axonius [Integration not found in source] Crest Contributions from Crest developement team. New Integration Issue or pull request for creating a new integration package. dashboard Relates to a Kibana dashboard bug, enhancement, or modification. labels Dec 1, 2025
Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you are using and older version of dispear. Please update the version the go.mod to the latest version.

Rémy suggests the following commit message:

axonius: add initial integration with adapter data stream

This introduces a new Axonius integration for cybersecurity asset 
management, enabling collection of adapter health and performance data 
through the Axonius API. The integration provides visibility into 
adapter status, connection health, and configuration details to help 
teams maintain reliable data ingestion and identify problematic 
integrations.

Test samples were derived from live data samples, which were subsequently
sanitized.
Comment on lines 228 to 242
- set:
tag: set_event_id
field: event.id
copy_from: axonius.adapter.id
ignore_empty_value: true
- set:
tag: set_event_outcome_1
if: ctx.axonius?.adapter?.status != null && ctx.axonius.adapter.status.toLowerCase() == 'success'
field: event.outcome
value: success
- set:
tag: set_event_outcome_2
if: ctx.axonius?.adapter?.status != null && ctx.axonius.adapter.status.toLowerCase() == 'error'
field: event.outcome
value: failure
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tags that dispear.SET autocreates are not as informative as they really should be (this is a limitation that arises from the nature of the set processor. So suggest making an explicit tag, e.g. .TAG(fmt.Sprintf("set %s to %s", field, value) for the event.outcome cases (this can be a for loop if you want). Similarly for the one before that, .TAG(fmt.Sprintf("set %s from %s", mapping.ecsField, mapping.customField).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you update the generator in the description to match the new pipeline? Note also the " to " and " from " in the format strings in my suggestion.

Also, can you tell me what the github.com/efd6/dispear line in the go.mod is? This still does not look like the behaviour I would expect from the latest version.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the current dispear version is outdated (v0.0.0-20250915050336-00bed7647aca).

We’ll update the script and switch to the latest official version.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this has happened yet.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@muskan-agarwal26, this appears to be still pending. Also the commit message update that @efd6 mentioned here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, missed it, will update it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@efd6 , @kcreddy ,
I have made both the pending changes, kindly have a look.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@muskan-agarwal26, can you update the Go code above in the PR description as well?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I have updated that too.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

1. Remove transform from readme and final new line.
2. Updated tags and indentation in scriptprocessor.
@muskan-agarwal26 muskan-agarwal26 requested a review from efd6 December 2, 2025 07:16
@narph narph added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Dec 9, 2025
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@muskan-agarwal26 muskan-agarwal26 changed the title [axonius][adapter] Add Axonius Adapter datastream Dec 30, 2025
Copy link
Contributor

@kcreddy kcreddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed the pending comments from Dan's review. LGTM!

@kcreddy kcreddy merged commit 27cbc7f into elastic:feature/axonius-0.1.0 Dec 31, 2025
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Crest Contributions from Crest developement team. dashboard Relates to a Kibana dashboard bug, enhancement, or modification. documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:axonius [Integration not found in source] New Integration Issue or pull request for creating a new integration package. Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

7 participants