ECE Adminconsole logs by philippkahr · Pull Request #13864 · elastic/integrations

philippkahr · 2025-05-09T13:02:51Z

Hey!

This is a big new integration that reads the adminconsole.ndjson. In certain enterprise setups we need an external tool that reads this file from the disk, for audit purposes. The log itself is already in NDJSON.

Currently this log is missing in the ECE logging-and-metrics cluster, but we need it in another cluster anyway.

I've created my own ECE, clicked through all the APIs and tried to capture all the needed documents. It is a bit rough here and there, but I wanted to prep it for future enhancements.

I haven't yet build any dashboards. What I would be after mostly is probably alerts but we cannot add alerts to integrations?

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

Integration UI

daniela-elastic · 2025-05-28T15:09:05Z

In observability we don't have a way yet to add alert presets to integrations (it's something we want to have at some point). I believe security have a way to add OOTB alerts CC @jamiehynds for details

jamiehynds · 2025-05-28T18:48:28Z

In observability we don't have a way yet to add alert presets to integrations (it's something we want to have at some point). I believe security have a way to add OOTB alerts CC @jamiehynds for details

Thanks for the contribution @philippkahr. Although not particularly common, security rules are a supported asset type with integrations (see here).

As for a responsible team for PR review and ownership of this integration - the stack management team may be the best fit @consulthys is the PM on that team.

consulthys · 2025-05-30T07:55:30Z

@jamiehynds

the stack management team may be the best fit @consulthys is the PM on that team.

Did you mean Stack Monitoring?

jamiehynds · 2025-05-30T08:09:59Z

@jamiehynds

the stack management team may be the best fit @consulthys is the PM on that team.

Did you mean Stack Monitoring?

Sorry, yes. Stack Monitoring.

consulthys · 2025-06-04T13:02:56Z

@philippkahr can you elaborate on the nature of those logs? Are they the same or similar in nature as what is being collected here for Cloud customers?

philippkahr · 2025-06-05T09:05:04Z

@philippkahr can you elaborate on the nature of those logs? Are they the same or similar in nature as what is being collected here for Cloud customers?

@consulthys hmm good question actually. I would say that this data is from the same origin. This takes the adminconsole.ndjson that is written to disk. This is available in ECE and it comes from the SDH, were we are not collecting this file. Additionally, we are not parsing it at all. That's why I generated this integration. I hope to properly parse it, configure it and ensure it is working.

philippkahr · 2025-07-01T08:24:47Z

Can we get some traction on this?

consulthys · 2025-07-01T13:15:32Z

@philippkahr Integrations are open and anyone can contribute, so I'd say feel free to get a crack at it and release it as beta once you've ensured that everything works properly. let me know once you think this PR is ready and we'll review it.

botelastic · 2025-07-31T17:16:58Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

P1llus

Had a chat today with Philip on this one, just adding the review notes in case others might look at this.

Let's also figure out why it ended up being here so long and stalled.

EDIT: Would also be good to add system tests, since this is filestream based we could add that from any other filestream integration, it is a folder in the top _dev and the datastream _dev folder called system.

P1llus · 2025-08-13T11:32:05Z

+        title: Collect logs from ECE Adminconsole
+        description: Collecting the ECE Adminconsole logs
+owner:
+  github: elastic/integrations


We have to decide on which team will own this, I will reach out and see if anyone can confirm what it should be.

philippkahr · 2025-08-18T14:21:56Z

Changes applied.

No more ignore_failure processors replaced dissect with a simple GROK
Removed unnecessary pipeline calls
fixed mapping
renamed data stream.
copied a build system test over and modified so it should work

I deployed a local ECE, used terraform to create, delete instances, click around the UI. This generated a total of ~24.000 events that are all parsed succesfully.

What am I waiting for to export the dashboard I made.

philippkahr · 2025-08-18T14:26:47Z

Do I still need to do an ecs.yml where I list all the fields I use from ECS?

bhapas · 2025-09-08T14:38:49Z

@elastic/stack-monitoring @consulthys Can you review this PR?

consulthys

Great job!! Just added a few comments

botelastic · 2026-01-14T10:48:44Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

consulthys

LGT Stack Monitoring

philippkahr · 2026-01-21T12:09:35Z

@elastic/integrations-triaging can one of you check this PR and approve? You are still mentioned as pending review.

teresaromero · 2026-01-23T08:56:01Z

Looking into the failing tests build/test-results/ece-system-1768997174567033953.xml i see the fields used in the system test configuration are undefined.
You should add the fields to the base-fields.yml or a separate yml file to declare them so the test can assert them. @mrodm i am not sure what is the best case here, to add them to base-fields or create a separate file for them

- name: input.type type: keyword description: Input type - name: log.offset type: long description: Log offset - name: log.file.device_id type: keyword description: Device Id of the log file this event came from. - name: log.file.inode type: keyword description: Inode number of the log file. - name: log.file.fingerprint type: keyword description: Fingerprint of the log file.

You can check this out locally by running the system test, this is what i've done in case it helps:

Run elastic-package stack up -d -v to have a running stack

Position yourself under the package directory /integrations/packages/ece and run elastic-package test system --data-streams adminconsole --report-output file this will run the system tests and create a report under the folder /integrations/build/test-results. There you can find the details of the failing test. You can run also elastic-package test --report-output file and this will run all of the test for the package.

Looking into the failing tests build/test-results/ece-system-1768997174567033953.xml i see the fields used in the system test configuration are undefined.

Tests also are shown as a Buildkite annotation:

You should add the fields to the base-fields.yml or a separate yml file to declare them so the test can assert them. @mrodm i am not sure what is the best case here, to add them to base-fields or create a separate file for them

They can be located at any file. This is more related to how organize the fields in the package, since at the end all of the files are merged.

As this integration is using the filestream input, there are some examples in that input package that you could follow to add the missing field definitions:

integrations/packages/filestream/fields/beats.yml

Lines 1 to 3 in bab3bcf

- name: input.type

description: Type of Filebeat input.

type: keyword

integrations/packages/filestream/fields/filestream.yml

Lines 1 to 13 in bab3bcf

- name: log.file.inode

type: keyword

description: |

inode of the ingested file.

- name: log.file.device_id

type: keyword

description: |

device ID from the device where the file is.

- name: log.file.fingerprint

type: keyword

index: false

description: |

The fingerprint of the file when using the fingerprint file identity.

integrations/packages/filestream/fields/ecs.yml

Lines 5 to 6 in bab3bcf

- name: log.offset

description: Current log offset

Should it be added other fields like log.file.path or log.level ?

unmapped fields

~Why do I have to map unmapped fields such as log.file.path explicitly?

If your integration only supports Elastic Stack 8.13.0 and above: You can rely on the ecs@mappings component template installed by Fleet. This makes explicitly declaring ECS fields unnecessary; the ecs@mappings component template in Elasticsearch will automatically detect and configure them. However, should ECS fields be explicitly defined, they will overwrite the dynamic mapping provided by the ecs@mappings component template.

https://www.elastic.co/docs/extend/integrations/add-mapping#use-ecs~

I see that is on me, because we decided with filestream to introduce fields that are not part of ECS therefore I need to handle them explicitly. Very annoying, why does the elastic-package create ... not automatically create all of that boilerplate for me :(

I do wonder though why the elastic-package test doesn't cry about it?

unmapped fields

~Why do I have to map unmapped fields such as log.file.path explicitly?

I guess as they are port of ECS is not required to define those. So there is no need to add them 👍

mrodm · 2026-01-23T09:17:44Z

Looking into the failing tests build/test-results/ece-system-1768997174567033953.xml i see the fields used in the system test configuration are undefined.

Tests also are shown as a Buildkite annotation:

You should add the fields to the base-fields.yml or a separate yml file to declare them so the test can assert them. @mrodm i am not sure what is the best case here, to add them to base-fields or create a separate file for them

They can be located at any file. This is more related to how organize the fields in the package, since at the end all of the files are merged.

As this integration is using the filestream input, there are some examples in that input package that you could follow to add the missing field definitions:

integrations/packages/filestream/fields/beats.yml

Lines 1 to 3 in bab3bcf

- name: input.type

description: Type of Filebeat input.

type: keyword

integrations/packages/filestream/fields/filestream.yml

Lines 1 to 13 in bab3bcf

- name: log.file.inode

type: keyword

description: |

inode of the ingested file.

- name: log.file.device_id

type: keyword

description: |

device ID from the device where the file is.

- name: log.file.fingerprint

type: keyword

index: false

description: |

The fingerprint of the file when using the fingerprint file identity.

integrations/packages/filestream/fields/ecs.yml

Lines 5 to 6 in bab3bcf

- name: log.offset

description: Current log offset

Should it be added other fields like log.file.path or log.level ?

elastic-vault-github-plugin-prod · 2026-01-23T13:20:41Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

elasticmachine · 2026-01-23T13:20:44Z

💚 Build Succeeded

Buildkite Build
Commit: 4d5d6ea

History

💔 Build #36836 failed 378e69e
💔 Build #36834 failed 6136dcd
💔 Build #36729 failed d0e7273
💔 Build #36714 failed e73ccde
💔 Build #36713 failed 9ac789b

mrodm

LGTM

mrodm · 2026-01-26T10:05:30Z

unmapped fields

~Why do I have to map unmapped fields such as log.file.path explicitly?

I guess as they are port of ECS is not required to define those. So there is no need to add them 👍

elastic-vault-github-plugin-prod · 2026-01-26T12:30:36Z

Package ece - 0.0.1 containing this change is available at https://epr.elastic.co/package/ece/0.0.1/

* Initial draft * unnecessary image * Fixed a few more things around parsing the url. * Big rewrite and recomposition as suggested * additional and more fixes * Add the docs * fixed typo and bump to etnerprise * fixed the public ip adresses to ones that work with the geoip procesosr * added release beta * Added elastic/stack-monitoring to the codeowners * added the owners * this should finally fix everything * Update manifest.yml * added more description * fixing the missing fields * bumped the version too low * added log.file.fingerprint * fix? * now finally!

Initial draft

855b5a4

philippkahr requested a review from a team as a code owner May 9, 2025 13:02

philippkahr added the enhancement New feature or request label May 9, 2025

unnecessary image

9a674be

andrewkroh added the New Integration Issue or pull request for creating a new integration package. label May 9, 2025

Fixed a few more things around parsing the url.

2b3e5fb

andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Jul 1, 2025

botelastic Bot added the Stalled label Jul 31, 2025

P1llus reviewed Aug 13, 2025

View reviewed changes

botelastic Bot removed the Stalled label Aug 13, 2025

bhapas reviewed Aug 14, 2025

View reviewed changes

Comment thread packages/ece_adminconsole/data_stream/log/elasticsearch/ingest_pipeline/api.yml Outdated

philippkahr added 3 commits August 18, 2025 12:46

Big rewrite and recomposition as suggested

c16fd84

additional and more fixes

2919919

Add the docs

4910fcb

bhapas requested review from a team and consulthys September 17, 2025 07:29

consulthys requested changes Sep 26, 2025

View reviewed changes

fixed typo and bump to etnerprise

d627fa6

botelastic Bot added the Stalled label Jan 14, 2026

consulthys reviewed Jan 19, 2026

View reviewed changes

Comment thread packages/ece/manifest.yml

botelastic Bot removed the Stalled label Jan 19, 2026

philippkahr added 2 commits January 19, 2026 13:13

added release beta

25101c6

Merge branch 'main' into ece-adminconsole-logs

d87149f

consulthys approved these changes Jan 19, 2026

View reviewed changes

philippkahr added 4 commits January 21, 2026 11:00

Added elastic/stack-monitoring to the codeowners

8088ac8

Merge branch 'main' into ece-adminconsole-logs

9ac789b

added the owners

e73ccde

this should finally fix everything

d0e7273

teresaromero reviewed Jan 23, 2026

View reviewed changes

mrodm reviewed Jan 23, 2026

View reviewed changes

philippkahr added 7 commits January 23, 2026 12:47

Update manifest.yml

75da34b

added more description

877adf8

fixing the missing fields

69930bb

bumped the version too low

4395533

added log.file.fingerprint

6136dcd

fix?

378e69e

now finally!

4d5d6ea

mrodm approved these changes Jan 26, 2026

View reviewed changes

philippkahr merged commit 5da24e3 into elastic:main Jan 26, 2026
9 checks passed

philippkahr deleted the ece-adminconsole-logs branch January 26, 2026 12:17

andrewkroh added the Integration:ece Elastic Cloud Enterprise label Jan 26, 2026

consulthys mentioned this pull request Jan 28, 2026

[Elastic Cloud Enterprise] Added dashboard! #17057

Merged

5 tasks

	- name: input.type
	description: Type of Filebeat input.
	type: keyword

	- name: log.file.inode
	type: keyword
	description: \|
	inode of the ingested file.
	- name: log.file.device_id
	type: keyword
	description: \|
	device ID from the device where the file is.
	- name: log.file.fingerprint
	type: keyword
	index: false
	description: \|
	The fingerprint of the file when using the fingerprint file identity.

Uh oh!

Conversation

philippkahr commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Related issues

daniela-elastic commented May 28, 2025

jamiehynds commented May 28, 2025

consulthys commented May 30, 2025

jamiehynds commented May 30, 2025

consulthys commented Jun 4, 2025

philippkahr commented Jun 5, 2025

philippkahr commented Jul 1, 2025

consulthys commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

botelastic Bot commented Jul 31, 2025

P1llus left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

P1llus Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

philippkahr commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

philippkahr commented Aug 18, 2025

bhapas commented Sep 8, 2025

consulthys left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

botelastic Bot commented Jan 14, 2026

Uh oh!

consulthys left a comment

Choose a reason for hiding this comment

philippkahr commented Jan 21, 2026

teresaromero Jan 23, 2026

Choose a reason for hiding this comment

mrodm Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

philippkahr Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

mrodm Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

mrodm Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jan 23, 2026

🚀 Benchmarks report

elasticmachine commented Jan 23, 2026

💚 Build Succeeded

History

mrodm left a comment

Choose a reason for hiding this comment

mrodm Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jan 26, 2026

Labels

10 participants

philippkahr commented May 9, 2025 •

edited

Loading

consulthys commented Jul 1, 2025 •

edited

Loading

P1llus left a comment •

edited

Loading

philippkahr commented Aug 18, 2025 •

edited

Loading

mrodm Jan 23, 2026 •

edited

Loading

philippkahr Jan 23, 2026 •

edited

Loading

mrodm Jan 23, 2026 •

edited

Loading