Skip to content

[GitLab] Added application, auth, and audit datastreams - #10644

Merged
tehbooom merged 19 commits into
elastic:mainfrom
tehbooom:gitlab-auth-app
Aug 6, 2024
Merged

[GitLab] Added application, auth, and audit datastreams#10644
tehbooom merged 19 commits into
elastic:mainfrom
tehbooom:gitlab-auth-app

Conversation

@tehbooom

@tehbooom tehbooom commented Jul 29, 2024

Copy link
Copy Markdown
Contributor

Proposed commit message

Added application, auth, and audit datastreams

Added dashboards for application and auth datastreams

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

gitlab-application-dashboard
gitlab-auth-dashboard
@tehbooom tehbooom added enhancement New feature or request Integration:gitlab GitLab labels Jul 29, 2024
@tehbooom
tehbooom requested a review from a team as a code owner July 29, 2024 16:42
@jamiehynds jamiehynds added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Jul 29, 2024
@elasticmachine

elasticmachine commented Jul 29, 2024

Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)
@efd6

efd6 commented Aug 2, 2024

Copy link
Copy Markdown
Contributor

@tehbooom What version have you tested this on. This is failing with Error: failed to setup system runner: can't install the package: there was an apply error: installation failed: can't install the package: could not zip-install package; API status code = 400; response body = {"statusCode":400,"error":"Bad Request","message":"Encountered 1 errors creating saved objects: [{\"type\":\"dashboard\",\"id\":\"gitlab-6b62c186-77b6-49f2-a820-ca377575bb96\",\"error\":{\"isBoom\":true,\"isServer\":false,\"data\":null,\"output\":{\"statusCode\":400,\"payload\":{\"message\":\"[attributes.controlGroupInput.showApplySelections]: definition for this key is missing: Bad Request\",\"statusCode\":400,\"error\":\"Bad Request\"},\"headers\":{}},\"type\":\"unknown\"}}]"}. I see this locally as well.
@kcreddy

kcreddy commented Aug 2, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

I got a similar error when I used 8.14.3 for creating dashboards, but the CI runs with minimum kibana.version i.e., 8.13.0. I had to remove this key attributes.controlGroupInput.showApplySelections from the dashboards indicated by the error message.
@tehbooom

tehbooom commented Aug 2, 2024

Copy link
Copy Markdown
Contributor Author

@efd6 The elasticstack I tested on is 8.14.1. Im trying what @kcreddy suggested to see if it that passes.
@kcreddy

kcreddy commented Aug 2, 2024

Copy link
Copy Markdown
Contributor

@tehbooom this time the error message is:
Error: can't install the package: could not zip-install package; API status code = 422; response body = {"statusCode":422,"error":"Unprocessable Entity","message":"Document \"gitlab-6b62c186-77b6-49f2-a820-ca377575bb96\" belongs to a more recent version of Kibana [10.2.0] when the last known version is [8.9.0]."}

You will need to change from "typeMigrationVersion": "10.2.0" to "typeMigrationVersion": "8.9.0" inside dashboards. I totally forgot about that one earlier.
@tehbooom

tehbooom commented Aug 2, 2024

Copy link
Copy Markdown
Contributor Author

@kcreddy ahh I reverted back to 10.2.0 thinking it wasn't necessary.
@elasticmachine

elasticmachine commented Aug 2, 2024

Copy link
Copy Markdown

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport
kcreddy
kcreddy reviewed Aug 5, 2024
View reviewed changes
Comment thread packages/gitlab/_dev/build/docs/README.md Outdated
Comment thread packages/gitlab/_dev/build/docs/README.md Outdated
Comment thread packages/gitlab/_dev/build/docs/README.md Outdated
Comment thread packages/gitlab/data_stream/application/elasticsearch/ingest_pipeline/default.yml Outdated
Comment thread packages/gitlab/data_stream/application/fields/fields.yml Outdated
Comment thread packages/gitlab/data_stream/audit/elasticsearch/ingest_pipeline/default.yml Outdated
Comment thread packages/gitlab/data_stream/auth/elasticsearch/ingest_pipeline/default.yml Outdated
Comment thread packages/gitlab/data_stream/auth/_dev/test/system/test-default-config.yml
Comment thread packages/gitlab/data_stream/auth/elasticsearch/ingest_pipeline/default.yml
Comment thread packages/gitlab/data_stream/auth/_dev/test/pipeline/test-auth.log-expected.json Outdated
kcreddy
kcreddy approved these changes Aug 6, 2024
View reviewed changes

@kcreddy kcreddy left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍🏼 Minor suggestion
Comment thread packages/gitlab/data_stream/application/elasticsearch/ingest_pipeline/default.yml Outdated
Comment on lines +234 to +239
- remove:
field: event.original
tag: remove_original_event
if: ctx?.tags == null || !(ctx.tags.contains("preserve_original_event"))
ignore_failure: true
ignore_missing: true

@kcreddy kcreddy Aug 6, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this processor as it is covered in fleet final pipeline.
@elasticmachine

elasticmachine commented Aug 6, 2024

Copy link
Copy Markdown

💚 Build Succeeded

History
@elastic-sonarqube

elastic-sonarqube Bot commented Aug 6, 2024

Copy link
Copy Markdown

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
96.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube
@tehbooom
tehbooom merged commit d4411a5 into elastic:main Aug 6, 2024
@elasticmachine

elasticmachine commented Aug 6, 2024

Copy link
Copy Markdown

Package gitlab - 0.3.0 containing this change is available at https://epr.elastic.co/search?package=gitlab
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025
@tehbooom
[GitLab] Added application, auth, and audit datastreams (elastic#10644)
858fee9 
* Added application, audit, and auth datastreams

* added auth dataset and more application example logs

* added more example logs for application

* added application and auth dashboards with screenshots

* updated pr number

* bump version

* updated migration version

* revert back to original and removed showApplySelections

* change typeMigrationVersion to 8.9.0

* docs:updated path to URLs and spelling

* chore:removed ecs mappings

* removed quotes from fields

* fix:added client ip with geo

* test:added assert.hit_count

* fix:updated url.path and url.query

* fix:syntax and add geo ip processing

* fix:add geoip processing, related.user, related.ip and fix syntax

* fix:remove processor handled in fleet
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025
@tehbooom
[GitLab] Added application, auth, and audit datastreams (elastic#10644)
1660530 
* Added application, audit, and auth datastreams

* added auth dataset and more application example logs

* added more example logs for application

* added application and auth dashboards with screenshots

* updated pr number

* bump version

* updated migration version

* revert back to original and removed showApplySelections

* change typeMigrationVersion to 8.9.0

* docs:updated path to URLs and spelling

* chore:removed ecs mappings

* removed quotes from fields

* fix:added client ip with geo

* test:added assert.hit_count

* fix:updated url.path and url.query

* fix:syntax and add geo ip processing

* fix:add geoip processing, related.user, related.ip and fix syntax

* fix:remove processor handled in fleet
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Integration:gitlab GitLab Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

5 participants

@tehbooom @elasticmachine @efd6 @kcreddy @jamiehynds