Skip to content

ZeroNetworks V2.0.0 - #8934

Merged
efd6 merged 19 commits into
elastic:mainfrom
dicolanl:zeronetworks-v2.0.0
May 7, 2024
Merged

ZeroNetworks V2.0.0#8934
efd6 merged 19 commits into
elastic:mainfrom
dicolanl:zeronetworks-v2.0.0

Conversation

@dicolanl

Copy link
Copy Markdown
Contributor
  • Enhancement

-Added a dashboard
-Updated AuditTypes

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • [ X] I have verified that all data streams collect metrics or logs.
  • [ X] I have added an entry to my package's changelog.yml file.
  • [ X] I have verified that Kibana version constraints are current according to guidelines.
@dicolanl
dicolanl requested a review from a team as a code owner January 19, 2024 22:04
@efd6

efd6 commented Jan 21, 2024

Copy link
Copy Markdown
Contributor

Is there an issue for this?

@dicolanl

Copy link
Copy Markdown
Contributor Author

@efd6 no. Just an enhancement.

Comment thread packages/zeronetworks/changelog.yml Outdated
Comment thread packages/zeronetworks/manifest.yml Outdated
@elasticmachine

Copy link
Copy Markdown

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@jamiehynds jamiehynds added the Integration:zeronetworks Zero Networks (Partner supported) label Jan 22, 2024

@efd6 efd6 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you provide a link in the PR description that we can use in the commit message detailing the source of information justifying the change.

Comment thread packages/zeronetworks/manifest.yml Outdated
@efd6

efd6 commented Jan 22, 2024

Copy link
Copy Markdown
Contributor

/test

@efd6

efd6 commented Jan 22, 2024

Copy link
Copy Markdown
Contributor
2024/01/22 22:19:08  INFO Skipped errors: found 1 validation error:
   1. conditions.kibana.version must be ^8.10.0 or greater to include saved object tags file: kibana/tags.yml (SVR00005)
Error: checking package failed: linting package failed: found 3 validation errors:
   1. file "/opt/buildkite-agent/builds/bk-agent-prod-gcp-1705961757859513788/elastic/integrations/packages/zeronetworks/changelog.yml" is invalid: field 0.changes.0: link is required
   2. file "/opt/buildkite-agent/builds/bk-agent-prod-gcp-1705961757859513788/elastic/integrations/packages/zeronetworks/changelog.yml" is invalid: field 0.changes.0: Additional property ink is not allowed
   3. file "/opt/buildkite-agent/builds/bk-agent-prod-gcp-1705961757859513788/elastic/integrations/packages/zeronetworks/kibana/dashboard/zeronetworks-eae973d0-b70d-11ee-9469-51dfba0e91e6.json" is invalid: expected filter in dashboard: no filter found (SVR00002)

You can resolve this by adding the SVR00002.

diff --git a/packages/zeronetworks/validation.yml b/packages/zeronetworks/validation.yml
index a96151416..c8f642bce 100644
--- a/packages/zeronetworks/validation.yml
+++ b/packages/zeronetworks/validation.yml
@@ -1,3 +1,4 @@
 errors:
   exclude_checks:
+    - SVR00002
     - SVR00005 # Kibana version for saved tags.
Comment thread packages/zeronetworks/changelog.yml Outdated
dicolanl and others added 3 commits January 23, 2024 09:55
Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>
Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>
@dicolanl

Copy link
Copy Markdown
Contributor Author
2024/01/22 22:19:08  INFO Skipped errors: found 1 validation error:
   1. conditions.kibana.version must be ^8.10.0 or greater to include saved object tags file: kibana/tags.yml (SVR00005)
Error: checking package failed: linting package failed: found 3 validation errors:
   1. file "/opt/buildkite-agent/builds/bk-agent-prod-gcp-1705961757859513788/elastic/integrations/packages/zeronetworks/changelog.yml" is invalid: field 0.changes.0: link is required
   2. file "/opt/buildkite-agent/builds/bk-agent-prod-gcp-1705961757859513788/elastic/integrations/packages/zeronetworks/changelog.yml" is invalid: field 0.changes.0: Additional property ink is not allowed
   3. file "/opt/buildkite-agent/builds/bk-agent-prod-gcp-1705961757859513788/elastic/integrations/packages/zeronetworks/kibana/dashboard/zeronetworks-eae973d0-b70d-11ee-9469-51dfba0e91e6.json" is invalid: expected filter in dashboard: no filter found (SVR00002)

You can resolve this by adding the SVR00002.

diff --git a/packages/zeronetworks/validation.yml b/packages/zeronetworks/validation.yml
index a96151416..c8f642bce 100644
--- a/packages/zeronetworks/validation.yml
+++ b/packages/zeronetworks/validation.yml
@@ -1,3 +1,4 @@
 errors:
   exclude_checks:
+    - SVR00002
     - SVR00005 # Kibana version for saved tags.

Done

@dicolanl

Copy link
Copy Markdown
Contributor Author

Can you provide a link in the PR description that we can use in the commit message detailing the source of information justifying the change.

No sir. We don't have public docs as a startup.

I am confused a bit too. I assume you are referring to the changes in audit types? not the dashboard add.

For dashboard we just want to make the integration better for our customers.

for auditypes, we are growing company adding features based on customer feedback. This will continue to evolve over time.

@efd6

efd6 commented Jan 24, 2024

Copy link
Copy Markdown
Contributor

for auditypes, we are growing company adding features based on customer feedback. This will continue to evolve over time.

This is the relevant answer. Thanks. I did not know that you were associated with ZeroNetworks. When documentation is available, it would be good to link. Making docs a priority is beneficial.

@efd6

efd6 commented Jan 24, 2024

Copy link
Copy Markdown
Contributor

/test

@jamiehynds

Copy link
Copy Markdown

Thanks for continuing to enhance the integration @dicolanl. Do you mind if we adjust the owner.type field of the integration to highlight that it was developed be a Partner? It's currently listed as an integration developed by a community member/user.

@dicolanl

Copy link
Copy Markdown
Contributor Author

@jamiehynds sure. Tell me what you like to put.
I'm happy to add our support contact too so customers or elastic can contact us regarding issues for the integration

When I built it. I couldn't find any samples with external companies listed.

@jamiehynds

jamiehynds commented Jan 24, 2024

Copy link
Copy Markdown

@dicolanl sounds good. For now, the only field that needs to be adjusted is the owner.type here. If you can set that to 'Partner' for now. We don't currently have fields there for contact details, but we plan to add contact details pretty soon.

@dicolanl
dicolanl requested a review from a team as a code owner January 24, 2024 15:37
@dicolanl

Copy link
Copy Markdown
Contributor Author

@jamiehynds updated owner.
email coming your way.

@efd6

efd6 commented Jan 25, 2024

Copy link
Copy Markdown
Contributor

/test

@efd6

efd6 commented Apr 16, 2024

Copy link
Copy Markdown
Contributor

/test

Comment thread packages/zeronetworks/_dev/deploy/docker/files/config.yml
Comment thread packages/zeronetworks/changelog.yml
Comment thread packages/zeronetworks/data_stream/audit/_dev/test/system/test-default-config.yml Outdated
@efd6

efd6 commented Apr 16, 2024

Copy link
Copy Markdown
Contributor

/test

@efd6 efd6 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The version in the manifest will need to be updated to match the changelog.

Error: checking package failed: linting package failed: found 1 validation error:
   1. current manifest version doesn't have changelog entry
@dicolanl

Copy link
Copy Markdown
Contributor Author

fixed maifest

@efd6

efd6 commented Apr 17, 2024

Copy link
Copy Markdown
Contributor

/test

@efd6 efd6 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after nit addressed.

Comment thread packages/zeronetworks/data_stream/audit/_dev/test/system/test-default-config.yml Outdated
@dicolanl

Copy link
Copy Markdown
Contributor Author

@efd6 updated last week.

@efd6

efd6 commented Apr 29, 2024

Copy link
Copy Markdown
Contributor

/test

@efd6

efd6 commented Apr 29, 2024

Copy link
Copy Markdown
Contributor

The system tests are not getting any events.

@dicolanl

Copy link
Copy Markdown
Contributor Author

Why would it not work in the cloud? I ran system test on my laptop was fine. Nothing changed since I shared that output with events.

@efd6

efd6 commented Apr 30, 2024

Copy link
Copy Markdown
Contributor

I will try to take a look later in the week.

@efd6

efd6 commented May 1, 2024

Copy link
Copy Markdown
Contributor

It looks like maybe pagination is not working correctly. When I run this locally, I only get 5 events coming through in the system tests. This count corresponds with the number of events in each of the rules you have defined in the system test deployment configs.

@dicolanl

dicolanl commented May 1, 2024

Copy link
Copy Markdown
Contributor Author

@efd6
something weird is going on. this was all working, now its not.
if i run system test locally, grabbing the docker for stream

2024-05-01 14:18:42 {"level":"debug","ts":"2024-05-01T18:18:42.966Z","caller":"httpserver/httpserver.go:130","msg":"Rule #0 matched: request #1 => Request path: GET /audit?_limit=5&from=1713982722959&to=1714587522959, Request Headers: 'User-Agent: [Elastic-Filebeat/8.12.2 (linux; arm64; 0b71acf2d6b4cb6617bff980ed6caf0477905efa; 2024-02-15 13:39:15 +0000 UTC)]' 'Accept: [application/json]' 'Authorization: [eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJtOjkwZDE0OGRhYjQ5MjczOGJlZDFlOTFmOWM3NzY0YmY3Y2M2ODljNGYiLCJuYW1lIjoiZWxhc3RpYyIsImVpZCI6ImNkMzg0NDcwLTkwZGYtNDYxOC1iMjE5LTEwMDNmNzY2OWNlMiIsInNjb3BlIjo1LCJlX25hbWUiOiJQb3dlcnNoZWxsVGVzdGluZyIsInYiOjEsImlhdCI6MTcxMzIxODEyNywiZXhwIjoxNzc2MjkwMTI2LCJhdWQiOiJwb3J0YWwuemVyb25ldHdvcmtzLmNvbSIsImlzcyI6Inplcm9uZXR3b3Jrcy5jb20vYXBpL3YxL2FjY2Vzcy10b2tlbiJ9.HlzU_uspUhBxy7pA7vEwUpMYz2GF-IWccBMM4TJxuvL_6bTNFi-cISz4FdDtKFw-lbwZ_wDplBJyM3x0XMe3H2__UF0ZzrDckXSRmSHbo1tWhpjHkeWYp92QnIyTDocPMigOlFagE2dJg6CfEsN_srb0yuEVmGrJr-Ihh5bTs3wh_vHnuXsWJDw-_obeCrt6bXIlM_2keNPeZtMi_vHIDsRMd5OXo2pFB5RsSva58yH6cgELVXw41GZMsP_HZIwAf4Cs0G7V1JTwpXaENy1s4hcDE7BRW1uPgHlTtGP1eZSWjblFz9-ljqMJFBs7sfbO_ZM5AXQj17kwNR1ihgW8hw]' 'Accept-Encoding: [gzip]' 'Connection: [close]' , Request Body: ","address":":8080"}
2024-05-01 14:18:44 {"level":"debug","ts":"2024-05-01T18:18:44.958Z","caller":"httpserver/httpserver.go:212","msg":"request did not match with any rule: Request path: GET /audit?_cursor=1711960236440&_limit=5&from=1.71196023644e%2B12&to=1714587524955, Request Headers: 'User-Agent: [Elastic-Filebeat/8.12.2 (linux; arm64; 0b71acf2d6b4cb6617bff980ed6caf0477905efa; 2024-02-15 13:39:15 +0000 UTC)]' 'Accept: [application/json]' 'Authorization: [eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJtOjkwZDE0OGRhYjQ5MjczOGJlZDFlOTFmOWM3NzY0YmY3Y2M2ODljNGYiLCJuYW1lIjoiZWxhc3RpYyIsImVpZCI6ImNkMzg0NDcwLTkwZGYtNDYxOC1iMjE5LTEwMDNmNzY2OWNlMiIsInNjb3BlIjo1LCJlX25hbWUiOiJQb3dlcnNoZWxsVGVzdGluZyIsInYiOjEsImlhdCI6MTcxMzIxODEyNywiZXhwIjoxNzc2MjkwMTI2LCJhdWQiOiJwb3J0YWwuemVyb25ldHdvcmtzLmNvbSIsImlzcyI6Inplcm9uZXR3b3Jrcy5jb20vYXBpL3YxL2FjY2Vzcy10b2tlbiJ9.HlzU_uspUhBxy7pA7vEwUpMYz2GF-IWccBMM4TJxuvL_6bTNFi-cISz4FdDtKFw-lbwZ_wDplBJyM3x0XMe3H2__UF0ZzrDckXSRmSHbo1tWhpjHkeWYp92QnIyTDocPMigOlFagE2dJg6CfEsN_srb0yuEVmGrJr-Ihh5bTs3wh_vHnuXsWJDw-_obeCrt6bXIlM_2keNPeZtMi_vHIDsRMd5OXo2pFB5RsSva58yH6cgELVXw41GZMsP_HZIwAf4Cs0G7V1JTwpXaENy1s4hcDE7BRW1uPgHlTtGP1eZSWjblFz9-ljqMJFBs7sfbO_ZM5AXQj17kwNR1ihgW8hw]' 'Accept-Encoding: [gzip]' 'Connection: [close]' , Request Body: ","address":":8080"}

I noticed that the second call is changing the "from" param to 1.7...... why is not sending epochms like the first call???

So I created new elastic stack in docker, went to the web and connected the integration to the real api.

2024-05-01 15:48:15 {"log.level":"info","@timestamp":"2024-05-01T19:48:15.345Z","message":"Process another repeated request.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"service.name":"filebeat","id":"httpjson-zeronetworks.audit-86034640-eae1-4e5a-acb6-7cee86df47e5","input_source":"https://portal.zeronetworks.com/api/v1/audit","log.origin":{"file.line":156,"file.name":"httpjson/input.go","function":"github.com/elastic/beats/v7/x-pack/filebeat/input/httpjson.run.func1"},"ecs.version":"1.6.0","log.logger":"input.httpjson-cursor","input_url":"https://portal.zeronetworks.com/api/v1/audit","ecs.version":"1.6.0"}
2024-05-01 15:48:15 {"log.level":"error","@timestamp":"2024-05-01T19:48:15.795Z","message":"error processing response: server responded with status code 400: {\"error\":\"bad request\",\"message\":\"'cursor' must be bigger than 'from', and smaller or equal to 'to'\"}","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.origin":{"file.line":769,"file.name":"httpjson/request.go","function":"github.com/elastic/beats/v7/x-pack/filebeat/input/httpjson.(*publisher).handleError"},"service.name":"filebeat","input_source":"https://portal.zeronetworks.com/api/v1/audit","input_url":"https://portal.zeronetworks.com/api/v1/audit","ecs.version":"1.6.0","log.logger":"input.httpjson-cursor","id":"httpjson-zeronetworks.audit-86034640-eae1-4e5a-acb6-7cee86df47e5","ecs.version":"1.6.0"}
2024-05-01 15:48:15 {"log.level":"info","@timestamp":"2024-05-01T19:48:15.795Z","message":"request finished: 60 events published","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"httpjson-default","type":"httpjson"},"log":{"source":"httpjson-default"},"log.logger":"input.httpjson-cursor","id":"httpjson-zeronetworks.audit-86034640-eae1-4e5a-acb6-7cee86df47e5","ecs.version":"1.6.0","log.origin":{"file.line":207,"file.name":"httpjson/request.go","function":"github.com/elastic/beats/v7/x-pack/filebeat/input/httpjson.(*requester).doRequest"},"service.name":"filebeat","input_source":"https://portal.zeronetworks.com/api/v1/audit","input_url":"https://portal.zeronetworks.com/api/v1/audit","ecs.version":"1.6.0"}

now it seems, which i didnt change anything in the httpjson.yml.hbs file that its setting cursor incorrrectly, so maybe something in the agent changed and im handling cursor wrong?

@efd6

efd6 commented May 2, 2024

Copy link
Copy Markdown
Contributor

The agent won't have changed. I can take a look again next week.

@dicolanl

dicolanl commented May 3, 2024

Copy link
Copy Markdown
Contributor Author

I think I found the problem. I'll update soon.

It was saving the next cursor as a string. Not int.

System test works now
--- Test results for package: zeronetworks - START ---
╭──────────────┬─────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE      │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├──────────────┼─────────────┼───────────┼───────────┼────────┼───────────────┤
│ zeronetworks │ audit       │ system    │ default   │ PASS   │ 34.809901167s │
╰──────────────┴─────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: zeronetworks - END   ---
@dicolanl

dicolanl commented May 6, 2024

Copy link
Copy Markdown
Contributor Author

@efd6 found a bug with paging. once I fixed that found some other issues with fields missing. so fixed everything.

2024/05/06 16:22:47  INFO New version is available - v0.100.0. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.100.0
Run test suite for the package
Run static tests for the package
--- Test results for package: zeronetworks - START ---
╭──────────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE      │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├──────────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ zeronetworks │ audit       │ static    │ Verify sample_event.json │ PASS   │ 126.011792ms │
╰──────────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: zeronetworks - END   ---
Done
Run system tests for the package
2024/05/06 16:23:32  INFO Write container logs to file: /Volumes/Repos/integrations/build/container-logs/zeronetworks-1715027012758638000.log
--- Test results for package: zeronetworks - START ---
╭──────────────┬─────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE      │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├──────────────┼─────────────┼───────────┼───────────┼────────┼───────────────┤
│ zeronetworks │ audit       │ system    │ default   │ PASS   │ 34.203242833s │
╰──────────────┴─────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: zeronetworks - END   ---
Done
Run asset tests for the package
--- Test results for package: zeronetworks - START ---
╭──────────────┬─────────────┬───────────┬───────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE      │ DATA STREAM │ TEST TYPE │ TEST NAME                                                             │ RESULT │ TIME ELAPSED │
├──────────────┼─────────────┼───────────┼───────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ zeronetworks │             │ asset     │ dashboard zeronetworks-eae973d0-b70d-11ee-9469-51dfba0e91e6 is loaded │ PASS   │        750ns │
│ zeronetworks │ audit       │ asset     │ index_template logs-zeronetworks.audit is loaded                      │ PASS   │        250ns │
│ zeronetworks │ audit       │ asset     │ ingest_pipeline logs-zeronetworks.audit-1.13.0 is loaded              │ PASS   │        208ns │
╰──────────────┴─────────────┴───────────┴───────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: zeronetworks - END   ---
Done
Run pipeline tests for the package
--- Test results for package: zeronetworks - START ---
╭──────────────┬─────────────┬───────────┬────────────────┬────────┬──────────────╮
│ PACKAGE      │ DATA STREAM │ TEST TYPE │ TEST NAME      │ RESULT │ TIME ELAPSED │
├──────────────┼─────────────┼───────────┼────────────────┼────────┼──────────────┤
│ zeronetworks │ audit       │ pipeline  │ test-audit.log │ PASS   │  25.252083ms │
╰──────────────┴─────────────┴───────────┴────────────────┴────────┴──────────────╯
--- Test results for package: zeronetworks - END   ---
Done
@efd6

efd6 commented May 7, 2024

Copy link
Copy Markdown
Contributor

/test

@elasticmachine

Copy link
Copy Markdown

💚 Build Succeeded

History

@efd6 efd6 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

@efd6
efd6 merged commit 3fc0568 into elastic:main May 7, 2024
@elasticmachine

Copy link
Copy Markdown

Package zeronetworks - 1.13.0 containing this change is available at https://epr.elastic.co/search?package=zeronetworks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Integration:zeronetworks Zero Networks (Partner supported) Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

6 participants