[Cisco Duo] New data stream for Activity logs by chemamartinez · Pull Request #11394 · elastic/integrations

chemamartinez · 2024-10-10T18:04:56Z

Proposed commit message

Added new data stream activity to collect Activity logs from Cisco Duo.

The CEL program follows the API specifications set at:

It also adds a feature request for the auth data stream, including geo enrichment for the IP into the access_device field.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Added pipeline and system tests for the new data stream:

Run asset tests for the package
--- Test results for package: cisco_duo - START ---
╭───────────┬────────────────────┬───────────┬────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE   │ DATA STREAM        │ TEST TYPE │ TEST NAME                                                          │ RESULT │ TIME ELAPSED │
├───────────┼────────────────────┼───────────┼────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ cisco_duo │                    │ asset     │ dashboard cisco_duo-0607d4a3-5322-41c1-b8fa-f0d29bcc2757 is loaded │ PASS   │      1.584µs │
│ cisco_duo │                    │ asset     │ dashboard cisco_duo-118061c3-3130-4c71-8899-157bab1ad447 is loaded │ PASS   │        292ns │
│ cisco_duo │                    │ asset     │ dashboard cisco_duo-5a0b80af-49ad-42ee-89b7-c89faa927826 is loaded │ PASS   │        416ns │
│ cisco_duo │                    │ asset     │ dashboard cisco_duo-7a135061-78a3-45d9-951b-4b9b665fa729 is loaded │ PASS   │        208ns │
│ cisco_duo │                    │ asset     │ dashboard cisco_duo-b386f94c-0856-4508-ba08-a525a2f3b70f is loaded │ PASS   │        417ns │
│ cisco_duo │                    │ asset     │ dashboard cisco_duo-c3336a66-68ff-4bcd-95ff-fb388793f721 is loaded │ PASS   │        375ns │
│ cisco_duo │                    │ asset     │ dashboard cisco_duo-e91470e5-2ded-4ff1-8bb5-24e06b949c1d is loaded │ PASS   │        250ns │
│ cisco_duo │                    │ asset     │ map cisco_duo-158c0e80-148c-11ec-9386-31989719f9db is loaded       │ PASS   │        250ns │
│ cisco_duo │                    │ asset     │ search cisco_duo-64869c89-8c44-4644-a84a-9815c0fddba0 is loaded    │ PASS   │        375ns │
│ cisco_duo │ activity           │ asset     │ index_template logs-cisco_duo.activity is loaded                   │ PASS   │        250ns │
│ cisco_duo │ activity           │ asset     │ ingest_pipeline logs-cisco_duo.activity-2.2.0 is loaded            │ PASS   │        292ns │
│ cisco_duo │ admin              │ asset     │ index_template logs-cisco_duo.admin is loaded                      │ PASS   │        166ns │
│ cisco_duo │ admin              │ asset     │ ingest_pipeline logs-cisco_duo.admin-2.2.0 is loaded               │ PASS   │        292ns │
│ cisco_duo │ auth               │ asset     │ index_template logs-cisco_duo.auth is loaded                       │ PASS   │        334ns │
│ cisco_duo │ auth               │ asset     │ ingest_pipeline logs-cisco_duo.auth-2.2.0 is loaded                │ PASS   │        125ns │
│ cisco_duo │ offline_enrollment │ asset     │ index_template logs-cisco_duo.offline_enrollment is loaded         │ PASS   │        209ns │
│ cisco_duo │ offline_enrollment │ asset     │ ingest_pipeline logs-cisco_duo.offline_enrollment-2.2.0 is loaded  │ PASS   │        125ns │
│ cisco_duo │ summary            │ asset     │ index_template logs-cisco_duo.summary is loaded                    │ PASS   │        208ns │
│ cisco_duo │ summary            │ asset     │ ingest_pipeline logs-cisco_duo.summary-2.2.0 is loaded             │ PASS   │        125ns │
│ cisco_duo │ telephony          │ asset     │ index_template logs-cisco_duo.telephony is loaded                  │ PASS   │        209ns │
│ cisco_duo │ telephony          │ asset     │ ingest_pipeline logs-cisco_duo.telephony-2.2.0 is loaded           │ PASS   │        125ns │
│ cisco_duo │ telephony_v2       │ asset     │ index_template logs-cisco_duo.telephony_v2 is loaded               │ PASS   │        375ns │
│ cisco_duo │ telephony_v2       │ asset     │ ingest_pipeline logs-cisco_duo.telephony_v2-2.2.0 is loaded        │ PASS   │        166ns │
│ cisco_duo │ trust_monitor      │ asset     │ index_template logs-cisco_duo.trust_monitor is loaded              │ PASS   │        375ns │
│ cisco_duo │ trust_monitor      │ asset     │ ingest_pipeline logs-cisco_duo.trust_monitor-2.2.0 is loaded       │ PASS   │        125ns │
╰───────────┴────────────────────┴───────────┴────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: cisco_duo - END   ---
Done
Run pipeline tests for the package
--- Test results for package: cisco_duo - START ---
╭───────────┬────────────────────┬───────────┬────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE   │ DATA STREAM        │ TEST TYPE │ TEST NAME                                              │ RESULT │ TIME ELAPSED │
├───────────┼────────────────────┼───────────┼────────────────────────────────────────────────────────┼────────┼──────────────┤
│ cisco_duo │ activity           │ pipeline  │ (ingest pipeline warnings test-activity.log)           │ PASS   │ 196.669625ms │
│ cisco_duo │ activity           │ pipeline  │ test-activity.log                                      │ PASS   │ 134.559833ms │
│ cisco_duo │ admin              │ pipeline  │ (ingest pipeline warnings test-admin.log)              │ PASS   │ 206.523291ms │
│ cisco_duo │ admin              │ pipeline  │ (ingest pipeline warnings test-empty.log)              │ PASS   │ 202.109375ms │
│ cisco_duo │ admin              │ pipeline  │ test-admin.log                                         │ PASS   │    151.045ms │
│ cisco_duo │ admin              │ pipeline  │ test-empty.log                                         │ PASS   │  40.457458ms │
│ cisco_duo │ auth               │ pipeline  │ (ingest pipeline warnings test-auth.log)               │ PASS   │ 218.412834ms │
│ cisco_duo │ auth               │ pipeline  │ (ingest pipeline warnings test-empty.log)              │ PASS   │   194.9385ms │
│ cisco_duo │ auth               │ pipeline  │ test-auth.log                                          │ PASS   │ 345.084917ms │
│ cisco_duo │ auth               │ pipeline  │ test-empty.log                                         │ PASS   │   40.33725ms │
│ cisco_duo │ offline_enrollment │ pipeline  │ (ingest pipeline warnings test-empty.log)              │ PASS   │  198.41075ms │
│ cisco_duo │ offline_enrollment │ pipeline  │ (ingest pipeline warnings test-offline-enrollment.log) │ PASS   │   200.7585ms │
│ cisco_duo │ offline_enrollment │ pipeline  │ test-empty.log                                         │ PASS   │  37.008042ms │
│ cisco_duo │ offline_enrollment │ pipeline  │ test-offline-enrollment.log                            │ PASS   │  44.096708ms │
│ cisco_duo │ summary            │ pipeline  │ (ingest pipeline warnings test-summary.log)            │ PASS   │ 205.246834ms │
│ cisco_duo │ summary            │ pipeline  │ test-summary.log                                       │ PASS   │   41.75075ms │
│ cisco_duo │ telephony          │ pipeline  │ (ingest pipeline warnings test-empty.log)              │ PASS   │ 234.215875ms │
│ cisco_duo │ telephony          │ pipeline  │ (ingest pipeline warnings test-telephony.log)          │ PASS   │ 247.474375ms │
│ cisco_duo │ telephony          │ pipeline  │ test-empty.log                                         │ PASS   │  40.395042ms │
│ cisco_duo │ telephony          │ pipeline  │ test-telephony.log                                     │ PASS   │  38.864542ms │
│ cisco_duo │ telephony_v2       │ pipeline  │ (ingest pipeline warnings test-telephony-v2.log)       │ PASS   │ 210.205625ms │
│ cisco_duo │ telephony_v2       │ pipeline  │ test-telephony-v2.log                                  │ PASS   │   43.08375ms │
│ cisco_duo │ trust_monitor      │ pipeline  │ (ingest pipeline warnings test-trust-monitor.log)      │ PASS   │ 204.730667ms │
│ cisco_duo │ trust_monitor      │ pipeline  │ test-trust-monitor.log                                 │ PASS   │  49.444791ms │
╰───────────┴────────────────────┴───────────┴────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: cisco_duo - END   ---
Done
Run policy tests for the package
--- Test results for package: cisco_duo - START ---
No test results
--- Test results for package: cisco_duo - END   ---
Done
Run static tests for the package
--- Test results for package: cisco_duo - START ---
╭───────────┬────────────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE   │ DATA STREAM        │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├───────────┼────────────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ cisco_duo │ activity           │ static    │ Verify sample_event.json │ PASS   │  52.114292ms │
│ cisco_duo │ admin              │ static    │ Verify sample_event.json │ PASS   │  41.618833ms │
│ cisco_duo │ auth               │ static    │ Verify sample_event.json │ PASS   │  57.805375ms │
│ cisco_duo │ offline_enrollment │ static    │ Verify sample_event.json │ PASS   │  42.232042ms │
│ cisco_duo │ summary            │ static    │ Verify sample_event.json │ PASS   │  38.962625ms │
│ cisco_duo │ telephony          │ static    │ Verify sample_event.json │ PASS   │  38.040167ms │
│ cisco_duo │ telephony_v2       │ static    │ Verify sample_event.json │ PASS   │  37.535666ms │
│ cisco_duo │ trust_monitor      │ static    │ Verify sample_event.json │ PASS   │  42.190791ms │
╰───────────┴────────────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: cisco_duo - END   ---
Done
Run system tests for the package
--- Test results for package: cisco_duo - START ---
╭───────────┬────────────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE   │ DATA STREAM        │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├───────────┼────────────────────┼───────────┼───────────┼────────┼───────────────┤
│ cisco_duo │ activity           │ system    │ default   │ PASS   │ 36.870015625s │
│ cisco_duo │ admin              │ system    │ default   │ PASS   │  33.85713475s │
│ cisco_duo │ auth               │ system    │ default   │ PASS   │ 33.772624833s │
│ cisco_duo │ offline_enrollment │ system    │ default   │ PASS   │ 38.540788792s │
│ cisco_duo │ summary            │ system    │ default   │ PASS   │ 34.455289708s │
│ cisco_duo │ telephony          │ system    │ default   │ PASS   │  33.41083725s │
│ cisco_duo │ telephony_v2       │ system    │ default   │ PASS   │ 39.567072958s │
│ cisco_duo │ trust_monitor      │ system    │ default   │ PASS   │ 39.527218083s │
╰───────────┴────────────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: cisco_duo - END   ---
Done

Related issues

Screenshots

Data stream configuration

Dashboard

elasticmachine · 2024-10-10T18:05:52Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

elastic-vault-github-plugin-prod · 2024-10-10T18:32:50Z

🚀 Benchmarks report

Package `cisco_duo` 👍(2) 💚(1) 💔(5)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`auth`	2564.1	1385.04	-1179.06 (-45.98%)	💔
`offline_enrollment`	25641.03	14925.37	-10715.66 (-41.79%)	💔
`telephony`	43478.26	27027.03	-16451.23 (-37.84%)	💔
`telephony_v2`	15873.02	11235.96	-4637.06 (-29.21%)	💔
`trust_monitor`	9009.01	7299.27	-1709.74 (-18.98%)	💔

To see the full report comment with /test benchmark fullreport

…ivity

efd6

Though note the concern about the string(list) conversion.

efd6

Something about real-world Cisco Duo documents is causing breakage with this code (as it exists in auth). So blocking.

…ivity

efd6

LGTM after nit addressed.

…ivity

elasticmachine · 2024-10-21T10:32:49Z

💚 Build Succeeded

Buildkite Build
Commit: ab1fb89

History

💚 Build #17395 succeeded b496f0f
💚 Build #17351 succeeded 2f89700
💚 Build #17280 succeeded 18213e8
💚 Build #17165 succeeded 3ee5a42
💚 Build #17032 succeeded f49238e

cc @chemamartinez

elastic-sonarqube · 2024-10-21T10:32:54Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
99.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elastic-vault-github-plugin-prod · 2024-10-21T11:06:53Z

Package cisco_duo - 2.2.0 containing this change is available at https://epr.elastic.co/search?package=cisco_duo

Added new data stream activity to collect Activity logs from Cisco Duo. It also adds a feature request for the auth data stream, including geo enrichment for the IP into the access_device field.

chemamartinez added 12 commits October 4, 2024 10:13

Fix typos in manifest

1c4282d

Add trust monitor data stream

3a7e727

Add docs for trust monitor data stream

2a5a686

Add dashboard for trust monitor events

384c306

Update docs

2158fff

Update changelog

eab9c9a

Add geo enrichment to access_device IP info

037678e

Add activity data stream

dbfd40b

Add docs for new activity data stream

b611c86

Bump version to 2.2.0

2f85dae

Add activity dashboard

9a002ef

Fix indentation in pipeline test samples

5272b16

chemamartinez added enhancement New feature or request dashboard Relates to a Kibana dashboard bug, enhancement, or modification. Integration:cisco_duo Cisco Duo Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Oct 10, 2024

chemamartinez self-assigned this Oct 10, 2024

Update changelog

f49238e

chemamartinez marked this pull request as ready for review October 10, 2024 18:05

chemamartinez requested a review from a team as a code owner October 10, 2024 18:05

efd6 reviewed Oct 10, 2024

View reviewed changes

chemamartinez added 3 commits October 15, 2024 17:50

Add requested changes for Trust Monitor

779590d

Merge branch '10961-cisco_duo-trust_monitor' into 10960-cisco_duo-act…

fb80dfd

…ivity

Add requested changes for Activity data stream

3ee5a42

chemamartinez requested a review from efd6 October 15, 2024 16:36

efd6 approved these changes Oct 15, 2024

View reviewed changes

Comment thread packages/cisco_duo/data_stream/trust_monitor/agent/stream/cel.yml.hbs Outdated

efd6 mentioned this pull request Oct 15, 2024

[Cisco Duo] New data stream for Trust Monitor logs #11327

Merged

4 tasks

efd6 requested changes Oct 15, 2024

View reviewed changes

chemamartinez added 7 commits October 16, 2024 18:45

Merge branch 'main' into 10961-cisco_duo-trust_monitor

790d160

Merge branch 'main' into 10961-cisco_duo-trust_monitor

1d03e2e

Set request rate limits for Trust Monitor

f48a2de

Include error response body when non-200

b866058

Merge branch '10961-cisco_duo-trust_monitor' into 10960-cisco_duo-act…

fabf42e

…ivity

Set request rate limits for Activity

7a44f56

Include error response body when non-200

18213e8

chemamartinez requested a review from efd6 October 17, 2024 08:17

chemamartinez added 4 commits October 18, 2024 16:34

Merge branch 'main' into 10961-cisco_duo-trust_monitor

cb02121

Make request limit configurable for Trust Monitor

8450495

Merge branch '10961-cisco_duo-trust_monitor' into 10960-cisco_duo-act…

ec9f6e6

…ivity

Make request limit configurable for Activity

2f89700

efd6 approved these changes Oct 20, 2024

View reviewed changes

Comment thread packages/cisco_duo/data_stream/activity/agent/stream/cel.yml.hbs Outdated

chemamartinez added 5 commits October 21, 2024 10:11

Last requested changes

4af5946

Adapt telephony CEL input to latest changes in other data streams

ddf5358

Merge branch '10961-cisco_duo-trust_monitor' into 10960-cisco_duo-act…

a64e957

…ivity

Last requested changes for Activity CEL code

b496f0f

Merge branch 'main' into 10960-cisco_duo-activity

ab1fb89

chemamartinez merged commit 2ea9938 into elastic:main Oct 21, 2024

chemamartinez deleted the 10960-cisco_duo-activity branch February 6, 2025 10:29

Navigation Menu

Uh oh!

[Cisco Duo] New data stream for Activity logs - #11394

[Cisco Duo] New data stream for Activity logs#11394
chemamartinez merged 32 commits into
elastic:mainfrom
chemamartinez:10960-cisco_duo-activity

chemamartinez commented Oct 10, 2024 •

edited

Loading

elasticmachine commented Oct 10, 2024

elastic-vault-github-plugin-prod Bot commented Oct 10, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 left a comment

Uh oh!

efd6 left a comment

efd6 left a comment

Uh oh!

elasticmachine commented Oct 21, 2024

elastic-sonarqube Bot commented Oct 21, 2024

elastic-vault-github-plugin-prod Bot commented Oct 21, 2024

Labels

3 participants

Uh oh!

Conversation

chemamartinez commented Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Oct 10, 2024

elastic-vault-github-plugin-prod Bot commented Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Package cisco_duo 👍(2) 💚(1) 💔(5)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Oct 21, 2024

💚 Build Succeeded

History

elastic-sonarqube Bot commented Oct 21, 2024

Quality Gate passed

elastic-vault-github-plugin-prod Bot commented Oct 21, 2024

Labels

3 participants

chemamartinez commented Oct 10, 2024 •

edited

Loading

elastic-vault-github-plugin-prod Bot commented Oct 10, 2024 •

edited

Loading

Package `cisco_duo` 👍(2) 💚(1) 💔(5)