[cisco_duo] Avoid obsolete cursor data in activity, telephony_v2

[chrisberkhout]

Proposed commit message

[cisco_duo] Avoid obsolete cursor data in activity, telephony_v2

In an earlier version of the CEL code for `activity`[1] and
`telephony_v2`[2], `cursor.last_published` was set to a UNIX timestamp
value. This was changed to use RFC3339 formatted times in later
PRs[3][4] (with the corresponding read-time parsing added in [5]).

Users who didn't create a new policy may have the current parsing logic
fail when it encounters an old UNIX timestamp value in
`cursor.last_published`.

This PR addresses that issue by renaming `cursor.last_published` to 
`cursor.last_response_ts`. That effectively clears the cursor so that
obsolete values will not be seen.

[1]: https://github.com/elastic/integrations/blob/2ea993/packages/cisco_duo/data_stream/activity/agent/stream/cel.yml.hbs#L111-L114
[2]: https://github.com/elastic/integrations/blob/2ea993/packages/cisco_duo/data_stream/telephony_v2/agent/stream/cel.yml.hbs#L112-L115
[3]: https://github.com/elastic/integrations/pull/11640
[4]: https://github.com/elastic/integrations/pull/11670
[5]: https://github.com/elastic/integrations/pull/11772

Discussion

The API documentation's description of the ts field remains consistent with #11772.

An alternative to this PR would be to parse both legacy and new data, as follows:

Details

diff --git a/packages/cisco_duo/data_stream/activity/agent/stream/cel.yml.hbs b/packages/cisco_duo/data_stream/activity/agent/stream/cel.yml.hbs
index c03f4a1ed3..ffea8d42de 100644
--- a/packages/cisco_duo/data_stream/activity/agent/stream/cel.yml.hbs
+++ b/packages/cisco_duo/data_stream/activity/agent/stream/cel.yml.hbs
@@ -39,7 +39,10 @@ program: |
             state.with({
                 "mintime": string(1000 * int(
                     state.?cursor.last_published.optMap(t,
-                        t.parse_time(time_layout.RFC3339Nano)
+                        !is_error(int(t)) ?
+                          timestamp(int(t)/1000)+duration(string(int(t)%1000)+"ms")
+                        :
+                          t.parse_time(time_layout.RFC3339Nano)
                     ).orValue(
                         now - duration(state.initial_interval)
                     )
diff --git a/packages/cisco_duo/data_stream/telephony_v2/agent/stream/cel.yml.hbs b/packages/cisco_duo/data_stream/telephony_v2/agent/stream/cel.yml.hbs
index 553258248b..000c160d67 100644
--- a/packages/cisco_duo/data_stream/telephony_v2/agent/stream/cel.yml.hbs
+++ b/packages/cisco_duo/data_stream/telephony_v2/agent/stream/cel.yml.hbs
@@ -39,7 +39,10 @@ program: |
             state.with({
                 "mintime": string(1000 * int(
                     state.?cursor.last_published.optMap(t,
-                        t.parse_time(time_layout.RFC3339Nano)
+                        !is_error(int(t)) ?
+                          timestamp(int(t)/1000)+duration(string(int(t)%1000)+"ms")
+                        :
+                          t.parse_time(time_layout.RFC3339Nano)
                     ).orValue(
                         now - duration(state.initial_interval)
                     )

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

• Relates [Cisco Duo] - Added fix for 400 and 401 errors in telephony v2 data stream #11640
• Relates cisco_duo: fix 400 and 401 errors in Auth and Activity #11670
• Relates cisco_duo: fix the parsing of last_published timestamp #11772

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[efd6]

The naming here makes me feel a little icky; last_published_ts or last_response_ts?

[chrisberkhout]

The naming here makes me feel a little icky; last_published_ts or last_response_ts?

Sure. Now it's last_response_ts.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: d933886246fb0e1ea76329abc9fad7f4854d0677

History

• 💚 Build #21054 succeeded 335ef0e7ceff3044d56e27997936c53c525c9880

cc @chrisberkhout

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[efd6]

Thanks

[elastic-vault-github-plugin-prod]

Package cisco_duo - 2.3.2 containing this change is available at https://epr.elastic.co/package/cisco_duo/2.3.2/