[Tenable IO] Update the input type from HTTPJSON to CEL.

[muskan-agarwal26]

Type of change
Bugfix

What does this PR do?

1. Update the input type to CEL for better debugging.
2. Added fingerprint in the plugin data stream.

Checklist

[x] I have reviewed tips for building integrations and this pull request is aligned with them.
[x] I have verified that all data streams collect metrics or logs.
[x] I have added an entry to my package's changelog.yml file.
[x] I have verified that Kibana version constraints are current according to guidelines.

All changes

[x] Change follows the contributing guidelines
[x] Supported versions of the monitoring target are documented
[x] Supported operating systems are documented (if applicable)
[x] Integration or System tests exist
[x] Documentation exists
[x] Fields follow ECS and naming conventions
[x] At least a manual test with ES / Kibana / Agent has been performed.
[x] Required Kibana version set to: ^8.12.0

How to test this PR locally

Clone integrations repo.
Install the elastic package locally.
Start the elastic stack using the elastic package.
Move to integrations/packages/tenable_io directory.
Run the following command to run tests.
elastic-package test -v

Related issues

Closes #9355

Automated Test

test-tenable.log

[cla-checker-service]

💚 CLA has been signed

[kcreddy]

/test

[elastic-sonarqube]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[kcreddy]

Suggested change

	- version: "2.10.0"
	- version: "3.0.0"

Lets update major

[kcreddy]

Suggested change

	type: bugfix
	type: enhancement

[kcreddy]

Add assert.hit_count to all datastreams system test config

[kcreddy]

In case we have PROCESSING export status, then state.want_more is true. This will continue and create a new export job even though the previous one is still PROCESSING. We are trying to address this issue by not creating new export job when previous one is already running. Wouldn't multiple jobs lead to corrupted state?

[muskan-agarwal26]

I've implemented a condition where, as long as the export status is PROCESSING, the state.want_more will remain true. This condition will continuously check the status in a loop until the export status is FINISHED. Consequently, no new export job will be initiated.

[kcreddy]

Alright, lets say after the 1st run, you created export job uuid - 1 and checked its status which resulted in PROCESSING. So, your want_more becomes true while checking /1/status as per:

integrations/packages/tenable_io/data_stream/asset/agent/stream/cel.yml.hbs

Line 91 in af807cf

"want_more": (inner_body.?status.orValue("") != "FINISHED" || size(inner_body.chunks_available) > 0) && int(now) - int(state.export_job_start_time) <= int(state.time_value),

Then, in your next iteration, since want_more is true, it calls post_request to create a new export job again.

integrations/packages/tenable_io/data_stream/asset/agent/stream/cel.yml.hbs

Lines 29 to 38 in af807cf

	has(state.want_more) && state.want_more ? state : state.with({
	"request_time": (
	state.?cursor.last_event_ts.orValue(null) != null ?
	state.cursor.last_event_ts
	:
	int(now - duration(state.initial_interval))
	)
	}).as(state, state.with(
	post_request(
	state.url + "/assets/export", "application/json",

I think there has to be a check if the previous job is finished or reached timeout, only then start a new export.
Has this been tested against a considerable payload where export status doesn't reach FINISHED immediately? You can try also with a system test config returning PROCESSING or QUEUED status and after reaching export_status_timeout, it creates a new export job.

[muskan-agarwal26]

If want_more is true based on the condition has(state.want_more) && state.want_more ? state : state.with({ , it won't run the state.with part of the code. Instead, it will proceed directly with state. So, for the as request: as(state, state.with( , it won't trigger the state.with code, meaning it won't call post_request to make a new export. Instead, it will call the /export/{export_uid}/status API.

Yes, we can provide a check if the previous job is finished or reached timeout, only then start a new export. I will add the code and update the PR soon.

[kcreddy]

how are you handling the case where export status is stuck in PROCESSING and never reaches FINISHED? I think this is possible on Tenable side. In this case, we may want to compare when our export job started and current time and if it still stuck in PROCESSING state, then clear the state for new export job to run.

[muskan-agarwal26]

What parameter shall we consider to deduce this?

[kcreddy]

You can calculate based on when the export job started (you can store this inside the state) and status is available from chunk_status

[kcreddy]

Bump major

[efd6]

Only CEL code reviewed.

[efd6]

This is a clean up of indentation, use of optional types, checking for field presence and updating user-agent version. It is untested.

Suggested change

	request("GET", state.url + "/plugins/plugin?last_updated=" + (
	has(state.want_more) && !state.want_more
	?
	(
	has(state.cursor) && has(state.cursor.last_update_time) && state.cursor.last_update_time != null
	?
	string((state.cursor.last_update_time)).split("T")[0]
	:
	"1970-01-01"
	)
	:
	(
	has(state.cursor) && has(state.cursor.first_update_time) && state.cursor.first_update_time != null
	?
	string((state.cursor.first_update_time)).split("T")[0]
	:
	""
	)) + "&page=" + string(state.next_page) + "&size=" + string(state.batch_size)
	).with({
	"Header":{
	"X-ApiKeys": ["accessKey=" + state.access_key + ";secretKey=" + state.secret_key],
	"User-Agent": ["Integration/1.0 (Elastic; Tenable.io; Build/2.9.0)"]
	}
	}).do_request().as(resp,
	resp.StatusCode == 200 ?
	bytes(resp.Body).decode_json().as(body, {
	"events": (has(body.data.plugin_details) && body.data.plugin_details.size() > 0 ?
	(
	body.data.plugin_details.map(e,{
	"message": e.encode_json()
	})
	)
	:
	(
	[{"message": body}]
	)
	),
	"cursor": {
	"last_update_time": (
	has(body.data.plugin_details) && body.data.plugin_details.size() > 0
	?
	(
	has(state.cursor) && has(state.cursor.last_update_time) && state.want_more
	?
	state.cursor.last_update_time
	:
	now
	)
	:
	has(state.cursor) && has(state.cursor.last_update_time)
	?
	state.cursor.last_update_time
	:
	null
	),
	"first_update_time": (
	has(state.cursor) && has(state.cursor.first_update_time) && state.cursor.first_update_time != null && has(body.data.plugin_details) && body.data.plugin_details.size() > 0
	?
	(
	body.size == int(state.batch_size) && state.want_more ? state.cursor.first_update_time : state.cursor.last_update_time
	)
	:
	"1970-01-01T00:00:00Z"
	),
	},
	"next_page": body.size == int(state.batch_size) ? int(state.next_page)+1 : 1,
	"want_more": body.size == int(state.batch_size),
	"access_key": state.access_key,
	"secret_key": state.secret_key,
	"batch_size": state.batch_size
	})
	:
	{
	"events": {
	"error": {
	"code": string(resp.StatusCode),
	"id": string(resp.Status),
	"message": string(resp.Body)
	},
	},
	"want_more": false,
	"next_page": 1,
	"batch_size": state.batch_size,
	"access_key": state.access_key,
	"secret_key": state.secret_key
	}
	)
	request("GET", state.url + "/plugins/plugin?last_updated=" + (
	has(state.want_more) && !state.want_more ?
	(
	has(state.?cursor.last_update_time) && state.cursor.last_update_time != null ?
	string(state.cursor.last_update_time).split("T")[0]
	:
	"1970-01-01"
	)
	:
	(
	has(state.?cursor.first_update_time) && state.cursor.first_update_time != null ?
	string((state.cursor.first_update_time)).split("T")[0]
	:
	""
	)
	) + "&page=" + string(state.next_page) + "&size=" + string(state.batch_size)
	).with({
	"Header":{
	"X-ApiKeys": ["accessKey=" + state.access_key + ";secretKey=" + state.secret_key],
	"User-Agent": ["Integration/1.0 (Elastic; Tenable.io; Build/"3.0.0)"] // Keep build version up-to-date with package version.
	}
	}).do_request().as(resp,
	resp.StatusCode == 200 ?
	bytes(resp.Body).decode_json().as(body, {
	"events": (
	has(body.?data.plugin_details) && body.data.plugin_details.size() > 0 ?
	body.data.plugin_details.map(e, {
	"message": e.encode_json()
	})
	:
	[{"message": body}]
	),
	"cursor": {
	"last_update_time": (
	has(body.?data.plugin_details) && body.data.plugin_details.size() > 0 ?
	(
	has(state.?cursor.last_update_time) && state.want_more ?
	state.cursor.last_update_time
	:
	now
	)
	:
	state.?cursor.last_update_time.orValue(null)
	),
	"first_update_time": (
	has(state.?cursor.first_update_time) && state.cursor.first_update_time != null && has(body.data.plugin_details) && body.data.plugin_details.size() > 0 ?
	(
	body.size == int(state.batch_size) && state.want_more ?
	state.cursor.first_update_time
	:
	state.cursor.last_update_time
	)
	:
	"1970-01-01T00:00:00Z"
	),
	},
	"next_page": body.size == int(state.batch_size) ? int(state.next_page)+1 : 1,
	"want_more": body.size == int(state.batch_size),
	"access_key": state.access_key,
	"secret_key": state.secret_key,
	"batch_size": state.batch_size
	})
	:
	{
	"events": {
	"error": {
	"code": string(resp.StatusCode),
	"id": string(resp.Status),
	"message": string(resp.Body)
	},
	},
	"want_more": false,
	"next_page": 1,
	"batch_size": state.batch_size,
	"access_key": state.access_key,
	"secret_key": state.secret_key
	}
	)

[efd6]

Fixes indentation and user-agent version. Untested.

Suggested change

	request("GET", state.url + "/scans").with({
	"Header":{
	"X-ApiKeys": ["accessKey=" + state.access_key + ";secretKey=" + state.secret_key],
	"User-Agent": ["Integration/1.0 (Elastic; Tenable.io; Build/2.9.0)"]
	}
	}).do_request().as(resp,
	resp.StatusCode == 200 ?
	bytes(resp.Body).decode_json().as(body, {
	"events": (has(body.scans) && body.scans.size() > 0 ?
	(
	body.scans.map(e, {
	"message": e.encode_json()
	})
	)
	:
	(
	[{"message": body}]
	)
	),
	"access_key": state.access_key,
	"secret_key": state.secret_key
	})
	:
	{
	"events": [{
	"error": {
	"code": string(resp.StatusCode),
	"id": string(resp.Status),
	"message": string(resp.Body)
	}
	}],
	"access_key": state.access_key,
	"secret_key": state.secret_key
	}
	)
	request("GET", state.url + "/scans").with({
	"Header":{
	"X-ApiKeys": ["accessKey=" + state.access_key + ";secretKey=" + state.secret_key],
	"User-Agent": ["Integration/1.0 (Elastic; Tenable.io; Build/3.0.0)"] // Keep up-to-date with package version.
	}
	}).do_request().as(resp,
	resp.StatusCode == 200 ?
	bytes(resp.Body).decode_json().as(body, {
	"events": (
	has(body.scans) && body.scans.size() > 0 ?
	(
	body.scans.map(e, {
	"message": e.encode_json()
	})
	)
	:
	[{"message": body}]
	),
	"access_key": state.access_key,
	"secret_key": state.secret_key
	})
	:
	{
	"events": [{
	"error": {
	"code": string(resp.StatusCode),
	"id": string(resp.Status),
	"message": string(resp.Body)
	}
	}],
	"access_key": state.access_key,
	"secret_key": state.secret_key
	}
	)

[efd6]

Program suggestion (GH will not let be scroll and select lines): fixes indentation, UA version, etc. Untested

  (
    state.?want_more.orValue(false) ? state : state.with({
      "request_time": (
        has(state.?cursor.last_event_time) && state.cursor.last_event_time != null ?
          state.cursor.last_event_time
        :
          int(now - duration(state.initial_interval))
      )
    }).as(state, state.with(
      post_request(state.url + "/vulns/export", "application/json",
        '{"num_assets":' + state.batch_size.encode_json() + ',"filters":{"since":' + state.request_time.encode_json() + ',"state": ["open","reopened","fixed"]}}'
      ).with({
        "Header":{
            "X-ApiKeys": ["accessKey=" + state.access_key + ";secretKey=" + state.secret_key],
            "User-Agent": ["Integration/1.0 (Elastic; Tenable.io; Build/3.0.0)"] // Keep up-to-date with package version.
        }
      }).do_request().as(resp,
        resp.StatusCode == 200 ?
          {
            "worklist": bytes(resp.Body).decode_json(),
            "cursor": {
              "last_event_time": int(now)
            }
          }
        :
          {
            "response_error": {
              "error": {
                "code": string(resp.StatusCode),
                "id": string(resp.Status),
                "message": string(resp.Body)
              },
            },
            "want_more": false,
            "worklist": {}
          }
        )
    ))
  ).as(state,
    has(state.?worklist.export_uuid) && state.?chunk_status.orValue("") != "FINISHED" ?
      request("GET",
        state.url +"/vulns/export/" + state.worklist.export_uuid + "/status"
      ).with({
        "Header":{
          "X-ApiKeys": ["accessKey=" + state.access_key + ";secretKey=" + state.secret_key],
          "User-Agent": ["Integration/1.0 (Elastic; Tenable.io; Build/3.0.0)"] // Keep up-to-date with package version.
        }
      }).do_request().as(response,
        response.StatusCode == 200 ?
          bytes(response.Body).decode_json().as(inner_body, {
            "want_more": inner_body.?status.orValue("") != "FINISHED" || size(inner_body.chunks_available) > 0,
            "chunk_status": inner_body.?status.orValue("") == "FINISHED" && size(inner_body.chunks_available) == 0 ? "QUEUED" : inner_body.status,
            "worklist": state.worklist,
            "chunks": inner_body.chunks_available,
            "url": state.url,
            "next": 0,
            "access_key": state.access_key,
            "secret_key": state.secret_key,
            "initial_interval": state.initial_interval,
            "batch_size": state.batch_size,
            "cursor": {
              "last_event_time": state.?cursor.last_event_time.orValue(int(now))
            }
          })
        :
          {
            "response_error": {
              "error": {
                "code": string(response.StatusCode),
                "id": string(response.Status),
                "message": string(response.Body)
              },
            },
            "url": state.url,
            "want_more": false,
            "chunks": [],
            "chunk_status": "QUEUED",
            "worklist": {},
            "next": 0,
            "access_key": state.access_key,
            "secret_key": state.secret_key,
            "initial_interval": state.initial_interval,
            "batch_size": state.batch_size,
          }
      )
    :
      {
        "url": state.url,
        "want_more": state.want_more,
        "chunks": state.?chunks.orValue([]),
        "chunk_status": state.chunk_status,
        "worklist": state.?worklist.orValue({}),
        "next": state.?next.orValue(0),
        "access_key": state.access_key,
        "secret_key": state.secret_key,
        "initial_interval": state.initial_interval,
        "batch_size": state.batch_size,
        "response_error": state.?response_error.orValue({}),
        "cursor": {
          "last_event_time": state.?cursor.last_event_time.orValue(int(now))
        }
      }
  ).as(state,
    has(state.?worklist.export_uuid) && state.?chunk_status.orValue("") == "FINISHED" && size(state.chunks) > 0 ?
      request("GET",
        state.url +"/vulns/export/" + state.worklist.export_uuid + "/chunks/" + string(state.chunks[state.next])
      ).with({
        "Header":{
          "X-ApiKeys": ["accessKey=" + state.access_key + ";secretKey=" + state.secret_key],
          "User-Agent": ["Integration/1.0 (Elastic; Tenable.io; Build/3.0.0)"] // Keep up-to-date with package version.
        }
      }).do_request().as(response1,
          response1.StatusCode == 200 ?
            bytes(response1.Body).decode_json().as(second_chain_body, {
              "events": (
                second_chain_body != null && size(second_chain_body) > 0 ?
                  second_chain_body.map(e, {
                    "message": e.encode_json()
                  })
                :
                  [{}]
              ),
              "want_more": (int(state.next)+1) < size(state.chunks),
              "worklist": (int(state.next)+1) < size(state.chunks) ? state.worklist : {},
              "next": (int(state.next)+1) < size(state.chunks) ? (int(state.next)+1) : 0,
              "chunk_status": ((int(state.next)+1) < size(state.chunks)) || (second_chain_body == null) ? state.chunk_status : "QUEUED",
              "access_key": state.access_key,
              "secret_key": state.secret_key,
              "initial_interval": state.initial_interval,
              "batch_size": state.batch_size,
              "chunks": (int(state.next)+1) < size(state.chunks) ? state.chunks : [],
              "url": state.url,
              "cursor": {
                "last_event_time": state.?cursor.last_event_time.orValue(int(now))
              }
            })
          :
            {
              "events": {
                "error": {
                  "code": string(response1.StatusCode),
                  "id": string(response1.Status),
                  "message": string(response1.Body)
                },
              },
              "url": state.url,
              "want_more": false,
              "chunks": [],
              "chunk_status": "QUEUED",
              "worklist": {},
              "next": 0,
              "access_key": state.access_key,
              "secret_key": state.secret_key,
              "initial_interval": state.initial_interval,
              "batch_size": state.batch_size,
            }
      )
    :
      {
        "events": state.?response_error.orValue(null) != null ? state.response_error : [{}],
        "url": state.url,
        "chunks": has(state.chunks) ? state.chunks : [],
        "chunk_status": state.chunk_status,
        "want_more": state.want_more,
        "worklist": has(state.worklist) ? state.worklist : {},
        "next": has(state.next) ? state.next : 0,
        "access_key": state.access_key,
        "secret_key": state.secret_key,
        "initial_interval": state.initial_interval,
        "batch_size": state.batch_size,
        "cursor": {
          "last_event_time": state.?cursor.last_event_time.orValue(int(now))
        }
      }
  )

[efd6]

Program suggestion. Untested. (Does not take into account comments by @kcreddy)

  (
    has(state.want_more) && state.want_more ? state : state.with({
      "request_time": (
        state.?cursor.last_event_ts.orValue(null) != null ?
          state.cursor.last_event_ts
        :
          int(now - duration(state.initial_interval))
      )
    }).as(state, state.with(
      post_request(
        state.url + "/assets/export", "application/json",
        '{"chunk_size":' + state.batch_size.encode_json() + ',"filters":{"updated_at": ' + state.request_time.encode_json() + '}}'
      ).with({
        "Header":{
          "X-ApiKeys": ["accessKey=" + state.access_key + ";secretKey=" + state.secret_key],
          "User-Agent": ["Integration/1.0 (Elastic; Tenable.io; Build/3.0.0)"] // Keep up-to-date with package version.
        }
      }).do_request().as(resp,
        resp.StatusCode == 200 ?
          {
            "worklist": bytes(resp.Body).decode_json(),
            "cursor": {
              "last_event_ts": int(now)
            }
          }
        :
          {
            "response_error": {
              "error": {
                "code": string(resp.StatusCode),
                "id": string(resp.Status),
                "message": string(resp.Body)
              },
            },
            "want_more": false,
            "worklist": {}
          }
      )
    ))
  ).as(state, 
    has(state.?worklist.export_uuid) && state.?chunk_status.compare.orValue("") != "FINISHED" ?
      request("GET",
        state.url +"/assets/export/" + state.worklist.export_uuid + "/status"
      ).with({
        "Header":{
          "X-ApiKeys": ["accessKey=" + state.access_key + ";secretKey=" + state.secret_key],
          "User-Agent": ["Integration/1.0 (Elastic; Tenable.io; Build/3.0.0)"] // Keep up-to-date with package version.
        }
      }).do_request().as(response,
        response.StatusCode == 200 ?
          bytes(response.Body).decode_json().as(inner_body, {
            "want_more": inner_body.?status.orValue("") != "FINISHED" || size(inner_body.chunks_available) > 0,
            "chunk_status": inner_body.?status.orValue("") == "FINISHED" && size(inner_body.chunks_available) == 0 ? "QUEUED" : inner_body.status,
            "worklist": state.worklist,
            "chunks": inner_body.chunks_available,
            "url": state.url,
            "next": 0,
            "access_key": state.access_key,
            "secret_key": state.secret_key,
            "initial_interval": state.initial_interval,
            "batch_size": state.batch_size,
            "cursor": {
              "last_event_ts": state.?cursor.last_event_ts.orValue(int(now))
            }
          })
        :
          {
            "response_error": {
              "error": {
                "code": string(response.StatusCode),
                "id": string(response.Status),
                "message": string(response.Body)
              },
            },
            "url": state.url,
            "want_more": false,
            "chunks": [],
            "chunk_status": "QUEUED",
            "worklist": {},
            "next": 0,
            "access_key": state.access_key,
            "secret_key": state.secret_key,
            "initial_interval": state.initial_interval,
            "batch_size": state.batch_size,
          }
      )
    :
      {
        "url": state.url,
        "want_more": state.want_more,
        "chunks": has(state.chunks) ? state.chunks : [],
        "chunk_status": state.chunk_status,
        "worklist": has(state.worklist) ? state.worklist : {},
        "next": has(state.next) ? state.next : 0,
        "access_key": state.access_key,
        "secret_key": state.secret_key,
        "initial_interval": state.initial_interval,
        "batch_size": state.batch_size,
        "response_error": has(state.response_error) ? state.response_error : {},
        "cursor": {
          "last_event_ts": (
            has(state.cursor) && has(state.cursor.last_event_ts)
            ?
              state.cursor.last_event_ts
            :
              int(now)
          )
        }
      }
  ).as(state,
    has(state.?worklist.export_uuid) && state.?chunk_status.orValue("") == "FINISHED" && size(state.chunks) > 0 ?
      request("GET",
        state.url +"/assets/export/" + state.worklist.export_uuid + "/chunks/" + string(state.chunks[state.next])
      ).with({
        "Header":{
          "X-ApiKeys": ["accessKey=" + state.access_key + ";secretKey=" + state.secret_key],
          "User-Agent": ["Integration/1.0 (Elastic; Tenable.io; Build/3.0.0)"] // Keep up-to-date with package version.
        }
      }).do_request().as(response1,
        response1.StatusCode == 200 ?
          bytes(response1.Body).decode_json().as(second_chain_body, {
            "events": (second_chain_body != null && size(second_chain_body) > 0 ?
              second_chain_body.map(e,{
                "message": e.encode_json()
              })
            :
              [{}]
            ),
            "want_more": (int(state.next)+1) < size(state.chunks),
            "worklist": (int(state.next)+1) < size(state.chunks) ? state.worklist : {},
            "next": (int(state.next)+1) < size(state.chunks) ? (int(state.next)+1) : 0,
            "chunk_status": ((int(state.next)+1) < size(state.chunks)) || (second_chain_body == null) ? state.chunk_status : "QUEUED",
            "access_key": state.access_key,
            "secret_key": state.secret_key,
            "initial_interval": state.initial_interval,
            "batch_size": state.batch_size,
            "chunks": (int(state.next)+1) < size(state.chunks) ? state.chunks : [],
            "url": state.url,
            "cursor": {
              "last_event_ts": state.?cursor.last_event_ts.orValue(int(now))
            }
          })
        :
          {
            "events": {
              "error": {
                "code": string(response1.StatusCode),
                "id": string(response1.Status),
                "message": string(response1.Body)
              },
            },
            "url": state.url,
            "want_more": false,
            "chunks": [],
            "chunk_status": "QUEUED",
            "worklist": {},
            "next": 0,
            "access_key": state.access_key,
            "secret_key": state.secret_key,
            "initial_interval": state.initial_interval,
            "batch_size": state.batch_size,
          }
      )
    :
      {
        "events": state.?response_error.orValue(null) != null ? state.response_error : [{}],
        "url": state.url,
        "chunks": state.?chunks.orValue([]),
        "chunk_status": state.chunk_status,
        "want_more": state.want_more,
        "worklist": state.?worklist.orValue({}),
        "next": state.?next.orValue(0),
        "access_key": state.access_key,
        "secret_key": state.secret_key,
        "initial_interval": state.initial_interval,
        "batch_size": state.batch_size,
        "cursor": {
          "last_event_ts": state.?cursor.last_event_ts.orValue(int(now))
        }
      }
  )

[kcreddy]

I added my comment here - #9466 (comment)

Essentially we are trying to solve both problems noted in the original issue by migrating httpjson to CEL: #9355 (comment) which I don't think CEL code is currently doing. Please add system tests to simulate both the conditions (or) atleast test both conditions on real data on test instance.

[efd6]

I agree with @kcreddy in #9466 (comment). The issues here are subtle enough and the code complex enough that this warrants having a test to show that it is fixed. If this cannot be done with the current tools provided by stream, then we need to make that possible.

[efd6]

Suggested change

	The minimum **kibana.version** required is **8.7.1**.
	The minimum **kibana.version** required is **8.12.0**.

[efd6]

This should include the bugfixes in the changes as well, as separate elements in the changes list.

[efd6]

What are these for? The export status timeout field is defined in the documentation to match the time.Duration format and CEL's duration type natively parses this format.

Type conversion. Duration strings should support the following suffixes: "h" (hour), "m" (minute), "s" (second), "ms" (millisecond), "us" (microsecond), and "ns" (nanosecond). Duration strings may be zero, negative, fractional, and/or compound. Examples: "0", "-1.5h", "1m6s"

I'd also say that "time_value" is not an informative name; it should refer to the function rather than the type (which I think is also wrong, since it's not a time, but a duration).

Suggest making this an actual time by adding the export_status_timeout to now into "expires" and then comparing now against that to evaluate whether the request is too old.

[efd6]

Is the "Status" part of this meaningful to users? I don't think so. Suggest "Export Request Timeout".

[muskan-agarwal26]

We've set up a way to check if the /export/{export_uuid}/status API has returned FINISHED in status. If it's not finished, we'll stop the process after a certain time (user configurable). After that, we won't keep checking the export status. So, the export_status_timeout variable represents this check for the /export/{export_uuid}/status API.

[efd6]

I've seen this .split("T")[0] in a number of places but I do not understand why it is being used; from the documentation for the various APIs that have CEL programs with this idiom, they use RFC3339 formatted time. string(timestamp) returns this, so it is surprising that this additional step is taken. Can you explain why it is needed, otherwise, please remove it.

Also, we don't need the parens around the field access.

Suggested change

	string((state.cursor.first_update_time)).split("T")[0]
	string(state.cursor.first_update_time).split("T")[0]

[efd6]

Suggested change

	has(state.?cursor.last_update_time) && state.cursor.last_update_time != null ?
	state.?cursor.last_update_time.orValue(null) != null ?

(similar throughout)

[efd6]

This is unsafe either without documentation or additonal string handling; if the user enters a URL with a trailing slash (some environments will add this to the clipboard without user intervention) then we have a URL with a double slash separating the path from the host. This breaks some things. Either add as a note in the docs that the base URL entered in to the UI should not have trailing slash, or (preferably) do

Suggested change

	request("GET", state.url + "/plugins/plugin?last_updated=" + (
	request("GET", state.url.trim_right("/") + "/plugins/plugin?last_updated=" + (

(throughout)

[efd6]

Suggested change

	has(state.?cursor.last_event_time) && state.cursor.last_event_time != null ?
	state.?cursor.last_event_time.orValue(null) != null ?

[efd6]

Same comment as above.

[efd6]

To clarify, this doesn't look like a worklist to me, but rather a next item to collect. Is that correct? It always holds an object and never an array. Can we change this to a less misleading name?

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[efd6]

/test

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[efd6]

This LGTM now. I have done some final indentation fixups which can be viewed in the final commit; best viewed with hide whitespace where you'll see only -2 +4 changes which are line diff whitespacing. Approving, but wait for @kcreddy.

[efd6]

/test

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 0c5cab7

History

• 💚 Build #11290 succeeded fb2f94d
• 💚 Build #9982 succeeded 11c90c4

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[kcreddy]

LGTM 👍🏼

[elasticmachine]

Package tenable_io - 3.0.0 containing this change is available at https://epr.elastic.co/search?package=tenable_io

[elasticmachine]

Package tenable_io - 3.0.0 containing this change is available at https://epr.elastic.co/search?package=tenable_io