[Atlassian JIRA & Confluence] Add support for Atlassian Cloud

[legoguy1000]

What does this PR do?

Adds support for the SaaS versions of Confluence and JIRA in the Atlassian Cloud for the audit datastream.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Closes [Integrations] Atlassian Jira/Confluence Cloud Integration  #2695

Screenshots

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-04-27T23:19:25.590+0000

• Duration: 15 min 43 sec

Test stats 🧪

Test	Results
Failed	0
Passed	18
Skipped	0
Total	18

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[legoguy1000]

found some bugs, put back to draft

[legoguy1000]

I'm getting this issue [elastic_agent.filebeat][debug] template execution failed: template: :1:18: executing "" at <.url.params.offset>: can't evaluate field params in type interface {}. Is this related to elastic/beats#30477 or elastic/beats#30476?? Also getting [elastic_agent.filebeat][error] error processing response: template: :1:29: executing "" at <.last_response.url.params.Get>: map has no entry for key "url" using a different agent config.

[legoguy1000]

I was able to get around the above issue but now with the current config whenever it paginates, instead of just updating the offset via the response.pagination config, it also updates the date params using the cursor setting so it never actually paginates.

[andrewkroh]

I rebased this in order to apply the formatting of the pipeline test files in separate commits. This way it will be easier to digest the non-formatting changes during review. The JSON is now normalized due to elastic/elastic-package#644, but until all of the packages are normalized you'll get these large diffs the first time you make changes.

[andrewkroh]

/test

[legoguy1000]

even with the latest 8.0.0 docker image, I can't get past the error, #2715 (comment). Any thoughts?

[andrewkroh]

Do the tests not expose this problem? The CI checks are passing for this. It's using 7.16.0 for the tests. The 8.0 branch was missing a backport, and that has now been fixed, but not released.

[legoguy1000]

Do the tests not expose this problem? The CI checks are passing for this. It's using 7.16.0 for the tests. The 8.0 branch was missing a backport, and that has now been fixed, but not released.

I have to keep playing with it as I'm testing it against an actual Atlassian cloud instance and I keep getting the error so idk if I have a bad docker image or something but I've cleared all my local images and still having issues.

[andrewkroh]

Try testing with either 7.17.0 or 8.1.0 (e.g. elastic-package stack up -d -v --version=7.17.0).

[legoguy1000]

I'll try with 8.1. I didn't have any issues when I built filebeat from source on main branch using a basic input config to test. So it's definitely weird.

[legoguy1000]

@andrewkroh 8.1 appears to work and i'm no longer getting an error that the key doesn't exist but still having an issue with Pagination. I currently have

response.pagination:
  - set:
      target: url.params.from
      value: '[[.last_response.url.params.Get "from"]]'
      fail_on_template_error: true
...

This should maintain the existing URL parameter so the pagination continues through the same time range. However it's being overridden by the cursor and the original request Transforms. Any thoughts on how to fix that?

cursor:
  last_timestamp:
    value: "[[.first_event.created]]"

request.transforms:
...
  - set:
      target: url.params.from
      value: "[[.cursor.last_timestamp]]"
      default: '[[formatDate (now (parseDuration "-{{initial_interval}}")) "2006-01-02T15:04:05.999-0700"]]'
...

[P1llus]

@legoguy1000 In the current situation the request transforms run for each pagination, so I believe we might have to take another approach on this.

If date is the only possible value here, we might have to drop the pagination and simply let it catch up with a smaller interval. Unless there is some sort of range query, or other values we can use from response headers or body to paginate.

Let's catch up when you got time to discuss this! :)

[legoguy1000]

@P1llus copy. I'm artificially inducing pagination in order to make sure there is a valid solution as idk how many events/time is normal for these services. Unfortunately this seems like the only way to paginate but I can check again to see if there is anything in the headers.

[P1llus]

@legoguy1000
I would say that if there is limitations in the API that we can't easily work around, then we should simply not paginate, and rather have a smaller interval.
We can always have a discussion over slack if you want, to try to find other ways around it, but if there is none then we should keep it more simple, to ensure that it works correctly.

[legoguy1000]

@P1llus Looks like instead of trying to set individual params, setting url.value and build the entire query string seems to be a work around. Only issue is it doesn't URL encode the values so the TZ offset in the JIRA date params +0000 messes up the url. It looks like just leaving off the TZ part will work since all the dates are in UTC anyways. BUt the better solution would probably be to create a URL encode template function. Thoughts??

[legoguy1000]

Just tested without the TZ offset and the url.value, worked great. It paginated through 9 pages, 10 items each for 76 events total and the start/end dates stayed the same. Just need an answer on the URL encode functionality and we can press to merge this.

[legoguy1000]

@andrewkroh @P1llus Can we rerun the tests to see where we are. I think it should be good now. Currently its just not using the Time Zone in the dates since its all in UTC but when 8.3 is released, we can use the URL encode template function I made for the httpjson input to add it back just to be safe.

[andrewkroh]

/test

[andrewkroh]

/test

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (2/2)	💚
Files	100.0% (6/6)	💚 2.857
Classes	100.0% (6/6)	💚 2.857
Methods	100.0% (45/45)	💚 11.905
Lines	98.822% (839/849)	👍 9.985
Conditionals	100.0% (0/0)	💚

[andrewkroh]

Does any of this depend on elastic/beats#30962? If so the kibana constrained will need change to ^8.3.0.

[legoguy1000]

Long term it should be updated to use that function, but I was able to get around it. Instead of wrapping the datetime with +000 for timezone, since its always UTC, i just left the TZ offset from the URL and it seemed to work fine.

[andrewkroh]

/test

[andrewkroh]

/test