Skip to content

atlassian_{bitbucket,confluence,jira}: improve pipeline robustness to empty strings - #12977

Merged
efd6 merged 1 commit into
elastic:mainfrom
efd6:s5722-atlassian
Mar 6, 2025
Merged

atlassian_{bitbucket,confluence,jira}: improve pipeline robustness to empty strings#12977
efd6 merged 1 commit into
elastic:mainfrom
efd6:s5722-atlassian

Conversation

@efd6

@efd6 efd6 commented Mar 6, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed commit message

atlassian_{bitbucket,confluence,jira}: improve pipeline robustness to empty strings

Atlassian will send empty string values rather than omit fields, so null
out all empty string values at the start of processing. Also improve
error reporting and script robustness.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots
@efd6 efd6 added enhancement New feature or request Integration:atlassian_jira Atlassian Jira Integration:atlassian_confluence Atlassian Confluence Integration:atlassian_bitbucket Atlassian Bitbucket bugfix Pull request that fixes a bug issue Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Mar 6, 2025
@efd6 efd6 self-assigned this Mar 6, 2025
@efd6
efd6 force-pushed the s5722-atlassian branch 2 times, most recently from cda573b to 4cf6aeb Compare March 6, 2025 01:32
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Mar 6, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport
@efd6
efd6 marked this pull request as ready for review March 6, 2025 01:55
@efd6
efd6 requested a review from a team as a code owner March 6, 2025 01:55
@elasticmachine

elasticmachine commented Mar 6, 2025

Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)
efd6
efd6 commented Mar 6, 2025
View reviewed changes
Comment thread packages/atlassian_bitbucket/data_stream/audit/_dev/test/pipeline/test-audit-api.log
{"timestamp":"2021-11-27T17:26:25.141Z","author":{"name":"System","type":"system","id":"-1","avatarUri":""},"type":{"categoryI18nKey":"bitbucket.service.audit.category.usersandgroups","category":"Users and groups","actionI18nKey":"bitbucket.service.user.audit.action.groupcreated","action":"User group created"},"affectedObjects":[{"name":"stash-users","type":"GROUP","uri":"http://bitbucket.internal:7990/admin/groups/view?name=stash-users","id":"stash-users"}],"changedValues":[],"node":"8767044c-1b98-4d64-82db-ef29af8c3792","method":"System","extraAttributes":[{"nameI18nKey":"bitbucket.audit.attribute.legacy.target","name":"target","value":"stash-users"}]}
{"timestamp":"2021-11-27T17:26:25.045Z","author":{"name":"System","type":"system","id":"-1","avatarUri":""},"type":{"categoryI18nKey":"bitbucket.service.audit.category.usersandgroups","category":"Users and groups","actionI18nKey":"bitbucket.service.user.audit.action.directorycreated","action":"User directory created"},"affectedObjects":[],"changedValues":[],"node":"8767044c-1b98-4d64-82db-ef29af8c3792","method":"System","extraAttributes":[{"nameI18nKey":"bitbucket.service.user.audit.attribute.directory.name","name":"Directory name","value":"Bitbucket Internal Directory"},{"nameI18nKey":"bitbucket.audit.attribute.legacy.target","name":"target","value":"Bitbucket Internal Directory"}]} No newline at end of file
{"timestamp":"2021-11-27T17:26:25.045Z","author":{"name":"System","type":"system","id":"-1","avatarUri":""},"type":{"categoryI18nKey":"bitbucket.service.audit.category.usersandgroups","category":"Users and groups","actionI18nKey":"bitbucket.service.user.audit.action.directorycreated","action":"User directory created"},"affectedObjects":[],"changedValues":[],"node":"8767044c-1b98-4d64-82db-ef29af8c3792","method":"System","extraAttributes":[{"nameI18nKey":"bitbucket.service.user.audit.attribute.directory.name","name":"Directory name","value":"Bitbucket Internal Directory"},{"nameI18nKey":"bitbucket.audit.attribute.legacy.target","name":"target","value":"Bitbucket Internal Directory"}]}
{"timestamp":"2021-11-27T17:38:53.360Z","author":{"name":"admin","type":"NORMAL","id":"2","uri":"http://bitbucket.internal:7990/users/admin","avatarUri":""},"type":{"categoryI18nKey":"bitbucket.service.audit.category.usersandgroups","category":"Users and groups","actionI18nKey":"bitbucket.service.user.audit.action.userrenamed","action":"Username changed"},"affectedObjects":[{"name":"test.user","type":"USER","id":"3"}],"changedValues":[{"key":"Username","i18nKey":"bitbucket.service.user.audit.attribute.user.name","from":"test","to":"test.user"}],"source":"","system":"http://bitbucket.internal:7990","node":"8767044c-1b98-4d64-82db-ef29af8c3792","method":"Browser","extraAttributes":[{"nameI18nKey":"bitbucket.audit.attribute.legacy.details","name":"details","value":"{\"oldUsername\":\"test\"}"},{"nameI18nKey":"bitbucket.audit.attribute.legacy.target","name":"target","value":"test.user"}]}

@efd6 efd6 Mar 6, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"source":""
Comment thread packages/atlassian_confluence/data_stream/audit/_dev/test/pipeline/test-audit-api.log
{"timestamp":"2021-11-28T17:06:11.805Z","author":{"name":"Joe Bob","type":"user","id":"2c9680837d4a3682017d4a375a280000","uri":"http://confluence.internal:8090/admin/users/viewuser.action?username=admin123","avatarUri":""},"type":{"categoryI18nKey":"audit.logging.category.user.management","category":"Users and groups","actionI18nKey":"audit.logging.summary.user.updated","action":"User details updated"},"affectedObjects":[{"name":"asdf asdfasdf","type":"User","uri":"http://confluence.internal:8090/admin/users/viewuser.action?username=asdf123","id":"2c9680837d4a3682017d67821e520003"}],"changedValues":[],"source":"10.100.100.2","system":"http://confluence.internal:8090","method":"Browser","extraAttributes":[]}
{"timestamp":"2021-11-28T17:05:37.158Z","author":{"name":"Joe Bob","type":"user","id":"2c9680837d4a3682017d4a375a280000","uri":"http://confluence.internal:8090/admin/users/viewuser.action?username=admin123","avatarUri":""},"type":{"categoryI18nKey":"audit.logging.category.user.management","category":"Users and groups","actionI18nKey":"audit.logging.summary.user.updated","action":"User details updated"},"affectedObjects":[{"name":"asdf asdfasdf","type":"User","uri":"http://confluence.internal:8090/admin/users/viewuser.action?username=asdf123","id":"2c9680837d4a3682017d67821e520003"}],"changedValues":[{"key":"Display name","i18nKey":"Display name","from":"asdf","to":"asdf asdfasdf"}],"source":"10.100.100.2","system":"http://confluence.internal:8090","method":"Browser","extraAttributes":[]} No newline at end of file
{"timestamp":"2021-11-28T17:05:37.158Z","author":{"name":"Joe Bob","type":"user","id":"2c9680837d4a3682017d4a375a280000","uri":"http://confluence.internal:8090/admin/users/viewuser.action?username=admin123","avatarUri":""},"type":{"categoryI18nKey":"audit.logging.category.user.management","category":"Users and groups","actionI18nKey":"audit.logging.summary.user.updated","action":"User details updated"},"affectedObjects":[{"name":"asdf asdfasdf","type":"User","uri":"http://confluence.internal:8090/admin/users/viewuser.action?username=asdf123","id":"2c9680837d4a3682017d67821e520003"}],"changedValues":[{"key":"Display name","i18nKey":"Display name","from":"asdf","to":"asdf asdfasdf"}],"source":"10.100.100.2","system":"http://confluence.internal:8090","method":"Browser","extraAttributes":[]}
{"timestamp":"2021-11-28T17:05:37.158Z","author":{"name":"Joe Bob","type":"user","id":"2c9680837d4a3682017d4a375a280000","uri":"http://confluence.internal:8090/admin/users/viewuser.action?username=admin123","avatarUri":""},"type":{"categoryI18nKey":"audit.logging.category.user.management","category":"Users and groups","actionI18nKey":"audit.logging.summary.user.updated","action":"User details updated"},"affectedObjects":[{"name":"asdf asdfasdf","type":"User","uri":"http://confluence.internal:8090/admin/users/viewuser.action?username=asdf123","id":"2c9680837d4a3682017d67821e520003"}],"changedValues":[{"key":"Display name","i18nKey":"Display name","from":"asdf","to":"asdf asdfasdf"}],"source":"","system":"http://confluence.internal:8090","method":"Browser","extraAttributes":[]}

@efd6 efd6 Mar 6, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"source":""
Comment thread packages/atlassian_jira/data_stream/audit/_dev/test/pipeline/test-audit-api.log
{"timestamp":"2021-11-28T18:23:20.278Z","author":{"name":"admin.user1","type":"ApplicationUser","id":"10000","uri":"http://jira.internal:8088/secure/ViewProfile.jspa?name=admin.user1","avatarUri":""},"type":{"categoryI18nKey":"jira.auditing.category.usermanagement","category":"user management","actionI18nKey":"jira.auditing.user.updated","action":"User updated"},"affectedObjects":[{"name":"admin.user1","type":"USER","uri":"http://jira.internal:8088/secure/ViewProfile.jspa?name=admin.user1","id":"JIRAUSER10000"}],"changedValues":[{"key":"Email","i18nKey":"common.words.email","from":"admin@example.com","to":"admin1@example.com"}],"source":"10.100.100.2","system":"http://jira.internal:8088","method":"Browser","extraAttributes":[]}
{"timestamp":"2021-11-28T18:23:13.741Z","author":{"name":"admin.user1","type":"ApplicationUser","id":"10000","uri":"http://jira.internal:8088/secure/ViewProfile.jspa?name=admin.user1","avatarUri":""},"type":{"categoryI18nKey":"jira.auditing.category.usermanagement","category":"user management","actionI18nKey":"jira.auditing.user.updated","action":"User updated"},"affectedObjects":[{"name":"admin.user1","type":"USER","uri":"http://jira.internal:8088/secure/ViewProfile.jspa?name=admin.user1","id":"JIRAUSER10000"}],"changedValues":[{"key":"Full name","i18nKey":"common.words.fullname","from":"Admin User","to":"Admin User1"}],"source":"10.100.100.2","system":"http://jira.internal:8088","method":"Browser","extraAttributes":[]} No newline at end of file
{"timestamp":"2021-11-28T18:23:13.741Z","author":{"name":"admin.user1","type":"ApplicationUser","id":"10000","uri":"http://jira.internal:8088/secure/ViewProfile.jspa?name=admin.user1","avatarUri":""},"type":{"categoryI18nKey":"jira.auditing.category.usermanagement","category":"user management","actionI18nKey":"jira.auditing.user.updated","action":"User updated"},"affectedObjects":[{"name":"admin.user1","type":"USER","uri":"http://jira.internal:8088/secure/ViewProfile.jspa?name=admin.user1","id":"JIRAUSER10000"}],"changedValues":[{"key":"Full name","i18nKey":"common.words.fullname","from":"Admin User","to":"Admin User1"}],"source":"10.100.100.2","system":"http://jira.internal:8088","method":"Browser","extraAttributes":[]}
{"timestamp":"2021-11-28T18:23:13.741Z","author":{"name":"admin.user1","type":"ApplicationUser","id":"10000","uri":"http://jira.internal:8088/secure/ViewProfile.jspa?name=admin.user1","avatarUri":""},"type":{"categoryI18nKey":"jira.auditing.category.usermanagement","category":"user management","actionI18nKey":"jira.auditing.user.updated","action":"User updated"},"affectedObjects":[{"name":"admin.user1","type":"USER","uri":"http://jira.internal:8088/secure/ViewProfile.jspa?name=admin.user1","id":"JIRAUSER10000"}],"changedValues":[{"key":"Full name","i18nKey":"common.words.fullname","from":"Admin User","to":"Admin User1"}],"source":"","system":"http://jira.internal:8088","method":"Browser","extraAttributes":[]}

@efd6 efd6 Mar 6, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"source":""
kcreddy
kcreddy reviewed Mar 6, 2025
View reviewed changes
Comment thread packages/atlassian_bitbucket/changelog.yml
Comment on lines +4 to +6
- description: Improve error reporting.
type: enhancement
link: https://github.com/elastic/integrations/pull/12977

@kcreddy kcreddy Mar 6, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In bitbucket changes, looks like improved error reporting is missing.

@efd6 efd6 Mar 6, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Added.
@efd6
atlassian_{bitbucket,confluence,jira}: improve pipeline robustness to…
3747c71 
… empty strings

Atlassian will send empty string values rather than omit fields, so null
out all empty string values at the start of processing. Also improve
error reporting and script robustness.
@efd6
efd6 force-pushed the s5722-atlassian branch from 4cf6aeb to 3747c71 Compare March 6, 2025 07:00
@elastic-sonarqube

elastic-sonarqube Bot commented Mar 6, 2025

Copy link
Copy Markdown

Quality Gate failed Quality Gate failed

Failed conditions
71.8% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube
@elasticmachine

elasticmachine commented Mar 6, 2025

Copy link
Copy Markdown

💚 Build Succeeded

History

cc @efd6
@efd6
efd6 merged commit 51347ba into elastic:main Mar 6, 2025
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Mar 6, 2025

Copy link
Copy Markdown
Contributor

Package atlassian_bitbucket - 2.5.0 containing this change is available at https://epr.elastic.co/package/atlassian_bitbucket/2.5.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Mar 6, 2025

Copy link
Copy Markdown
Contributor

Package atlassian_confluence - 1.29.0 containing this change is available at https://epr.elastic.co/package/atlassian_confluence/1.29.0/
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Mar 6, 2025

Copy link
Copy Markdown
Contributor

Package atlassian_jira - 1.30.0 containing this change is available at https://epr.elastic.co/package/atlassian_jira/1.30.0/
flexitrev pushed a commit that referenced this pull request Mar 20, 2025
@efd6 @flexitrev
atlassian_{bitbucket,confluence,jira}: improve pipeline robustness to…
08b06e9 
… empty strings (#12977)

Atlassian will send empty string values rather than omit fields, so null
out all empty string values at the start of processing. Also improve
error reporting and script robustness.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Pull request that fixes a bug issue enhancement New feature or request Integration:atlassian_bitbucket Atlassian Bitbucket Integration:atlassian_confluence Atlassian Confluence Integration:atlassian_jira Atlassian Jira Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

3 participants

@efd6 @elasticmachine @kcreddy