packages/salesforce: Revamp Salesforce integration by kush-elastic · Pull Request #9629 · elastic/integrations

kush-elastic · 2024-04-17T06:32:49Z

Description

This PR introduces a new input method for our Salesforce integration and related changes, designed to overcome the limitations of HTTPJSON input in handling large data volumes from Salesforce Event Log Files (ELF) and real-time event monitoring. The new approach utilizes SOQL queries to directly collect data from Salesforce, offering improved performance, reliability, and scalability. This change addresses several issues with the current implementation and enables more efficient processing of Salesforce data, enhancing our overall integration capabilities.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Related issues

Screenshots

Configurations

Before

After

- remove login_stream and logout_stream - update config for login, logout, apex and setupaudittrail data-stream - update configurations

…nto 1278_revamp_salesforce_integration

- Update handlebar files for all the datastreams. - Update pipelines for all the datastreams - Update field mappings - update dashboards

…ps://github.com/kush-elastic/integrations into 1278_revamp_salesforce_integration

kush-elastic · 2024-05-15T09:55:23Z

Updates:

Enhanced Authentication Configuration Flexibility
Discussions are underway to offer users a more flexible approach to configuring the authentication mechanism. Several issues have been identified in relation to supporting new configuration methods. After assessing the current situation and challenges, the decision has been made to continue with the existing method for now. Future plans include transitioning to support the new configuration method.
System Test Status
Currently, system tests are encountering failures due to the absence of the new Salesforce input in the latest elastic-agent images. The resolution is pending the availability of this input in the updated images.

shmsr · 2024-06-11T12:49:52Z

@kush-elastic @aliabbas-elastic I have made a lot of changes to login data stream. Do have a look: 8c805a8

Changes include:

Fixed types
Fixed field names and groupings
Fixed descriptions
Use the correct processors
Updated the sample logs to get the correct UA
Use match_only_text for AdditionalInfo (space-optimized, no scoring)
etc.

…om/kush-elastic/integrations into 1278_revamp_salesforce_integration

Co-authored-by: subham sarkar <sarkar.subhams2@gmail.com>

ali786XI · 2024-06-24T08:37:06Z

We found this during testing that some of the values (especially for Apex data stream) a re received as -1 in response. For example attaching one field for which this is encountered

There was one more field with this value json.REQUEST_SIZE mapped to http.request.bytes. We don't have any documentation for these -1 values encountered currently as they are received from the logs only. If it would be given in Salesforce documentation somewhere that gives confidence over dropping/ignoring these field values then we would have ideally done that.

Co-authored-by: Aliabbas Attarwala <124054599+aliabbas-elastic@users.noreply.github.com>

shmsr · 2024-06-24T09:10:39Z

-          unit: ms
+          type: long
+          description: The time in nanoseconds for the database round trip during login, including time spent in the JDBC driver, network, and database CPU time.
+          unit: nanos


muthu-mps · 2024-06-25T07:33:58Z

@shmsr - As I see this is a breaking change based on the field rename and the dashboard removal. We consider making this change as the integration is still in BETA and I am just looking into the comment in the meta issue.

@lalit-satapathy - I hope this is fine.

To be honest, this seems like the way to go. I personally don't have an example to compare with to weigh the pros and cons here. But for beta (0.x) stuff, it's understandable that there could be breaking changes. I think that's why it seems like a fair decision. Anyway, we're just merging this pull request. Kush isn't going to push the CHANGELOG changes. We will do it after dashboard PR is also merged; it is nearing completion. So, we still have some time. If we find a better alternative, we'll go with that instead.

@kush-elastic Removing changelog right as we discussed?

kush-elastic · 2024-06-26T09:19:29Z

To be honest, this seems like the way to go. I personally don't have an example to compare with to weigh the pros and cons here. But for beta (0.x) stuff, it's understandable that there could be breaking changes. I think that's why it seems like a fair decision. Anyway, we're just merging this pull request. Kush isn't going to push the CHANGELOG changes. We will do it after dashboard PR is also merged; it is nearing completion. So, we still have some time. If we find a better alternative, we'll go with that instead.

@kush-elastic Removing changelog right as we discussed?

Yes @muthu-mps,
I have updated the change log to use next tag and we will release integration once all the PRs are merged.

Co-authored-by: subham sarkar <sarkar.subhams2@gmail.com>

elasticmachine · 2024-06-26T11:02:55Z

💚 Build Succeeded

Buildkite Build
Commit: 2636e80

History

💚 Build #12914 succeeded 7f32265
💚 Build #12904 succeeded 74f62f7
💚 Build #12781 succeeded e770f43
💔 Build #12760 failed 3ba3461
💚 Build #12754 succeeded eacd0a0
💚 Build #12749 succeeded fb7c118

cc @kush-elastic

elastic-sonarqube · 2024-06-26T11:02:57Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
97.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

milan-elastic

LGTM

ali786XI

Looks good

elasticmachine · 2024-07-11T10:47:44Z

Package salesforce - 0.15.0 containing this change is available at https://epr.elastic.co/search?package=salesforce

…a streams (#19717) * [salesforce] Re-expose Request Timeout (resource.timeout) for all data streams The Request Timeout setting was dropped in the v0.15.0 revamp (#9629) when the integration moved from the httpjson input to the dedicated salesforce Filebeat input. Since then there was no way to override the input's default 30s HTTP client timeout, so large EventLogFile downloads (notably Apex logs) could fail with "context deadline exceeded" with no available remedy. Re-expose it as a `resource_timeout` variable (default 30s, in Advanced options) across all four data streams (apex, login, logout, setupaudittrail), wiring it to `resource.timeout` in each stream template. Update the troubleshooting docs to reflect the current error signature and bump the package version to 1.8.0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [salesforce] Point changelog link to the PR Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

kush-elastic added 3 commits January 31, 2024 14:09

Initial commit:

ae7c502

- remove login_stream and logout_stream - update config for login, logout, apex and setupaudittrail data-stream - update configurations

Merge branch 'main' of https://github.com/kush-elastic/integrations i…

1006ac7

…nto 1278_revamp_salesforce_integration

Updated following:

cae52c9

- Update handlebar files for all the datastreams. - Update pipelines for all the datastreams - Update field mappings - update dashboards

kush-elastic requested a review from a team as a code owner April 17, 2024 06:32

Merge branches '1278_revamp_salesforce_integration' and 'main' of htt…

48dac43

…ps://github.com/kush-elastic/integrations into 1278_revamp_salesforce_integration

kush-elastic self-assigned this Apr 17, 2024

kush-elastic added breaking change Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] Integration:salesforce Salesforce labels Apr 17, 2024

shmsr added the enhancement New feature or request label Apr 17, 2024

kush-elastic added 5 commits April 30, 2024 12:40

add special condition for log_file_interval

dab8c37

update pipeline tests for apex

2707ae6

Pipeline Tests and Field Mappings

2a5ed62

update Login Object pipeline test

f51481c

Add Object pipeline tests and fix mappings

e0ae732

shmsr changed the title ~~integrations/packages/salesforce: Revamp Salesforce integration~~ May 7, 2024

fix hbs and manifest files

338bcc6

kush-elastic and others added 4 commits June 3, 2024 10:57

system tests

d2d511a

Improve docs

15763c4

Fix some bugs and descriptions

64c379d

Revert a small cahnge

698c670

shmsr reviewed Jun 11, 2024

View reviewed changes

Comment thread packages/salesforce/data_stream/apex/manifest.yml Outdated

shmsr requested a review from a team June 11, 2024 10:15

Fix bugs in login datastream

8c805a8

shmsr and others added 4 commits June 11, 2024 18:24

Make additional Info a more space optimized variant of text

2903846

Merge branch '1278_revamp_salesforce_integration' of https://github.c…

8826def

…om/kush-elastic/integrations into 1278_revamp_salesforce_integration

fix and add system tests for login and logout

426335e

Fix logout .hbs file

cffab2d

shmsr requested changes Jun 24, 2024

View reviewed changes

Comment thread packages/salesforce/manifest.yml Outdated

Comment thread packages/salesforce/manifest.yml Outdated

Comment thread packages/salesforce/manifest.yml Outdated

ali786XI and others added 4 commits June 24, 2024 12:27

fix some descriptions

e0636d4

Update packages/salesforce/manifest.yml

b812f77

Co-authored-by: subham sarkar <sarkar.subhams2@gmail.com>

Update packages/salesforce/manifest.yml

fb7c118

Co-authored-by: subham sarkar <sarkar.subhams2@gmail.com>

Update packages/salesforce/manifest.yml

eacd0a0

ali786XI reviewed Jun 24, 2024

View reviewed changes

Comment thread packages/salesforce/data_stream/login/fields/fields.yml Outdated

kush-elastic and others added 2 commits June 24, 2024 14:09

remove default values from client_id and client_secret

e2b7722

Update packages/salesforce/data_stream/login/fields/fields.yml

3ba3461

Co-authored-by: Aliabbas Attarwala <124054599+aliabbas-elastic@users.noreply.github.com>

shmsr reviewed Jun 24, 2024

View reviewed changes

fix descriptions

e770f43

kush-elastic requested review from ali786XI and shmsr June 25, 2024 06:16

shmsr approved these changes Jun 25, 2024

View reviewed changes

muthu-mps reviewed Jun 25, 2024

View reviewed changes

Comment thread packages/salesforce/data_stream/setupaudittrail/manifest.yml

update changelog entry

74f62f7

shmsr reviewed Jun 26, 2024

View reviewed changes

Comment thread packages/salesforce/changelog.yml Outdated

kush-elastic and others added 2 commits June 26, 2024 15:32

Update packages/salesforce/changelog.yml

7f32265

Co-authored-by: subham sarkar <sarkar.subhams2@gmail.com>

update readme with setup audit trail supported events longest interval

2636e80

muthu-mps approved these changes Jun 26, 2024

View reviewed changes

milan-elastic approved these changes Jun 26, 2024

View reviewed changes

ali786XI approved these changes Jun 26, 2024

View reviewed changes

kush-elastic merged commit 3177699 into elastic:main Jun 27, 2024

stefans-elastic mentioned this pull request Jun 23, 2026

[salesforce] Re-expose Request Timeout (resource.timeout) for all data streams #19717

Merged

5 tasks

Navigation Menu

Uh oh!

packages/salesforce: Revamp Salesforce integration - #9629

packages/salesforce: Revamp Salesforce integration#9629
kush-elastic merged 48 commits into
elastic:mainfrom
kush-elastic:1278_revamp_salesforce_integration

kush-elastic commented Apr 17, 2024 •

edited by shmsr

Loading

kush-elastic commented May 15, 2024

Uh oh!

shmsr commented Jun 11, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ali786XI commented Jun 24, 2024

shmsr Jun 24, 2024

Uh oh!

muthu-mps commented Jun 25, 2024 •

edited by shmsr

Loading

kush-elastic commented Jun 26, 2024

Uh oh!

elasticmachine commented Jun 26, 2024

elastic-sonarqube Bot commented Jun 26, 2024

milan-elastic left a comment

ali786XI left a comment

elasticmachine commented Jul 11, 2024

Labels

7 participants

Uh oh!

Conversation

kush-elastic commented Apr 17, 2024 • edited by shmsr Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Related issues

Screenshots

Configurations

kush-elastic commented May 15, 2024

Uh oh!

shmsr commented Jun 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ali786XI commented Jun 24, 2024

shmsr Jun 24, 2024

Choose a reason for hiding this comment

Uh oh!

muthu-mps commented Jun 25, 2024 • edited by shmsr Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

kush-elastic commented Jun 26, 2024

Uh oh!

elasticmachine commented Jun 26, 2024

💚 Build Succeeded

History

elastic-sonarqube Bot commented Jun 26, 2024

Quality Gate passed

milan-elastic left a comment

Choose a reason for hiding this comment

ali786XI left a comment

Choose a reason for hiding this comment

elasticmachine commented Jul 11, 2024

Labels

7 participants

kush-elastic commented Apr 17, 2024 •

edited by shmsr

Loading

shmsr commented Jun 11, 2024 •

edited

Loading

muthu-mps commented Jun 25, 2024 •

edited by shmsr

Loading