Improve DGA/LotL doc's Installation Procedures

[rseldner]

Proposed commit message

The Living off the Land integration's installation procedure could use these improvements:

1. Noted that one of the listed ingest pipeline methods should be used
2. added a missing mapping step that is required for the ML job card to appear as an option (as described here: https://www.elastic.co/security-labs/detecting-living-off-the-land-attacks-with-new-elastic-integration)
3. added optional step of creating a data view that is specific to the windows process data (rather than selecting logs-* which hits impertinent indices)
4. noted that the ML job card can also be missing if the mapping is not in place

Ask for Reviewers:

• - evaluate these doc changes are appropriate

[jmcarlock]

@rseldner Thank you for your contribution to this documentation!

I do not believe the index mapping changes are still necessary as they now are handled in custom.yml

Also, when would a user need to use the Rollover API?

[rseldner]

I found the mapping was needed while assisting a user in troubleshooting the missing ML job card despite adding the ingest pipeline and confirming the problemchild fields had been enriched.

I reproduced this myself and can demonstrate. I will reach out internally.

The rollover would be needed after defining the mapping in the index/component template so that the subsequent index picks up the mapping.

[jmcarlock]

@rseldner Can you confirm the version of Kibana?

[rseldner]

User was on Elastic's Govcloud 8.13.4
I tested on ESS deployments running 8.13.4 and 8.14.2

[willemdh]

Please also add what data from what Agent is required. I once lost time testing this while it needs Agent events to work? Or has that changed since

[susan-shu-c]

Thank you for the PR, @jmcarlock on our team has been investigating some related details, and will provide more feedback + keep you posted

[jmcarlock]

@rseldner , I changed the instructions to runtime mappings, which doesn't require a rollover. Can you give it one more test on your end?

We also needed to make the changes for DGA, which had the same issue. I also bumped the version number, added a changelog entry, and tested both with elastic-package and with cloud deployments.

[sodhikirti07]

New instructions to add Mappings lgtm!

[rseldner]

At least in the current state where these fields are not getting mapped, We are missing the point that if the mappings do not exist (runtime or not), they need to be added to the relevant index/component template. Doing PUT INDEX_NAME*/_mapping will solely apply it to the specified index.
If using time series data (which I imagine everyone would be for this integration...), then the index will eventually rollover, and the new resulting index will not contain these mappings. If not defined in the template, PUT INDEX_NAME*/_mapping would be needed every time there's a rollover.

Adding runtime mappings to an index do not get around this and do not quite make sense to me (if we are adding a mapping to an index, why make it a runtime mapping? We prefer query overhead v.s. storage overhead?)

Unless you meant to instead apply a runtime field on the data view to emit these field values from _source. In which case, I believe that might work; but that is a separate step to perform within Kibana UI or Kibana API.

Sidenote, shouldn't prediction_probability be float? I see that got changed from my original commit.

          "problemchild": {
            "prediction": "0",
            "prediction_probability": 0.9999999537601167
          }

per @rseldner, will require additional changes

[jmcarlock]

@rseldner , ok, I understand! Thank you for the explanation. I will change the method back to your suggestions and test again.

Will also change prediction_probability to the correct data type.

[jmcarlock]

Previews of the documentation changes can be viewed at:

• https://github.com/elastic/integrations/blob/rseldner-patch-1/packages/dga/docs/README.md
• https://github.com/elastic/integrations/blob/rseldner-patch-1/packages/problemchild/docs/README.md

[jmcarlock]

Thank you for the review, @susan-shu-c !

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: e2e8fe4

History

• 💚 Build #14058 succeeded 2c9020d
• 💔 Build #13941 failed 04b091d
• 💚 Build #13894 succeeded 4473585
• 💚 Build #13878 succeeded 9bb3f1a
• 💚 Build #13876 succeeded ba07644
• 💚 Build #13873 succeeded 49963ce

cc @jmcarlock

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[sodhikirti07]

@jmcarlock Nicely done! LGTM!

[susan-shu-c]

Thanks all for the contributions, and Gus for the research and testing!

[elasticmachine]

Package dga - 2.0.3 containing this change is available at https://epr.elastic.co/search?package=dga

[elasticmachine]

Package problemchild - 2.1.4 containing this change is available at https://epr.elastic.co/search?package=problemchild

[rseldner]

☝️ ➕ Thanks for your attention on this @jmcarlock !