Updates to the ProblemChild and DGA integration packages by ajosh0504 · Pull Request #3777 · elastic/integrations

ajosh0504 · 2022-07-20T21:58:00Z

What does this PR do?

Fixes some bugs in/arising from the ProblemChild and DGA integrations packages.
Some changes to improve discoverability of rules associated with the two packages.

Changes made:

ProblemChild integration package
- Fix a script that was causing the inference pipeline to fail
- Added additional tags to rules associated with the integration package for easy discoverability
- Updated the query and influencers in problemchild-ml.json
- Updated all rule versions to 2 since this is an actual update to the rules (there were some rules which started with a version 2, but we can probably disregard that since this is still an experimental package)
DGA integration package
- Fixed a rule that was pointing to a non-existent ML job
- Added additional tags to rules associated with the integration package for easy discoverability
- Corrected the n-gram feature creation in dga_inference_pipeline script.
- Updated all rule versions to 2 since this is an actual update to the rules (there were some rules which started with a version 2, but we can probably disregard that since this is still an experimental package)

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Related issues

elasticmachine · 2022-07-20T22:10:32Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-08-10T21:28:35.803+0000
Duration: 16 min 37 sec

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

This reverts commit bf968cc.

ajosh0504 · 2022-07-20T22:58:09Z

@mtojek Does this update to the packages count as a new version or the same? I was hoping for this to be a sort of bug fix PR for the existing packages.

P.S. I just noticed that some of the rules in the initial packages have some version 2s- maybe we want to reset those to 1? I can take care of that in this PR too- not sure if this breaks stuff for users who have the packages currently installed? Then again we're calling these packages "experimental" so might be better to focus on fixing things regardless, for now.

@brokensound77 Any thoughts on this^

mtojek

Please check to make sure the structure of the data_stream directory in both the packages looks correct

Please make sure that CI passes for this PR. CI uses elastic-package tool to validate against the package-spec.

Is a manifest.yml file in the data_stream/predictions directory mandatory?

Manifest describes the data stream and if you want to have data stream it's mandatory. I see that you don't have much experience with package development, so you may find this blog post helpful.

What should it contain?

As I replied to you over Slack, the content is defined by package-spec. Any property that is "legal", must be defined by spec.

Since I am adding new index mappings to the packages, is there anything else required other than this integrations PR?

I don't know, not sure what are you trying to achieve. You've implemented an incomplete data stream - just added a few fields, and modified ingest pipeline and security rules. Please sync with @alvarezmelissa87 who has already successfully contributed to Integrations.

Does this update to the packages count as a new version or the same? I was hoping for this to be a sort of bug fix PR for the existing packages.

If you want to release those changes, you need to bump up the version and add a corresponding changelog entry. There is a background job set to determine if the package version has been updated and if so, it syncs changes with Package Storage.

ajosh0504 · 2022-07-21T22:23:20Z

@mtojek Thanks for the detailed response. Looks like CI is indeed failing on the missing manifests now.

I was reading your blog post but looks like the example you walk through is for a new integration, creating a new data stream. I'm working with an existing package and existing data streams, specifically the logs-* , winlogbeat-* and endgame-* data streams, based on which indices the user wants to run the ingest pipeline (defined in the top-level elasticsearch folder) on. It seems to me then that I should have the ability to define an index template asset in the top-level elasticsearch folder as well but that doesn't seem to be supported by package-spec yet.

Eitherway, I'll follow-up with @alvarezmelissa87 and see where we end up on this. Thanks!

ajosh0504 · 2022-08-02T15:14:39Z

Tested the packages with elastic-package to make sure that the bug fixes worked.

andrewkroh · 2022-08-02T16:36:01Z

-        if (ctx.containsKey('process') && ctx['process'].containsKey('Ext') && ctx['process']['Ext'].containsKey('code_signature') && ctx['process']['Ext']['code_signature'].containsKey('exists')) {
-          ctx.feature_process_signed = ctx['process']['Ext']['code_signature']['exists']
-        } else if (ctx.containsKey('process') && ctx['process'].containsKey('code_signature') && ctx['process']['code_signature'].containsKey('exists')) {
+        if (ctx.containsKey('process') && ctx['process'].containsKey('code_signature') && ctx['process']['code_signature'].containsKey('exists')) {


This can safely be expressed as ctx.process?.code_signature?.exists != null using the null-safe operator.

We could cover this in a follow-up PR, since this PR only deals with bug fixes?

alvarezmelissa87 · 2022-08-02T17:22:51Z

+                "minimum_should_match": 1,
                "should": [
                    {
-                        "bool": {


Could you explain why this filter isn't needed anymore?

The original query basically translated to checking whether the fields ["process.name", "host.hostname", "process.command_line"] were present OR if the problemchild.prediction_probability field was present.

This is not what we want to check for, I don't think. We want to check if the events being passed to the anomaly detection jobs either have a problemchild. prediction or a blocklist_label, which is what I changed the query to.

As an aside, I also cross-checked with DGA, and the query there seems to match what's in the datafeeds as well, which is basically what I've done.

Is that not what the query should reflect?

alvarezmelissa87 · 2022-08-02T17:29:34Z

Overall LGTM. Re: the questions in the description - I don't see any data_stream changes in this PR so maybe these comments are out of date?

ajosh0504 · 2022-08-02T21:56:54Z

Overall LGTM. Re: the questions in the description - I don't see any data_stream changes in this PR so maybe these comments are out of date?

Right, updated the description.

ajosh0504 · 2022-08-02T22:23:23Z

@alvarezmelissa87 Here's screenshots to prove that things are working as expected:

DGA

Making sure that the right version is being used for testing:

This job was previously referring to the ML job experimental-high-sum-dga-probability which did not exist. Removed reference to that job and updated the rule with the correct name:

DGA tag allows filtering on all DGA-related rules

ProblemChild

Making sure that the right version is being used for testing:

Predictions were previously not coming through due to a bug in the inference pipeline. Seeing predictions on Windows process events now:

Anomaly Detection job datafeed was previously failing to start due to mapping errors. Added mappings to the component template (confirmed with @andrewkroh that this is the way to add custom mappings to existing datafeeds) and not seeing that error anymore. The warning you see here is because I don't have any malicious data on my cluster and the job only runs on malicious data:

LotL tag allows filtering for rules related to the LotL Attack Detection package:

alvarezmelissa87

LGTM ⚡

szeitlin · 2022-08-09T17:22:50Z

+            "DGA",
+            "ML",
+            "Supervised",
+            "Anomalies"


how is Anomalies meant to be used? Seems kinda generic?
same for ML vs. Supervised - are these intended for internal use, or to be exposed to the customer?

We want to expose Anomalies vs Advanced Threats in the centralized Entity Analytics page. Anomalies would be things flagged as suspicious by individual ML jobs. Advanced Threats are things flagged as suspicious by more than one method.

For example, in ProblemChild, suspicious events are identified by a supervised model and an anomaly detection job.

ML is a generic tag for all ML jobs in the Security solution. Supervised vs Unsupervised is based on which model contributed to the event being flagged as anomalous. Now that I think about it, this might be too much information for the user.

This helps a lot, thank you!

I think maybe it would help to write this up in a separate doc (since it might be hard to track back to this ticket later)

ok, discussed with Sourin and put a couple comments in the spreadsheet, but the short version is that as far as the product is concerned, we only need 3 kinds of tags:

package tags - what rules and dashboards are tied to a package

threat category (internal vs. external threat) for future product features

entity tag (user vs. host)

For the others, if we want internal tags for our own tracking/packaging/debugging/support purposes, that's good to know, just want to make sure we're keeping an eye on that as we along, so we don't end up with a proliferation of unused/overlapping/incorrect tag tech debt like we have on the malware pipelines.

szeitlin · 2022-08-09T17:24:05Z

            "ML",
-            "DGA"
+            "Unsupervised",
+            "Advanced Threats"


what constitutes an Advanced Threat? Do we have a list of what's included in those? (also, why am I having deja vu as I write this?)

Refer to the comment above.

* Fixing some bugs, adding mappings, tags etc. * Updating version numbers * Revert "Updating version numbers" This reverts commit 9b74911. * Formatting yml files * Updating changelog and version * Update ml_dga_inference_pipeline.yml * Updated the problemchild query * Deleting data_stream folder since it's not necessary * Updating rule numbers * Updating changelog with specifics of what bugs were fixed * Updating detector descriptions and changelog * formatting changelog * Adding better tags to rules, influencers where necessary * Finalizing rule tags * Adding missing influencers Co-authored-by: Wei Wang <47184485+wwang500@users.noreply.github.com>

Fixing some bugs, adding mappings, tags etc.

660dbc1

ajosh0504 added the bug Something isn't working, use only for issues label Jul 20, 2022

ajosh0504 requested a review from mtojek July 20, 2022 21:58

ajosh0504 requested a review from a team as a code owner July 20, 2022 21:58

ajosh0504 requested a review from alvarezmelissa87 July 20, 2022 21:58

ajosh0504 added 2 commits July 20, 2022 15:24

Updating version numbers

bf968cc

Revert "Updating version numbers"

ad77b2d

This reverts commit bf968cc.

mtojek reviewed Jul 21, 2022

View reviewed changes

Formatting yml files

34d7f64

ajosh0504 and others added 2 commits July 26, 2022 14:18

Updating changelog and version

d960ed3

Update ml_dga_inference_pipeline.yml

ef17d3b

wwang500 mentioned this pull request Jul 28, 2022

[DGA package] Update painless script to add missing n-grams fields #3861

Closed

4 tasks

ajosh0504 added 3 commits July 28, 2022 11:34

Updated the problemchild query

c36e69f

Deleting data_stream folder since it's not necessary

5a4f9ad

Updating rule numbers

569a3d9

ajosh0504 requested a review from andrewkroh July 29, 2022 14:31

ajosh0504 changed the title ~~[DRAFT] Updates to the ProblemChild and DGA integration packages~~ Aug 2, 2022

andrewkroh reviewed Aug 2, 2022

View reviewed changes

alvarezmelissa87 reviewed Aug 2, 2022

View reviewed changes

Comment thread packages/problemchild/changelog.yml Outdated

alvarezmelissa87 reviewed Aug 2, 2022

View reviewed changes

Updating changelog with specifics of what bugs were fixed

adfb900

Updating detector descriptions and changelog

2fb8bf0

formatting changelog

a48c264

alvarezmelissa87 approved these changes Aug 3, 2022

View reviewed changes

Adding better tags to rules, influencers where necessary

46e4541

szeitlin reviewed Aug 9, 2022

View reviewed changes

ajosh0504 added 2 commits August 10, 2022 10:14

Finalizing rule tags

2ca1751

Adding missing influencers

b1e384e

ajosh0504 merged commit 8898ca0 into main Aug 10, 2022

ajosh0504 mentioned this pull request Aug 15, 2022

Fixes the ML jobs query for the LotL Attack Detection Package #3999

Merged

4 tasks

github-actions Bot mentioned this pull request May 7, 2026

[Slack] New users data stream for entity support #18882

Open

Navigation Menu

Uh oh!

Updates to the ProblemChild and DGA integration packages - #3777

Updates to the ProblemChild and DGA integration packages#3777
ajosh0504 merged 15 commits into
mainfrom
problemchild_dga_update

ajosh0504 commented Jul 20, 2022 •

edited

Loading

elasticmachine commented Jul 20, 2022 •

edited

Loading

Build stats

ajosh0504 commented Jul 20, 2022 •

edited

Loading

mtojek left a comment •

edited

Loading

ajosh0504 commented Jul 21, 2022 •

edited

Loading

ajosh0504 commented Aug 2, 2022

Uh oh!

andrewkroh Aug 2, 2022

ajosh0504 Aug 2, 2022

Uh oh!

alvarezmelissa87 Aug 2, 2022

ajosh0504 Aug 2, 2022 •

edited

Loading

ajosh0504 Aug 2, 2022

alvarezmelissa87 commented Aug 2, 2022

ajosh0504 commented Aug 2, 2022

ajosh0504 commented Aug 2, 2022 •

edited

Loading

alvarezmelissa87 left a comment

szeitlin Aug 9, 2022 •

edited

Loading

ajosh0504 Aug 9, 2022

szeitlin Aug 9, 2022

szeitlin Aug 9, 2022

szeitlin Aug 9, 2022

ajosh0504 Aug 9, 2022

Labels

7 participants

Uh oh!

Conversation

ajosh0504 commented Jul 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Changes made:

Checklist

Related issues

elasticmachine commented Jul 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

🤖 GitHub comments

ajosh0504 commented Jul 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

mtojek left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

ajosh0504 commented Jul 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ajosh0504 commented Aug 2, 2022

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

ajosh0504 Aug 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alvarezmelissa87 commented Aug 2, 2022

ajosh0504 commented Aug 2, 2022

ajosh0504 commented Aug 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

alvarezmelissa87 left a comment

Choose a reason for hiding this comment

szeitlin Aug 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Labels

7 participants

ajosh0504 commented Jul 20, 2022 •

edited

Loading

elasticmachine commented Jul 20, 2022 •

edited

Loading

ajosh0504 commented Jul 20, 2022 •

edited

Loading

mtojek left a comment •

edited

Loading

ajosh0504 commented Jul 21, 2022 •

edited

Loading

ajosh0504 Aug 2, 2022 •

edited

Loading

ajosh0504 commented Aug 2, 2022 •

edited

Loading

szeitlin Aug 9, 2022 •

edited

Loading