Skip to content

[ProblemChild] Bug in Ingest Pipeline for blocklist_label String/Integer - #7577

Closed
MakoWish wants to merge 5 commits into
elastic:mainfrom
MakoWish:bug_problemchild_pipeline_string_value
Closed

[ProblemChild] Bug in Ingest Pipeline for blocklist_label String/Integer#7577
MakoWish wants to merge 5 commits into
elastic:mainfrom
MakoWish:bug_problemchild_pipeline_string_value

Conversation

@MakoWish

@MakoWish MakoWish commented Aug 28, 2023

Copy link
Copy Markdown
Contributor

Type of change

  • Bug

What does this PR do?

This PR adds a convert processor to ensure the problemchild.prediction value is an integer before comparing to an integer value.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have invremented the version in my package's manifest.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Related issues

@MakoWish
MakoWish marked this pull request as ready for review August 28, 2023 19:33
@MakoWish
MakoWish requested review from a team as code owners August 28, 2023 19:33
@elasticmachine

elasticmachine commented Aug 28, 2023

Copy link
Copy Markdown

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-08-29T17:25:29.615+0000

  • Duration: 14 min 22 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@alvarezmelissa87 alvarezmelissa87 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code LGTM

@alvarezmelissa87

alvarezmelissa87 commented Aug 29, 2023

Copy link
Copy Markdown
Contributor

@MakoWish - Could you confirm this has been tested?
@ajosh0504 - would love your eyes on this as well when you get a chance 🙏

@MakoWish

Copy link
Copy Markdown
Contributor Author

@MakoWish - Could you confirm this has been tested?

This has been tested. I was not getting any matches at all for the blocklist_label, and I thought something was strange about that. After looking into it, I threw a few test documents at the pipeline using _simulate, and sure enough, the problemchild.prediction value was coming through as a string value. I tested again with _simulate using the convert processor, and it was now properly flagging the document with blocklist_label: 1. I made the change on the production pipeline, and I am now getting the matches I was expecting from the start.

@susan-shu-c susan-shu-c left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes sense to me re: testing it, but will confirm with Apoorva when she's back!

@ajosh0504

Copy link
Copy Markdown
Contributor

@MakoWish Here's the blog that has all the details of the package installation steps. Step 2 under the Getting Started section states the mappings that need to be added to your index, which includes setting problemchild.prediction as long, which should solve the problem being addressed here.

I did, however, notice that we have linked to the wrong blog in the package docs, so if anything, we should update that link with the following so folks don't run into this in the future: https://www.elastic.co/security-labs/detecting-living-off-the-land-attacks-with-new-elastic-integration

If you want to change this PR to update that link instead, that'd be great, otherwise we'll cover it in a follow-up PR.

@MakoWish

Copy link
Copy Markdown
Contributor Author

@MakoWish Here's the blog that has all the details of the package installation steps. Step 2 under the Getting Started section states the mappings that need to be added to your index, which includes setting problemchild.prediction as long, which should solve the problem being addressed here.

I have the mappings set already, but that unfortunately does not resolve the issue. The mappings ensure the value is indexed correctly, but the problem is within the "inference" ingest pipeline itself (before indexing). The pipeline creates a string value of "1" and then checks for an integer value of 1, so there will never be a match. This can be demonstrated with _simulate:

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "script": {
          "lang": "painless",
          "source": """if (ctx.containsKey('process') && ctx['process'].containsKey('name') && !ctx['process']['name'].empty) {
  ctx.feature_process_name = ctx['process']['name']
  } else {
  ctx.feature_process_name = 'unknown'
  }
  if (ctx.containsKey('process') && ctx['process'].containsKey('parent') && ctx['process']['parent'].containsKey('name') && !ctx['process']['parent']['name'].empty) {
  ctx.feature_process_parent_name = ctx['process']['parent']['name']
  } else {
  ctx.feature_process_parent_name = 'unknown'
  }
  if (ctx.containsKey('process') && ctx['process'].containsKey('command_line') && !ctx['process']['command_line'].empty) {
  ctx.feature_command_line = ctx['process']['command_line']
  } else if (ctx.containsKey('endgame') && ctx['endgame'].containsKey('command_line') && !ctx['endgame']['command_line'].empty) {
  ctx.feature_command_line = ctx['endgame']['command_line']
  } else if (ctx.containsKey('process') && ctx['process'].containsKey('args') && !ctx['process']['args'].empty) {
  ctx.feature_command_line = ctx['process']['args'].join(" ")
  } else {
  ctx.feature_command_line = 'unknown'
  }
  if (ctx.containsKey('event') && ctx['event'].containsKey('action') && !ctx['event']['action'].empty) {
  ctx.feature_event_action = ctx['event']['action']
  } else {
  ctx.feature_event_action = 'unknown'
  }
  if (ctx.containsKey('process') && ctx['process'].containsKey('executable') && !ctx['process']['executable'].empty) {
  ctx.feature_process_executable = ctx['process']['executable']
  } else {
  ctx.feature_process_executable = 'unknown'
  }
  if (ctx.containsKey('process') && ctx['process'].containsKey('parent') && ctx['process']['parent'].containsKey('executable') && !ctx['process']['parent']['executable'].empty) {
  ctx.feature_process_parent_executable = ctx['process']['parent']['executable']
  } else {
  ctx.feature_process_parent_executable = 'unknown'
  }
  if (ctx.containsKey('process') && ctx['process'].containsKey('Ext') && ctx['process']['Ext'].containsKey('token') && ctx['process']['Ext']['token'].containsKey('elevation_type') && !ctx['process']['Ext']['token']['elevation_type'].empty) {
  ctx.feature_elevation_type = ctx['process']['Ext']['token']['elevation_type']
  } else if (ctx.containsKey('endgame') && ctx['endgame'].containsKey('elevation_type') && !ctx['endgame']['elevation_type'].empty) {
  ctx.feature_elevation_type = ctx['endgame']['elevation_type']
  } else if (ctx.containsKey('winlog') && ctx['winlog'].containsKey('event_data') && ctx['winlog']['event_data'].containsKey('TokenElevationType') && !ctx['winlog']['event_data']['TokenElevationType'].empty) {
  ctx.feature_elevation_type = ctx['winlog']['event_data']['TokenElevationType']
  } else {
  ctx.feature_elevation_type = 'unknown'
  }
  if (ctx.containsKey('process') && ctx['process'].containsKey('Ext') && ctx['process']['Ext'].containsKey('token') && ctx['process']['Ext']['token'].containsKey('integrity_level_name') && !ctx['process']['Ext']['token']['integrity_level_name'].empty) {
  ctx.feature_integrity_level = ctx['process']['Ext']['token']['integrity_level_name']
  } else if (ctx.containsKey('endgame') && ctx['endgame'].containsKey('integrity_level') && !ctx['endgame']['integrity_level'].empty) {
  ctx.feature_integrity_level = ctx['endgame']['integrity_level']
  } else if (ctx.containsKey('winlog') && ctx['winlog'].containsKey('event_data') && ctx['winlog']['event_data'].containsKey('IntegrityLevel') && !ctx['winlog']['event_data']['IntegrityLevel'].empty) {
  ctx.feature_integrity_level = ctx['winlog']['event_data']['IntegrityLevel']
  } else {
  ctx.feature_integrity_level = 'unknown'
  }
  if (ctx.containsKey('process') && ctx['process'].containsKey('pe') && ctx['process']['pe'].containsKey('original_file_name') && !ctx['process']['pe']['original_file_name'].empty) {
  ctx.feature_process_pe_original_file_name = ctx['process']['pe']['original_file_name']
  } else if (ctx.containsKey('winlog') && ctx['winlog'].containsKey('event_data') && ctx['winlog']['event_data'].containsKey('OriginalFileName') && !ctx['winlog']['event_data']['OriginalFileName'].empty) {
  ctx.feature_process_pe_original_file_name = ctx['winlog']['event_data']['OriginalFileName']
  } else {
  ctx.feature_process_pe_original_file_name = 'unknown'
  }
  if (ctx.containsKey('user') && ctx['user'].containsKey('id') && !ctx['user']['id'].empty) {
  ctx.feature_user_id = ctx['user']['id']
  } else if (ctx.containsKey('winlog') && ctx['winlog'].containsKey('user') && ctx['winlog']['user'].containsKey('identifier') && !ctx['winlog']['user']['identifier'].empty) {
  ctx.feature_user_id = ctx['winlog']['user']['identifier']
  } else {
  ctx.feature_user_id = 'unknown'
  }
  if (ctx.containsKey('process') && ctx['process'].containsKey('Ext') && ctx['process']['Ext'].containsKey('token') && ctx['process']['Ext']['token'].containsKey('elevation')) {
  ctx.feature_elevated = ctx['process']['Ext']['token']['elevation']
  } else if (ctx.containsKey('endgame') && ctx['endgame'].containsKey('elevated')) {
  ctx.feature_elevated = ctx['endgame']['elevated']
  } else if (ctx.containsKey('winlog') && ctx['winlog'].containsKey('event_data') && ctx['winlog']['event_data'].containsKey('TokenElevationType') && !ctx['winlog']['event_data']['TokenElevationType'].empty) {
  ctx.feature_elevated = true;
  } else {
  ctx.feature_elevated = false
  }
  if (ctx.containsKey('process') && ctx['process'].containsKey('code_signature') && ctx['process']['code_signature'].containsKey('exists')) {
  ctx.feature_process_signed = ctx['process']['code_signature']['exists']
  } else if (ctx.containsKey('winlog') && ctx['winlog'].containsKey('event_data') && ctx['winlog']['event_data'].containsKey('Signed') && !ctx['winlog']['event_data']['Signed'].empty) {
  ctx.feature_process_signed = Boolean.parseBoolean(ctx['winlog']['event_data']['Signed'])
  } else {
  ctx.feature_process_signed = false
  }
  if (ctx.containsKey('process') && ctx['process'].containsKey('code_signature') && ctx['process']['code_signature'].containsKey('trusted')) {
  ctx.feature_process_signer_trusted = ctx['process']['code_signature']['trusted']
  } else if (ctx.containsKey('winlog') && ctx['winlog'].containsKey('event_data') && ctx['winlog']['event_data'].containsKey('SignatureStatus') && ctx['winlog']['event_data']['SignatureStatus'] == "Valid") {
  ctx.feature_process_signer_trusted = true
  } else {
  ctx.feature_process_signer_trusted = false
  }"""
        }
      },
      {
        "script": {
          "lang": "painless",
          "source": "if(ctx.feature_process_name != ctx.feature_process_pe_original_file_name) {ctx.feature_filename_mismatch = true} else {ctx.feature_filename_mismatch = false}"
        }
      },
      {
        "script": {
          "lang": "painless",
          "params": {
            "win_system_sids": [
              "S-1-5-18",
              "S-1-5-19",
              "S-1-5-20"
            ]
          },
          "source": "if(params.win_system_sids.contains(ctx.feature_user_id)) {ctx.feature_running_as_system = true} else {ctx.feature_running_as_system = false}"
        }
      },
      {
        "script": {
          "lang": "painless",
          "source": """if(ctx.feature_process_name.contains(".exe")) {ctx.feature_ends_with_exe = true} else {ctx.feature_ends_with_exe = false}"""
        }
      },
      {
        "lowercase": {
          "field": "feature_command_line"
        }
      },
      {
        "lowercase": {
          "field": "feature_process_executable"
        }
      },
      {
        "lowercase": {
          "field": "feature_process_parent_executable"
        }
      },
      {
        "script": {
          "lang": "painless",
          "source": """if(ctx.feature_process_executable =~ /:\\windows\\system32/i) {
    ctx.feature_normalized_ppath = "win_system_dir"
  } else if(ctx.feature_process_executable =~ /:\\windows\\syswow64/i) {
    ctx.feature_normalized_ppath = "win_system_dir"
  } else if(ctx.feature_process_executable =~ /:\\windows\\temp/i) {
    ctx.feature_normalized_ppath = "win_temp_dir"
  } else if(ctx.feature_process_executable =~ /(:\\users\\)(.*?)(\\appdata\\local\\temp)/i) {
    ctx.feature_normalized_ppath = "win_user_appdata_local_temp"
  } else if(ctx.feature_process_executable =~ /(:\\users\\)(.*?)(\\appdata)/i) {
    ctx.feature_normalized_ppath = "win_user_appdata"
  } else if(ctx.feature_process_executable =~ /(:\\users\\)(.*?)(\\downloads)/i) {
    ctx.feature_normalized_ppath = "win_user_downloads"
  } else if(ctx.feature_process_executable =~ /(:\\users\\)(.*?)(\\)/i) {
    ctx.feature_normalized_ppath = "win_user_dir"
  } else if(ctx.feature_process_executable =~ /:\\users$/i) {
    ctx.feature_normalized_ppath = "win_users_root_dir"
  } else if(ctx.feature_process_executable =~ /:\\program files/i) {
    ctx.feature_normalized_ppath = "win_program_files"
  } else if(ctx.feature_process_executable =~ /:\\programdata/i) {
    ctx.feature_normalized_ppath = "win_program_data_dir"
  } else if(ctx.feature_process_executable =~ /:\\system volume information/i) {
    ctx.feature_normalized_ppath = "win_sys_vol_info_dir"
  } else if(ctx.feature_process_executable =~ /:\\\$recycle\.bin/i) {
    ctx.feature_normalized_ppath = "win_recycle_bin_dir"
  } else if(ctx.feature_process_executable =~ /:\\windows/i) {
    ctx.feature_normalized_ppath = "win_dir"
  } else if(ctx.feature_process_executable =~ /[a-z]:\\$/i) {
    ctx.feature_normalized_ppath = "win_root"
  } else {
  ctx.feature_normalized_ppath = "unknown"
  }
  """
        }
      },
      {
        "gsub": {
          "field": "feature_command_line",
          "pattern": "\"",
          "replacement": ""
        }
      },
      {
        "gsub": {
          "field": "feature_process_executable",
          "pattern": "\"",
          "replacement": ""
        }
      },
      {
        "gsub": {
          "field": "feature_process_parent_executable",
          "pattern": "\"",
          "replacement": ""
        }
      },
      {
        "gsub": {
          "field": "feature_command_line",
          "pattern": """\\{2,}""",
          "replacement": """\\"""
        }
      },
      {
        "gsub": {
          "field": "feature_process_executable",
          "pattern": """\\{2,}""",
          "replacement": """\\"""
        }
      },
      {
        "gsub": {
          "field": "feature_process_parent_executable",
          "pattern": """\\{2,}""",
          "replacement": """\\"""
        }
      },
      {
        "gsub": {
          "field": "feature_command_line",
          "pattern": """(users)\\([^\\]+)""",
          "replacement": "$1"
        }
      },
      {
        "gsub": {
          "field": "feature_process_executable",
          "pattern": """(users)\\([^\\]+)""",
          "replacement": "$1"
        }
      },
      {
        "gsub": {
          "field": "feature_process_parent_executable",
          "pattern": """(users)\\([^\\]+)""",
          "replacement": "$1"
        }
      },
      {
        "gsub": {
          "field": "feature_command_line",
          "pattern": """windows\\system32|windows/system32|windows\\syswow64|windows/syswow64""",
          "replacement": "win_system_dir"
        }
      },
      {
        "gsub": {
          "field": "feature_process_executable",
          "pattern": """windows\\system32|windows/system32|windows\\syswow64|windows/syswow64""",
          "replacement": "win_system_dir"
        }
      },
      {
        "gsub": {
          "field": "feature_process_parent_executable",
          "pattern": """windows\\system32|windows/system32|windows\\syswow64|windows/syswow64""",
          "replacement": "win_system_dir"
        }
      },
      {
        "gsub": {
          "field": "feature_command_line",
          "pattern": "[0-9a-f]{4,}-[0-9a-f]{4,}-[0-9a-f]{4,}-[0-9a-f-]{4,}",
          "replacement": "process_id"
        }
      },
      {
        "gsub": {
          "field": "feature_command_line",
          "pattern": "0x[0-9a-f]+|[0-9a-f]{8}",
          "replacement": "hex"
        }
      },
      {
        "gsub": {
          "field": "feature_command_line",
          "pattern": "[a-f0-9]{128}|[a-f0-9]{64}|[a-f0-9]{56}|[a-f0-9]{40}|[a-f0-9]{32}",
          "replacement": "hash"
        }
      },
      {
        "gsub": {
          "field": "feature_command_line",
          "pattern": "[0-9]{5,}",
          "replacement": "num"
        }
      },
      {
        "script": {
          "lang": "painless",
          "source": """String nGramAtPosition(String field, int fieldcount, int n) {
  if (fieldcount+n>field.length()) {
    return ''
  } else {
  return field.substring(fieldcount, fieldcount+n)
  }
  }
  String fieldtext = ctx[params.field];
  if(fieldtext.length()>=params.max_length) {
  fieldtext = fieldtext.substring(0, params.max_length);
  } else {
  fieldtext = fieldtext;
  }
  for (int i=0;i<(fieldtext.length());i++) {
  ctx[params.field+'_'+Integer.toString(params.ngram_count)+'-gram_feature'+Integer.toString(i)] = nGramAtPosition(fieldtext, i, params.ngram_count)
  }""",
          "params": {
            "field": "feature_process_name",
            "max_length": 100,
            "ngram_count": 2
          }
        }
      },
      {
        "script": {
          "lang": "painless",
          "source": """String nGramAtPosition(String field, int fieldcount, int n) {
  if (fieldcount+n>field.length()) {
    return ''
  } else {
  return field.substring(fieldcount, fieldcount+n)
  }
  }
  String fieldtext = ctx[params.field];
  if(fieldtext.length()>=params.max_length) {
  fieldtext = fieldtext.substring(0, params.max_length);
  } else {
  fieldtext = fieldtext;
  }
  for (int i=0;i<(fieldtext.length());i++) {
  ctx[params.field+'_'+Integer.toString(params.ngram_count)+'-gram_feature'+Integer.toString(i)] = nGramAtPosition(fieldtext, i, params.ngram_count)
  }""",
          "params": {
            "field": "feature_process_parent_name",
            "max_length": 100,
            "ngram_count": 2
          }
        }
      },
      {
        "script": {
          "lang": "painless",
          "source": """String nGramAtPosition(String field, int fieldcount, int n) {
  if (fieldcount+n>field.length()) {
    return ''
  } else {
  return field.substring(fieldcount, fieldcount+n)
  }
  }
  String fieldtext = ctx[params.field];
  if(fieldtext.length()>=params.max_length) {
  fieldtext = fieldtext.substring(0, params.max_length);
  } else {
  fieldtext = fieldtext;
  }
  for (int i=0;i<(fieldtext.length());i++) {
  ctx[params.field+'_'+Integer.toString(params.ngram_count)+'-gram_feature'+Integer.toString(i)] = nGramAtPosition(fieldtext, i, params.ngram_count)
  }""",
          "params": {
            "field": "feature_process_executable",
            "max_length": 100,
            "ngram_count": 2
          }
        }
      },
      {
        "script": {
          "lang": "painless",
          "source": """String nGramAtPosition(String field, int fieldcount, int n) {
  if (fieldcount+n>field.length()) {
    return ''
  } else {
  return field.substring(fieldcount, fieldcount+n)
  }
  }
  String fieldtext = ctx[params.field];
  if(fieldtext.length()>=params.max_length) {
  fieldtext = fieldtext.substring(0, params.max_length);
  } else {
  fieldtext = fieldtext;
  }
  for (int i=0;i<(fieldtext.length());i++) {
  ctx[params.field+'_'+Integer.toString(params.ngram_count)+'-gram_feature'+Integer.toString(i)] = nGramAtPosition(fieldtext, i, params.ngram_count)
  }""",
          "params": {
            "field": "feature_process_parent_executable",
            "max_length": 100,
            "ngram_count": 2
          }
        }
      },
      {
        "script": {
          "lang": "painless",
          "source": """String nGramAtPosition(String field, int fieldcount, int n) {
  if (fieldcount+n>field.length()) {
    return ''
  } else {
  return field.substring(fieldcount, fieldcount+n)
  }
  }
  String fieldtext = ctx[params.field];
  if(fieldtext.length()>=params.max_length) {
  fieldtext = fieldtext.substring(0, params.max_length);
  } else {
  fieldtext = fieldtext;
  }
  for (int i=0;i<(fieldtext.length());i++) {
  ctx[params.field+'_'+Integer.toString(params.ngram_count)+'-gram_feature'+Integer.toString(i)] = nGramAtPosition(fieldtext, i, params.ngram_count)
  }""",
          "params": {
            "field": "feature_command_line",
            "max_length": 100,
            "ngram_count": 2
          }
        }
      },
      {
        "inference": {
          "field_map": {},
          "inference_config": {
            "classification": {
              "num_top_classes": 0,
              "num_top_feature_importance_values": 0,
              "results_field": "prediction"
            }
          },
          "model_id": "problemchild_20210526_1.0",
          "target_field": "problemchild"
        }
      },
      {
        "script": {
          "lang": "painless",
          "source": """for(item in params.blocklist){
  if(ctx['feature_command_line'].contains(item)){
    ctx.blocklist_label = 1
  }
  }
  """,
          "if": "ctx.containsKey('problemchild') && ctx['problemchild'].containsKey('prediction') && ctx['problemchild']['prediction'] == 0",
          "params": {
            "blocklist": [
              "dump",
              "-embedding",
              "privilege",
              "administrator",
              "password",
              "key",
              "shadowcopy",
              "delete",
              "masquerade",
              "evil",
              "bad",
              "anonymous",
              "allprofiles",
              ".hta",
              "createobject",
              "target",
              "suspicious",
              "obfuscated",
              "bypass",
              "bootstatuspolicy",
              "recoveryenabled",
              "systemstatebackup",
              "comspec",
              "base64string",
              ".text.encoding",
              ".compression.",
              "memorystream",
              "writeallbytes",
              "webclient",
              "downloadfile",
              "downloadstring",
              "bitstransfer",
              "invoke-exp",
              "invoke-web",
              "reflection.assembly",
              "assembly.gettype",
              "sockets"
            ]
          }
        }
      },
      {
        "script": {
          "lang": "painless",
          "source": "ctx.entrySet().removeIf(field -> field.getKey() =~ /feature_.*/);ctx['problemchild'].remove('prediction_score');ctx['problemchild'].remove('model_id');"
        }
      }
    ],
    "on_failure": [
      {
        "append": {
          "field": "event.kind",
          "value": "pipeline_error"
        }
      },
      {
        "append": {
          "field": "error.message",
          "value": "{{{ _ingest.on_failure_message }}}"
        }
      }
    ],
    "_meta": {
      "managed_by": "fleet",
      "managed": true,
      "package": {
        "name": "problemchild"
      }
    }
  },
  "docs": [
    {
      "_source": {
        "process": {
          "command_line": "C:\\Program Files (x86)\\foo\\bar.exe -Embedding"
        }
      }
    }
  ]
}

You will see the resulting document with problemchild.prediction: "1" indicating a string value of "1", and blocklist_label is not set. Add the convert processor that I am suggesting, then run the same _simulate again, and you will see problemchild.prediction: 1 is now an integer value, and blocklist_label is now correctly set.

@ajosh0504

Copy link
Copy Markdown
Contributor

@MakoWish Ah I see what you're saying. An alternative to adding an additional processor to the pipeline could also be to change the if check in the blocklist script processor to compare with "0" instead, or do the conversion to int in Painless using Integer.parseInt()

@MakoWish

Copy link
Copy Markdown
Contributor Author

I did, however, notice that we have linked to the wrong blog in the package docs, so if anything, we should update that link with the following so folks don't run into this in the future: https://www.elastic.co/security-labs/detecting-living-off-the-land-attacks-with-new-elastic-integration

If you want to change this PR to update that link instead, that'd be great, otherwise we'll cover it in a follow-up PR.

Please see #7618

@MakoWish

Copy link
Copy Markdown
Contributor Author

@MakoWish Ah I see what you're saying. An alternative to adding an additional processor to the pipeline could also be to change the if check in the blocklist script processor to compare with "0" instead, or do the conversion to int in Painless using Integer.parseInt()

Up to you. All options would accomplish the same. I just happened to go with the convert processor.

@ajosh0504

Copy link
Copy Markdown
Contributor

Let's go with comparing against a string for simplicity. Here's the result of a simple test:

PUT _ingest/pipeline/my-pipeline
{
"description": "My optional pipeline description",
"processors": [
  {
    "script": {
      "description": "My optional processor description",
      "lang": "painless",
      "source": """if (ctx.containsKey('problemchild') && ctx['problemchild'].containsKey('prediction') && ctx['problemchild']['prediction'] == '0') {
        ctx.blocklist_label = 0
        }
        else {
        ctx.blocklist_label = 1
        }"""
    }
  }
  ]
}
POST _ingest/pipeline/my-pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "problemchild": {
          "prediction": "0"
      }
    }
    }
  ]
}

Result:

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "-3",
        "_source": {
          "blocklist_label": 0,
          "problemchild": {
            "prediction": "0"
          }
        },
        "_ingest": {
          "timestamp": "2023-08-31T09:12:20.172890549Z"
        }
      }
    }
  ]
}
@ajosh0504

ajosh0504 commented Aug 31, 2023

Copy link
Copy Markdown
Contributor

Also, can we consolidate all these bug /broken link fixes into a single PR resulting in a single patch version bump? Thanks!

@MakoWish

Copy link
Copy Markdown
Contributor Author

Consolidating into #7618

@MakoWish MakoWish closed this Aug 31, 2023
@MakoWish
MakoWish deleted the bug_problemchild_pipeline_string_value branch August 31, 2023 16:33
susan-shu-c pushed a commit that referenced this pull request Sep 1, 2023
* Update docs with new blog link

* Commit changes from code review

Co-authored-by: Apoorva Joshi <30438249+ajosh0504@users.noreply.github.com>

* Commit changes from code review

Co-authored-by: Apoorva Joshi <30438249+ajosh0504@users.noreply.github.com>

* Consolidate #7521 and #7577 into #7618

---------

Co-authored-by: Apoorva Joshi <30438249+ajosh0504@users.noreply.github.com>
orestisfl pushed a commit to orestisfl/integrations that referenced this pull request May 15, 2026
* Update docs with new blog link

* Commit changes from code review

Co-authored-by: Apoorva Joshi <30438249+ajosh0504@users.noreply.github.com>

* Commit changes from code review

Co-authored-by: Apoorva Joshi <30438249+ajosh0504@users.noreply.github.com>

* Consolidate elastic#7521 and elastic#7577 into elastic#7618

---------

Co-authored-by: Apoorva Joshi <30438249+ajosh0504@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

5 participants