[Security Solution] add policy_response_failure defend insight type by joeypoon · Pull Request #231908 · elastic/kibana

joeypoon · 2025-08-15T05:22:27Z

Summary

Adds a new Defend Insight (AKA. Automatic Troubleshooting) type, policy_response_failure. This Defend Insight type checks the endpoint policy responses for warnings and failures and provides remediation suggestions.

In order to provide better responses for policy response failures, this PR also introduces static KB assets for Defend Insights. policy_response_failure type requests are enriched with relevant KB assets.

The new policy_response_failure Defend Insight type is feature flagged under defendInsightsPolicyResponseFailure.

anonymized_events_retriever and get_anonymized_events directories renamed to events_retriever and get_events due to max path length restriction.

This PR only contains the API changes for this feature.

Corresponding PR to update Security AI Prompt package.

Checklist

Unit or functional tests were updated or added to match the most common scenarios

elasticmachine · 2025-08-15T13:31:27Z

Pinging @elastic/security-defend-workflows (Team:Defend Workflows)

ferullo · 2025-08-15T17:50:20Z

...ant/server/knowledge_base/defend_insights/policy_response_failure/download_user_artifacts.md

+* macOS: `sudo /Library/Elastic/Endpoint/elastic-endpoint test output`
+* Windows: `C:\\Program Files\\Elastic\\Endpoint\\elastic-endpoint.exe test output`
+
+If network connectivity is the problem and the output doesn't clarify the issue, consider using a tool like curl for further diagnosis. If incorrect proxy information is displayed, review the proxy configuration, noting that Defend advanced options can override these settings. For certificate issues, check the Fleet Server configuration and explore using one of the `advanced.artifacts.user.*` Defend advanced settings.


Can we include links to online webpages in these snippets?

yeah, I think we can make it work.

szwarckonrad

A few questions, but otherwise the code looks good to me. Two wishes though 😉:

Could you add a script/tool to hydrate events with the policy response failure ones? It’ll make future development easier so we don’t have to generate them manually each time.
Could you include usage examples - i.e., sample request and response? A quick reference for UI implementation phase

szwarckonrad · 2025-08-19T07:26:55Z

...es/shared/kbn-elastic-assistant-common/impl/schemas/defend_insights/common_attributes.gen.ts

+  /**
+   * The suggested remediation for the insight
+   */
+  remediation: z.object({}).catchall(z.unknown()).optional(),


Q: Are we sure we cant tighten this?

I'm intentionally leaving this more open since we're not sure what future remediation objects might look like.

szwarckonrad · 2025-08-19T07:29:36Z

...ights_graph/nodes/retriever/events_retriever/get_events/enrichers/policy_response_failure.ts

+  },
+  index: number
+): string {
+  return `${event['actions.name'][index]}${splitKey}${event['actions.message'][index]}${splitKey}${event['host.os.name'][0]}`;


Could you leave a comment with an example of a returned value?

szwarckonrad · 2025-08-19T07:45:42Z

...aph/nodes/retriever/events_retriever/get_events/retrievers/policy_response_failure_events.ts

+      .filter((bucket) => {
+        const actions = bucket.latest_event.hits.hits[0]._source.Endpoint.policy.applied.actions;
+        return actions.some((action) => action.status === 'failure' || action.status === 'warning');
+      })
+      .map((bucket) => {
+        const latestPolicyResponse = bucket.latest_event.hits.hits[0];
+        const failedActions = latestPolicyResponse._source.Endpoint.policy.applied.actions.filter(
+          (action) => action.status === 'failure' || action.status === 'warning'
+        );


We filter out failure || warning actions in both filter and map, is there a need for that?

This is so that we don't have nulls in the returned array.

szwarckonrad · 2025-08-19T07:57:02Z

.../lib/defend_insights/graphs/default_defend_insights_graph/prompts/policy_response_failure.ts

+    promptGroupId: promptGroupId.defendInsights.policyResponseFailure,
+    promptIds: [
+      promptDictionary.defendInsightsPolicyResponseFailureDefault,
+      promptDictionary.defendInsightsPolicyResponseFailureRefine,
+      promptDictionary.defendInsightsPolicyResponseFailureContinue,
+      promptDictionary.defendInsightsPolicyResponseFailureGroup,
+      promptDictionary.defendInsightsPolicyResponseFailureEvents,
+      promptDictionary.defendInsightsPolicyResponseFailureEventsId,
+      promptDictionary.defendInsightsPolicyResponseFailureEventsEndpointId,
+      promptDictionary.defendInsightsPolicyResponseFailureEventsValue,
+      promptDictionary.defendInsightsPolicyResponseFailureRemediation,
+      promptDictionary.defendInsightsPolicyResponseFailureRemediationMessage,


Q: Isnt there a mechanism to fetch all by promptGroupId?

I don't think so, I only see getPrompt and getPromptsByGroupId in the exports and the promptIds arg is required.

szwarckonrad · 2025-08-19T07:58:24Z

...c_assistant/server/lib/defend_insights/graphs/default_defend_insights_graph/schemas/index.ts

-    return getDefendInsightsIncompatibleVirusGenerationSchema(prompts);
+  switch (type) {
+    case DefendInsightType.Enum.incompatible_antivirus:
+      return getDefendInsightsIncompatibleVirusGenerationSchema(prompts);


getDefendInsightsIncompatibleVirusGenerationSchema can we stick to AntiVirus? Might be confusing to someone not familiar with this part of Kibana :D

Ah yeah, good catch.

szwarckonrad · 2025-08-19T07:59:54Z

.../lib/defend_insights/graphs/default_defend_insights_graph/schemas/policy_response_failure.ts

+          .describe(prompts.events),
+        remediation: z
+          .object({
+            message: z.string().describe(prompts.remediationMessage ?? ''),


I see we expect a message field there. Maybe we can build the schema above step by step and start with message instead of leaving it open-ended for now?

I might be misinterpreting your suggestion here but I think you're suggesting that we remove remediation and just have message? I did it this way:

to make it clear it's a remediation message, not just a generic message

to keep the schema for insights more consistent as we might have different remediation shapes in the future

szwarckonrad · 2025-08-19T08:05:30Z

...urity/plugins/security_solution/server/endpoint/services/workflow_insights/builders/index.ts

+    case DefendInsightType.Enum.policy_response_failure:
+      return buildPolicyResponseFailureWorkflowInsights(params);


Should we put it behind a feature flag?

This is flagged at the API level. Agree that we'll want a flag at the UI level as well but that will be in separate PR when we add the API call for this insight type.

joeypoon · 2025-08-19T10:29:55Z

Thanks for taking a look @szwarckonrad 🙇.

Could you add a script/tool to hydrate events with the policy response failure ones? It’ll make future development easier so we don’t have to generate them manually each time.

I believe the scripts/endpoint/resolver_generator script already randomly adds policy response failures. Can generate a handful of endpoints for more failure type coverage.

Could you include usage examples - i.e., sample request and response? A quick reference for UI implementation phase

This is kind of chunky, I'll share with you on slack.

Adds a new Defend Insight type, `policy_response_failure`. This Defend Insight type checks the endpoint policy responses for warnings and failures and provides remediation suggestions.

spong · 2025-08-21T18:56:50Z

...tions/security/plugins/elastic_assistant/server/routes/defend_insights/get_defend_insight.ts

          if (!ctx.licensing.license.hasAtLeast('enterprise')) {
            return response.forbidden({
              body: {
                message:
-                  'Your license does not support Defend Workflows. Please upgrade your license.',
+                  'Your license does not support Automatic Troubleshooting. Please upgrade your license.',
              },
            });
          }


FYI, there's a helper utility for performing license, authenticated user and FF checks you can use:

kibana/x-pack/solutions/security/plugins/elastic_assistant/server/routes/evaluate/post_evaluate.ts

Lines 115 to 121 in c132340

// Perform license, authenticated user and evaluation FF checks

const checkResponse = await performChecks({

capability: 'assistantModelEvaluation',

context: ctx,

request,

response,

});

wasn't aware of this, thanks for the tip. since performChecks is explicitly checking the license req for AI assistant, going to leave this as is for now since we want defend insights to maintain a separate license req (even though technically it's the same level as AI assistant right now).

.../security/plugins/elastic_assistant/server/ai_assistant_data_clients/knowledge_base/index.ts

x-pack/platform/packages/shared/kbn-elastic-assistant-common/impl/capabilities/index.ts

…ghts-policy-response-failures

natasha-moore-elastic

API doc changes LGTM

…no-cache --fix'

…ures

spong

Checked out, tested KB features locally with FF off, and code reviewed relevant GenAI changes -- LGTM! 👍

…ures

elasticmachine · 2025-08-28T23:28:25Z

💚 Build Succeeded

Buildkite Build
Commit: 4f72c7b

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	10.2MB	10.2MB	+108.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`elasticAssistant`	273.8KB	273.8KB	+39.0B
`securitySolution`	96.0KB	96.1KB	+39.0B
total			+78.0B

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
`securitySolution`	677	678	+1

Total ESLint disabled count

id	before	after	diff
`securitySolution`	777	778	+1

History

💚 Build #332600 succeeded 1a80aa4
💔 Build #332497 failed cbf1c59
💚 Build #331956 succeeded 12243c1
💔 Build #331769 failed 2b5e08b
💔 Build #331751 failed 2d67960
💚 Build #331163 succeeded c132340

…lastic#231908) ## Summary Adds a new Defend Insight (AKA. Automatic Troubleshooting) type, `policy_response_failure`. This Defend Insight type checks the endpoint policy responses for warnings and failures and provides remediation suggestions. In order to provide better responses for policy response failures, this PR also introduces static KB assets for Defend Insights. `policy_response_failure` type requests are enriched with relevant KB assets. The new `policy_response_failure` Defend Insight type is feature flagged under `defendInsightsPolicyResponseFailure`. `anonymized_events_retriever` and `get_anonymized_events` directories renamed to `events_retriever` and `get_events` due to max path length restriction. This PR only contains the API changes for this feature. Corresponding [PR](elastic/integrations#14946) to update Security AI Prompt package. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Garrett Spong <spong@users.noreply.github.com>

stephmilovic · 2025-09-05T22:59:22Z

x-pack/solutions/security/plugins/elastic_assistant/server/lib/prompt/defend_insight_prompts.ts

+- combine duplicate insights into the same 'group' (e.g. AVG + AVG Free + AVG Hub + AVG Antivirus)
+- remove insights with no events
+    `,
+    CONTINUE: `Continue exactly where you left off in the JSON output below, generating only the additional JSON output when it's required to complete your work. The additional JSON output MUST ALWAYS follow these rules:


Hey I'm updating the prompts for something else and I think you may have forgotten to update the integration when you made these changes. My PR will include your changes, so no action needed, but please remember to update the integration in the future

joeypoon added backport:skip This PR does not require backporting Team:Defend Workflows “EDR Workflows” sub-team of Security Solution release_note:feature Makes this part of the condensed release notes labels Aug 15, 2025

joeypoon force-pushed the feature/defend-insights-policy-response-failures branch 2 times, most recently from c116eef to a34117f Compare August 15, 2025 10:05

joeypoon marked this pull request as ready for review August 15, 2025 13:31

joeypoon requested review from a team as code owners August 15, 2025 13:31

joeypoon requested review from gergoabraham, parkiino and szwarckonrad August 15, 2025 13:31

ferullo reviewed Aug 15, 2025

View reviewed changes

szwarckonrad reviewed Aug 19, 2025

View reviewed changes

joeypoon force-pushed the feature/defend-insights-policy-response-failures branch 2 times, most recently from 2988783 to 7a46a7d Compare August 19, 2025 11:57

szwarckonrad approved these changes Aug 19, 2025

View reviewed changes

joeypoon force-pushed the feature/defend-insights-policy-response-failures branch from 7a46a7d to d06f7f7 Compare August 19, 2025 14:14

[Security Solution] add policy_response_failure defend insight type

c132340

Adds a new Defend Insight type, `policy_response_failure`. This Defend Insight type checks the endpoint policy responses for warnings and failures and provides remediation suggestions.

joeypoon force-pushed the feature/defend-insights-policy-response-failures branch from d06f7f7 to c132340 Compare August 20, 2025 12:21

spong reviewed Aug 21, 2025

View reviewed changes

.../security/plugins/elastic_assistant/server/ai_assistant_data_clients/knowledge_base/index.ts Outdated Show resolved Hide resolved

spong reviewed Aug 21, 2025

View reviewed changes

.../security/plugins/elastic_assistant/server/ai_assistant_data_clients/knowledge_base/index.ts Outdated Show resolved Hide resolved

spong reviewed Aug 21, 2025

View reviewed changes

x-pack/platform/packages/shared/kbn-elastic-assistant-common/impl/capabilities/index.ts Show resolved Hide resolved

joeypoon added 2 commits August 22, 2025 19:20

PR comments

8862eb2

Merge remote-tracking branch 'upstream/main' into feature/defend-insi…

2d67960

…ghts-policy-response-failures

joeypoon requested a review from a team as a code owner August 22, 2025 10:28

natasha-moore-elastic approved these changes Aug 22, 2025

View reviewed changes

joeypoon and others added 6 commits August 22, 2025 19:55

lint

bf44185

[CI] Auto-commit changed files from 'node scripts/eslint_all_files --…

2b5e08b

…no-cache --fix'

Merge branch 'main' into feature/defend-insights-policy-response-fail…

12243c1

…ures

Merge branch 'main' into feature/defend-insights-policy-response-fail…

03af336

…ures

FF doc loader

cbf1c59

fix loader test

1a80aa4

spong approved these changes Aug 28, 2025

View reviewed changes

Merge branch 'main' into feature/defend-insights-policy-response-fail…

4f72c7b

…ures

joeypoon merged commit f6e2d22 into elastic:main Aug 29, 2025
13 checks passed

kibanamachine added the v9.2.0 label Aug 29, 2025

stephmilovic mentioned this pull request Sep 5, 2025

[Security AI Prompts] Add prompts for value report elastic/integrations#15213

Merged

stephmilovic reviewed Sep 5, 2025

View reviewed changes

joeypoon mentioned this pull request Sep 16, 2025

[Internal]: Doc updates/additions for automatic troubleshooting GA elastic/docs-content#2968

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] add policy_response_failure defend insight type#231908

[Security Solution] add policy_response_failure defend insight type#231908
joeypoon merged 10 commits intoelastic:mainfrom
joeypoon:feature/defend-insights-policy-response-failures

joeypoon commented Aug 15, 2025 •

edited

Loading

elasticmachine commented Aug 15, 2025

ferullo Aug 15, 2025

joeypoon Aug 18, 2025

szwarckonrad left a comment •

edited

Loading

szwarckonrad Aug 19, 2025

joeypoon Aug 19, 2025

szwarckonrad Aug 19, 2025

szwarckonrad Aug 19, 2025

joeypoon Aug 19, 2025

szwarckonrad Aug 19, 2025

joeypoon Aug 19, 2025

szwarckonrad Aug 19, 2025

joeypoon Aug 19, 2025

szwarckonrad Aug 19, 2025

joeypoon Aug 19, 2025 •

edited

Loading

szwarckonrad Aug 19, 2025

joeypoon Aug 19, 2025

joeypoon commented Aug 19, 2025 •

edited

Loading

spong Aug 21, 2025

joeypoon Aug 22, 2025

Uh oh!

Uh oh!

Uh oh!

natasha-moore-elastic left a comment

spong left a comment

elasticmachine commented Aug 28, 2025

ESLint disabled line counts

Total ESLint disabled count

Uh oh!

stephmilovic Sep 5, 2025

Labels

8 participants

		case DefendInsightType.Enum.policy_response_failure:
		return buildPolicyResponseFailureWorkflowInsights(params);

	// Perform license, authenticated user and evaluation FF checks
	const checkResponse = await performChecks({
	capability: 'assistantModelEvaluation',
	context: ctx,
	request,
	response,
	});

Conversation

joeypoon commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

elasticmachine commented Aug 15, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

szwarckonrad left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joeypoon Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joeypoon commented Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

natasha-moore-elastic left a comment

Choose a reason for hiding this comment

spong left a comment

Choose a reason for hiding this comment

elasticmachine commented Aug 28, 2025

💚 Build Succeeded

Metrics [docs]

Async chunks

Page load bundle

ESLint disabled line counts

Total ESLint disabled count

History

Uh oh!

Choose a reason for hiding this comment

Labels

8 participants

joeypoon commented Aug 15, 2025 •

edited

Loading

szwarckonrad left a comment •

edited

Loading

joeypoon Aug 19, 2025 •

edited

Loading

joeypoon commented Aug 19, 2025 •

edited

Loading