Fix missing field in transforms of ti packages - #11008
Conversation
| - name: osint | ||
| type: boolean | ||
| - name: "*" | ||
| type: keyword |
There was a problem hiding this comment.
This is a different case, that causes issues after enabling synthetic source because of elastic/package-spec#784.
🚀 Benchmarks reportPackage
|
| Data stream | Previous EPS | New EPS | Diff (%) | Result |
|---|---|---|---|---|
malwarebazaar |
3623.19 | 2801.12 | -822.07 (-22.69%) | 💔 |
threatfox |
4739.34 | 3546.1 | -1193.24 (-25.18%) | 💔 |
url |
5208.33 | 4115.23 | -1093.1 (-20.99%) | 💔 |
Package ti_crowdstrike 👍(1) 💚(0) 💔(1)
Expand to view
| Data stream | Previous EPS | New EPS | Diff (%) | Result |
|---|---|---|---|---|
ioc |
4784.69 | 3623.19 | -1161.5 (-24.28%) | 💔 |
Package ti_eset 👍(2) 💚(2) 💔(3)
Expand to view
| Data stream | Previous EPS | New EPS | Diff (%) | Result |
|---|---|---|---|---|
cc |
12500 | 9009.01 | -3490.99 (-27.93%) | 💔 |
ip |
12987.01 | 9345.79 | -3641.22 (-28.04%) | 💔 |
url |
14084.51 | 8547.01 | -5537.5 (-39.32%) | 💔 |
Package ti_maltiverse 👍(0) 💚(0) 💔(1)
Expand to view
| Data stream | Previous EPS | New EPS | Diff (%) | Result |
|---|---|---|---|---|
indicator |
2624.67 | 2066.12 | -558.55 (-21.28%) | 💔 |
Package ti_mandiant_advantage 👍(0) 💚(0) 💔(1)
Expand to view
| Data stream | Previous EPS | New EPS | Diff (%) | Result |
|---|---|---|---|---|
threat_intelligence |
1557.63 | 1267.43 | -290.2 (-18.63%) | 💔 |
Package ti_misp 👍(1) 💚(0) 💔(1)
Expand to view
| Data stream | Previous EPS | New EPS | Diff (%) | Result |
|---|---|---|---|---|
threat_attributes |
3134.8 | 2659.57 | -475.23 (-15.16%) | 💔 |
Package ti_opencti 👍(0) 💚(0) 💔(1)
Expand to view
| Data stream | Previous EPS | New EPS | Diff (%) | Result |
|---|---|---|---|---|
indicator |
2020.2 | 1602.56 | -417.64 (-20.67%) | 💔 |
Package ti_threatconnect 👍(0) 💚(0) 💔(1)
Expand to view
| Data stream | Previous EPS | New EPS | Diff (%) | Result |
|---|---|---|---|---|
indicator |
534.76 | 444.84 | -89.92 (-16.82%) | 💔 |
To see the full report comment with /test benchmark fullreport
💚 Build Succeeded
History
cc @jsoriano |
|
|
@jsoriano Can you link to the failure this fixes? |
andrewkroh
added
Integration:ti_threatconnect
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
You can find them in https://buildkite.com/elastic/integrations/builds/15135 Basically: The issue is only reproduced if logsdb (or synthetic source) is enabled. |
|
Package ti_abusech - 2.3.2 containing this change is available at https://epr.elastic.co/search?package=ti_abusech |
|
Package ti_anomali - 1.22.2 containing this change is available at https://epr.elastic.co/search?package=ti_anomali |
|
Package ti_cif3 - 1.14.2 containing this change is available at https://epr.elastic.co/search?package=ti_cif3 |
|
Package ti_crowdstrike - 1.1.5 containing this change is available at https://epr.elastic.co/search?package=ti_crowdstrike |
|
Package ti_cybersixgill - 1.30.2 containing this change is available at https://epr.elastic.co/search?package=ti_cybersixgill |
|
Package ti_eclecticiq - 1.2.2 containing this change is available at https://epr.elastic.co/search?package=ti_eclecticiq |
|
Package ti_eset - 1.2.3 containing this change is available at https://epr.elastic.co/search?package=ti_eset |
|
Package ti_maltiverse - 1.2.2 containing this change is available at https://epr.elastic.co/search?package=ti_maltiverse |
|
Package ti_mandiant_advantage - 1.4.1 containing this change is available at https://epr.elastic.co/search?package=ti_mandiant_advantage |
|
Package ti_misp - 1.35.2 containing this change is available at https://epr.elastic.co/search?package=ti_misp |
|
Package ti_opencti - 2.3.3 containing this change is available at https://epr.elastic.co/search?package=ti_opencti |
|
Package ti_otx - 1.25.2 containing this change is available at https://epr.elastic.co/search?package=ti_otx |
|
Package ti_rapid7_threat_command - 2.0.2 containing this change is available at https://epr.elastic.co/search?package=ti_rapid7_threat_command |
|
Package ti_recordedfuture - 1.26.2 containing this change is available at https://epr.elastic.co/search?package=ti_recordedfuture |
|
Package ti_threatconnect - 1.2.2 containing this change is available at https://epr.elastic.co/search?package=ti_threatconnect |
|
Package ti_threatq - 1.28.2 containing this change is available at https://epr.elastic.co/search?package=ti_threatq |
The distinction between source and destination records was lost when #11008 added a field definition for `labels.is_ioc_transform_source` to the destination indices with a value of `"true"`. Here it is restored by: - Moving `labels.is_ioc_transform_source` field definitions into separate files. - Setting the value to `"false"` in the transform destinations. - Incrementing transform version numbers so new destination indices will be built with correct values. - Adjusting dashboards to filter for not `"true"` rather than not exists. Security rules already filter for not `"true"`.
Issue detected in elastic#10919, there is a field in transforms that is not documented. This was only detected when enabling synthetic source (through logsdb), because only in this case we check fields that are not in the source, but are in the index, such as constant_keywords. Not having these fields defined will be a problem when we enable testing with logsdb, and/or when we fix validation of these fields.
The distinction between source and destination records was lost when elastic#11008 added a field definition for `labels.is_ioc_transform_source` to the destination indices with a value of `"true"`. Here it is restored by: - Moving `labels.is_ioc_transform_source` field definitions into separate files. - Setting the value to `"false"` in the transform destinations. - Incrementing transform version numbers so new destination indices will be built with correct values. - Adjusting dashboards to filter for not `"true"` rather than not exists. Security rules already filter for not `"true"`.
Issue detected in elastic#10919, there is a field in transforms that is not documented. This was only detected when enabling synthetic source (through logsdb), because only in this case we check fields that are not in the source, but are in the index, such as constant_keywords. Not having these fields defined will be a problem when we enable testing with logsdb, and/or when we fix validation of these fields.
The distinction between source and destination records was lost when elastic#11008 added a field definition for `labels.is_ioc_transform_source` to the destination indices with a value of `"true"`. Here it is restored by: - Moving `labels.is_ioc_transform_source` field definitions into separate files. - Setting the value to `"false"` in the transform destinations. - Incrementing transform version numbers so new destination indices will be built with correct values. - Adjusting dashboards to filter for not `"true"` rather than not exists. Security rules already filter for not `"true"`.
4 participants
Issue detected in #10919, there is a field in transforms that is not documented.
This was only detected when enabling synthetic source (through logsdb), because only in this case we check fields that are not in the source, but are in the index, such as
constant_keywords.Not having these fields defined will be a problem when we enable testing with logsdb, and/or when we fix validation of these fields.