[tychon] New integration

[chrisberkhout]

About the PR

The current commit history breaks changes into small chunks and explains relevant details in commit messages, but it will be easier to review the final, simplified version, rather than all the details of the process of creating it.

Summary of the approach

We wanted to tidy up the integration (received as TYCHON_Elastic_Integration_Source_2_5.zip) to prepare it for merging into the integrations repository.

The existing integration was large, with many data streams and Kibana objects:

• 19 data streams
• 29 transforms
• 38 data views / index patterns
• 26 dashboards (and separately: 3 visualizations, 2 lens objects)
• 13 tags
• 16 security rules

Data is ingested from JSON files using the filestream input.

ECS fields were used where possible but many non-ECS mappings were added alongside them, mostly outside of a vendor prefix.

We decided to add a tychon.* prefix to all the existing fields, and copy them out to the root wherever they do match an ECS field.

We planned to reduce or remove objects that we don't typically use in our integrations, such as data views, tags, etc.

Summary of changes

There were many miscellaneous changes to fix and tidy the package overall. These included consistently formatting and sorting the content of various file types, removing insignificant differences between files, fixing syntax, updating generated files, normalizing file modes, and updating manifest content.

Work on data streams and their pipeline and system tests involved several steps. First was to get the tests to pass for existing test cases and for the data provided in sample_results_2_5.zip. This involved fixing test files that contained data that was from a different data stream or otherwise incorrect, normalizing test file names and data formatting to be correct and avoid problems with the formatter, adding missing field definitions (and removing some redundant ones), adding or fixing logic to convert or remove values that conflicted with their field type, fixing or improving other processing, and regenerating expected output and sample_event.json files to match current logic.

The next step was to add the tychon prefix to field names wherever it was missing. This involved changes to field defintions, pipeline logic and expected test output. Values from the prefixed fields were copied out to ECS fields wherever possible. Data stream names had the redundant tychon_ prefix removed.

The final step was to extract duplicate pipeline logic by splitting it into multiple pipelines. For all data streams, the default.yml pipeline would only run other pipelines and handle errors, the common_init.yml and common_host.yml pipelines would be the same wherever they appear, and the remaining logic that varied between data streams was put into rest.yml files.

Work was done on transforms to clean up their definitions, remove redundant conditions, run them more frequently, use more consistent index names, align their field definitions with those in the source data stream, differentiate between source and destination events, set data stream ILM policies, and consolidate unnecessarily duplicated transforms. The general purpose of transforms was to de-duplicate events. However, for the data streams cve, softwareinventory, stig and systemcerts, there were multiple transforms, which served the purpose of building indexes with different subsets of the source data. These were consolidated for simplicity, as the subsets can be easily distinguished using filters on existing fields (or in the case of systemcerts, on the new field labels.source).

Changes were made to dashboards to fix export, embed rather than link to visualizations and lens objects, to switch from using custom data views to using logs-* with appropriate filters, and to use prefixed field names. Some panels in dashboards were removed: one LOG_STREAM_EMBEDDABLE panel because it is currently broken and soon to be deprecated, and a panel using Windows Application Logs data that isn't collected by this integration. Two dashboards were deleted: '[TYCHON] - Agentless Event Logs' because its only data source was the 'Windows Application Logs (TYCHON)' data view, and '[TYCHON] - Device Control Events' since available data doesn't make it render anything useful. Some other minor tweaks were made to make data show correctly. The now unused visualizations, lens objects and data views were removed. Kibana tags were removed. Screenshots were taken of each dashboard displaying current test and sample data and these were added to the package.

I reviewed the security rules, enabled the two that weren't already enabled, added the TYCHON rule tag where it was missing, and updated the definitions to use new index and field names.

Summary of current state

Dashboards

No changes have been made to the dashboards except to have them use updated data sources and fix anything broken. Look at the screenshots to see what is shown on each dashboard.

Each of the 'Endpoint Browser' dashboards, with the exception of 'Host Information' and 'Software Inventory', can be paired with a non-'Endpoint Browser' dashboard that uses the same source data. Although it's not clear that one set of these is intended to replace the other, it may be possible to reduce the number of dashboards by having one per data stream.

Some data streams are not used by any dashboards: cmrs, coams and features.

File reuse and consistency

I used explicit ECS field definitions in data streams, since these definitions are needed for transforms anyway (these were later removed from the data streams).

All copies of the following files are identical throughout the package:

• common_host.yml (ingest pipeline)
• common_init.yml (ingest pipeline)
• default_policy.json (ILM policy)
• ecs-required.yml (field definitions)
• filestream.yml (field definitions)
• is-transform-source-false.yml (field definitions)
• is-transform-source-true.yml (field definitions)
• tychon-agent.yml (field definitions)
• filestream.yml.hbs (input configuration)
• elasticsearch/transform/.../manifest.yml (transform definitions)
• test-default-config.yml (test config)
• docker-compose.yml (test setup)

For each data stream, the default.yml pipelines vary only in description and, in one case, which other pipelines are called.

For each data stream, pipeline and system test input data is the same. Each has a single example. These could be extended with data from sample_results_2_5.zip.

Field definition files in transforms are the same as in the corresponding data streams, with the exception of is-transform-source-*.yml files, and the absence of ECS definitions that aren't necessary in the data streams.

Each transform.yml differs only in description, and source, destination and alias names.

Data streams

Most data streams read data that includes a similar set of base fields. The unusual data streams, which serve special purposes are externaldevicecontrol and coams.

The externaldevicecontrol data stream collects from files of two different names: tychon_dcm_event_info.json and tychon_dcm_policy_info.json. The sample_results_2_5.zip file has an example named tychon_devicecontrolpolicy_info.json that would appear to match the latter. The "event" data may differ.

The systemcerts data stream collects from both tychon_hostcerts_info.json and tychon_listeningcerts_info.json, but the distinction between these seems to be clear based on the original transform condititions.

The cmrs data stream collects from tychon_cmrs.json, but in sample_results_2_5.zip the corresponding file is named tychon_cmrsinventory_info.json, so one may be out of date.

Tools

I wrote several small scripts to automate work on the package. One that may be useful during review is to copy data from tests and sample_results_2_5.zip into a running agent container.

I also have some summary information in a spreadsheet.

Related issues

• Top level issue [New Integration] Tychon #9214

Outstanding issues for consideration

Is it worth adding a "remove these if they are in ECS" option?
➛ I don't think so

Keeping identical files in sync may warrant inclusion in the package of some kind of script or at least documentation.
➛ Documentation added in DEVELOPMENT.md

The data comes from JSON files created by an external software. It often has a lot of information about a given host. The files could be collected from anywhere, so I thought it made sense not to add information about machine where the Elastic Agent happened to be running, and I set publisher_pipeline.disable_host: true. However, some dashboards were using those host fields. If there will never be a mismatch between the host described in the data files and the host on which Elastic Agent is running, and if it's really best to have Elastic Agent add host fields, we could bring that data back. In that case I would suggest bring it back as an option that can be disabled when adding a policy.
➛ Removed the publisher_pipeline.disable_host: true setting

Should transforms filter for event.kind == "state" as was done previously? Now without that filter we have the benefit that error events appear in the destination indexes and are easier to find.
➛ This filter has been left out

We may be able to clarify some terminology. For example, the systemcerts data stream collects data about certifications from two sources. One is listening certificates, the other is is referred to by several names ("host", "filesystem").

We could expand the README to be a closer match to our usual structure and content.

In visualizations, when some fields were changed to use the tychon prefix, there is a warning that "This might be an approximation. For more precise results, you can enable accuracy mode, but it increases load on the Elasticsearch cluster." I understand why some calculations are approximate, but I'm not sure why this warning wasn't there for ECS fields but appeared when switched to prefixed by otherwise identical field. Perhaps the warning was a delayed effect of switching from using custom data views to using logs-* with filters.

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[chrisberkhout]

/test

[efd6]

I've only looked at the first two data streams.

There is a bunch of fields that are duplicated under tychon. … with exactly the same definitions as the corresponding ECS fields. Do we want to do that? Is there some differentiation that make this worthwhile?

[chrisberkhout]

I've only looked at the first two data streams.

Thanks!

There is a bunch of fields that are duplicated under tychon. … with exactly the same definitions as the corresponding ECS fields. Do we want to do that? Is there some differentiation that make this worthwhile?

The intended benefit is that users can find everything under tychon.* without having to think about the fact that, for example, destination.ip is an ECS field but destination.name is not. And whatever is duplicated outside of the prefix is a real ECS field. That's the approach we agreed on with Tychon in the beginning. What do you think?

[efd6]

That's the approach we agreed on with Tychon in the beginning. What do you think?

I think that's fair. It might be worth adding a "remove these if they are in ECS" option, but discuss with others.

[efd6]

Added one data stream to reviewed set.

[kcreddy]

LGTM based on the 3 datastreams I reviewed: volume, systemcerts, and stig.

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[efd6]

Minor comments only. Then LGTM.

I mechanically checked all the files noted in your DEVELOPMENT.md comment, and the files that I added as inclusions. On the basis that the context for all of these is the same in the different data streams, they are all good. I checked all the remaining files manually (conditions apply†). I also looked at the dashboard screen shots.

GitHub really struggles to cope with a set of changes this large; I frequently needed to close and reload the page due to it grinding and I did most of the review in my editor because of that. Let's not do a PR this large again.

†I did not look at any of the transform files or the dashboard definitions.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: b307077

History

• 💔 Build #16466 failed f38a3b4
• 💚 Build #16459 succeeded a2bc82f
• 💚 Build #16430 succeeded d88f7da
• 💚 Build #16420 succeeded 9bc3f95
• 💚 Build #16194 succeeded d2d4edd

cc @chrisberkhout

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
86.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elastic-vault-github-plugin-prod]

Package tychon - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=tychon

[efd6]

🎉