OpenCanary update, resolves multiple issues

[colin-stubbs]

• Bug
• Enhancement

Proposed commit message

Resolves issues #12911, #13024, #13025. Relevant to resolution of #2518.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

Resolves multiple issues:

• Only set event.kind == alert if event is clearly not a generic application log message
• Migrate logfile to filestream based filebeat configuration
• Add http_endpoint for webhook based ingest
• Add system tests for both filestream and http_endpoint as none currently exist.
• Add pipeline test for webhook'ed events
• Retain password fields if desired, e.g. only remove password field if redaction requested. The current behaviour always removes passwords.
• Provide option to remove or retain ECS mapped fields, currently this option does not exist and ECS mapped fields are always removed.
• Add dashboard and dashboard screenshot. Currently not included in integration.
• Fix confused tftp/vnc field names. This is a typeo/bug due to lack of sufficient sample logs for pipeline or system testing.
• Add more complete/wider variety of sample logs for testing, e.g. TFTP & VNC events, NTP events, SNMP events etc
• Define appropriate fields based on more complete/wider variety of sample logs
• Fix as yet unknown logtype handling, e.g. current ingest pipeline script allows the logtype integer value to be left in a field defined as keyword leading to type conflicts and incomplete search results.
• Update known logtype map based on latest opencanary repo code. Current list is not up to date with opencanary code.

Testing:

• elastic-package lint && check && build
• elastic-package test system --generate
• elastic-package test pipeline --generate
• elastic-package test
• Manual deploy on local elastic-package managed stack and ingest of logfile
• Manual deploy on remote Elastic Cloud stack and ingest of webhooks from real opencanary honeypots

How to test this PR locally

Install and operate OpenCanary, ideally via docker.
Use Elastic Agent to ingest OpenCanary log file OR webhooks.
Scan OpenCanary with nmap with scripting to trigger events, e.g. nmap -sC 127.0.0.1
Review

Related issues

• Closes [opencanary]: Sets all log message event.kind == alert leading to unwanted alert noise in Elastic Security #12911
• Closes [opencanary]: Add http_endpoint to support OpenCanary webhook output #13024
• Closes [opencanary]: Various bug fixes / enhancements #13025
• Relates Migrate uses of the logfile input to filestream #2518

Screenshots

New basic summary dashboard added,

[efd6]

/test

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: b1617e1

Failed CI Steps

• Check integrations opencanary

History

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[botelastic]

Hi! This PR has been stale for a while and we're going to close it as part of our cleanup procedure. We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team. Feel free to re-open this PR if you think it should stay open and is worth rebasing. Thank you for your contribution!

[elastic-vault-github-plugin-prod]

Package opencanary - 0.6.1 containing this change is available at https://epr.elastic.co/package/opencanary/0.6.1/