[sophos] Various improvements and log samples from 18.5 by andrewkroh �� Pull Request #3127 · elastic/integrations

andrewkroh · 2022-04-19T19:02:35Z

What does this PR do?

Remove space from sophos.xg.trans_src_ip field name in mapping. Sync fix from [Filebeat] Remove space from field sophos.xg.trans_src_ ip beats#25250.
Do not modify event.original.
Populate url.* fields based on sophos.xg.url.
Rename sophos.xg.reason to event.reason (ECS).
Normalize all sophos.xg.* keys to lowercase.
Lowercase network.transport as per ECS.
Format source.mac and destination.mac as per ECS.
Add Sophos Firewall 18.5 log samples taken from documentation.
Set the event.code from the message ID (and remove sophos.xg.messageid).
Consolidate geoip enrichment
Consolidate related.ip processors
Consolidate lowercase network.transport/protocol
Consolidate related.user processing
Consolidate related.hash
Consolidate network.{bytes,packets} calculations
Add network.community_id
Add {url,source,destination}.domain to related.hosts
Set event.duration (ns) for WAF events using reponsetime (us)
Remove client/server mappings. This was a large amount of duplication causing increased event sizes.
Add reference link to xg syslog formats

Anti-Virus

Set destination.domain from sophos.xg.dstdomain in anti-virus log.
Use sophos.xg.dst_domainname to set destination.domain for Anti-Virus SMTP events.
Use sophos.xg.src_domainname to set source.domain for Anti-Virus SMTP events.
Set network.protocol for anti-virus logs.

Anti-Spam

Set network.protocol for anti-spam logs.

Content Filtering

Set network.protocol for Content Filtering. Use the URL scheme to derive the protocol.

Sandstorm

Set file.name for Sandstorm events (there was a typo in the if condition).
Set url.domain, destination.domain, and destination.ip for Sandstorm web events.
Handle sha256 Sandstorm file hashes. In 17.5 and earlier the sha1sum contained the sha1 checksum of the file being analyzed. In 18.0 and later the sha1sum contains the sha256 checksum of the file.

System Health

Convert sophos.xg.collisions, sophos.xg.receiveddrops, and sophos.xg.transmitteddrops to float in document _source.

Wireless Protection

Change sophos.xg.clients_conn_ssid to a long in the mapping.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

elasticmachine · 2022-04-19T19:05:46Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-04-21T15:53:11.842+0000
Duration: 16 min 51 sec

Test stats 🧪

Test	Results
Failed	0
Passed	28
Skipped	0
Total	28

�� GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

Sync fix from elastic/beats#25250.

Fix inconsistency in that only some keys were being lowercase. And the way it was done before was not good because it modified the event.original value.

Set the event.code from the message ID (and remove sophos.xg.messageid). Set destination.domain from dstdomain in anti-virus log. Fix event.original handling. The original message was being modified by the pipeline. Now the raw message is stored in event.original without modification. Lower case network.transport as per ECS. Set event.reason from sophos.xg.reason (and remove sophos.xg.reason).

Sandstorm Set file.name for sandstorm events (there was a typo in the if condition). Set url.domain, destination.domain, and destination.ip for sandstorm web events. System Health Convert sophos.xg.collisions, sophos.xg.receiveddrops, and sophos.xg.transmitteddrops to float in _source. Wireless Protection Change sophos.xg.clients_conn_ssid to long in mapping.

Use dst_domainname to set destination.domain for Anti-Virus SMTP events. Use src_domainname to set source.domain for Anti-Virus SMTP events.

In 17.5 and earlier the sha1sum contained the sha1 checksum of the file being analyzed. In 18.0 and later the sha1sum contains the sha256 checksum of the file.

Use the URL scheme to derive the protocol.

This was a large amount of duplication causing increased event sizes.

[git-generate] cd packages/sophos elastic-package format elastic-package build elastic-package test pipeline -g

elasticmachine · 2022-04-20T21:45:53Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

Replaces `ctx?.` with `ctx.` in all of the sophos.xg pipelines.

…ations into feature/labels-cloud-ephemeral-components * 'feature/labels-cloud-ephemeral-components' of github.com:v1v/integrations: (23 commits) use branch standardise labels/tags Update .ci/Jenkinsfile [ci][terraform][aws] tags with metadata Bump github.com/elastic/elastic-package from 0.46.0 to 0.47.0 (elastic#3182) Spring boot package [Memory - data stream] (elastic#2979) [cisco_ise] Add Cisco ISE package (elastic#2855) [apache_spark][executor] Add Apache Spark package with Executor data stream (elastic#2943) [apache_spark][driver] Add Apache Spark package with Driver data stream (elastic#2945) cisco_duo: simplify grok expression for handling ports (elastic#3170) cisco_duo: fix handling of IP addresses with port numbers (elastic#3117) [sophos] Various improvements and log samples from 18.5 (elastic#3127) [cisco_asa] fix visualizations (elastic#3146) [apache_spark][application] Add Apache Spark package with Application data stream (elastic#2941) Fix pagination bug that skipped events when more than one page is present. (elastic#3140) Cis k8s name migration (elastic#3113) remove exported fields; these will be added to Kibana docs instead (elastic#3093) Exclude ecosystem as owner of subdirectories of packages (elastic#3132) [awsfargate] bump package version (elastic#3130) add first csp rule template (elastic#3081) ...

Remove space from sophos.xg.trans_src_ip field name in mapping. Sync fix from elastic/beats#25250. Do not modify event.original. Populate `url.*` fields based on `sophos.xg.url`. Rename `sophos.xg.reason` to `event.reason` (ECS). Normalize all `sophos.xg.*` keys to lowercase. Lowercase `network.transport` as per ECS. Format `source.mac` and `destination.mac` as per ECS. Add Sophos Firewall 18.5 log samples taken from documentation. Set the `event.code` from the message ID (and remove `sophos.xg.messageid`). Consolidate geoip enrichment Consolidate related.ip processors Consolidate lowercase network.transport/protocol Consolidate related.user processing Consolidate related.hash Consolidate network.{bytes,packets} calculations Add network.community_id Add {url,source,destination}.domain to related.hosts Set event.duration (ns) for WAF events using reponsetime (us) Remove client/server mappings. This was a large amount of duplication causing increased event sizes. Add reference link to xg syslog formats Anti-Virus - Set `destination.domain` from `sophos.xg.dstdomain` in anti-virus log. - Use `sophos.xg.dst_domainname` to set `destination.domain` for Anti-Virus SMTP events. - Use `sophos.xg.src_domainname` to set `source.domain` for Anti-Virus SMTP events. - Set `network.protocol` for anti-virus logs. Anti-Spam - Set `network.protocol` for anti-spam logs. Content Filtering - Set `network.protocol` for Content Filtering. Use the URL scheme to derive the protocol. Sandstorm - Set `file.name` for Sandstorm events (there was a typo in the if condition). - Set `url.domain`, `destination.domain`, and `destination.ip` for Sandstorm web events. - Handle sha256 Sandstorm file hashes. In 17.5 and earlier the sha1sum contained the sha1 checksum of the file being analyzed. In 18.0 and later the sha1sum contains the sha256 checksum of the file. System Health - Convert `sophos.xg.collisions`, `sophos.xg.receiveddrops`, and `sophos.xg.transmitteddrops` to float in document `_source`. Wireless Protection Change `sophos.xg.clients_conn_ssid` to a long in the mapping. * Update files [git-generate] cd packages/sophos elastic-package format elastic-package build elastic-package test pipeline -g * Add changelog * Remove null-safe operator when accessing ctx Replaces `ctx?.` with `ctx.` in all of the sophos.xg pipelines. * Sort ecs.yml and fields.yml by name

andrewkroh added bug Something isn't working, use only for issues enhancement New feature or request Team:Security-External Integrations Integration:sophos Sophos labels Apr 19, 2022

andrewkroh added 20 commits April 20, 2022 12:29

Remove space from trans_src_ip

c790137

Sync fix from elastic/beats#25250.

Lowercase sophos.xg fields

06ea003

Fix inconsistency in that only some keys were being lowercase. And the way it was done before was not good because it modified the event.original value.

Set domain name

1a630d7

Use dst_domainname to set destination.domain for Anti-Virus SMTP events. Use src_domainname to set source.domain for Anti-Virus SMTP events.

Consolidate geoip enrichment

7e9e105

Consolidate lowercase network.transport/protocol

0fdc468

Consolidate related.ip processors

1778832

Consolidate related.user processing

3d39467

Consolidate related.hash and handle sha256 sandstorm file hashes

31c8f7b

In 17.5 and earlier the sha1sum contained the sha1 checksum of the file being analyzed. In 18.0 and later the sha1sum contains the sha256 checksum of the file.

Consolidate network.bytes/packets, add community_id

76f2d7d

Add {url,source,destination}.domain to related.hosts

346e161

Set event.duration (ns) for WAF events using reponsetime (us)

906adc2

Set network.protocol for anti-spam logs

e48b1e0

Set network.protocol for anti-virus logs

bee6afb

Format MAC addresses as per ECS

d2743be

Add event.reason mapping

f4bd720

Set network.protocol for Content Filtering

251be59

Use the URL scheme to derive the protocol.

Remove client/server mappings

5aa2303

This was a large amount of duplication causing increased event sizes.

Add reference link to xg syslog formats

1a92089

andrewkroh changed the title ~~[sophos] Lowercase names, Add uri_parts, Fix trans_src_ip~~ Apr 20, 2022

andrewkroh added 3 commits April 20, 2022 17:15

Update xg sample_event.json

980029d

Update files

e36a8a5

[git-generate] cd packages/sophos elastic-package format elastic-package build elastic-package test pipeline -g

Add changelog

8216183

andrewkroh marked this pull request as ready for review April 20, 2022 21:45

andrewkroh requested a review from a team as a code owner April 20, 2022 21:45

efd6 approved these changes Apr 20, 2022

View reviewed changes

Comment thread packages/sophos/data_stream/xg/elasticsearch/ingest_pipeline/sandstorm.yml Outdated

andrewkroh added 2 commits April 21, 2022 11:23

Remove null-safe operator when accessing ctx

624f50e

Replaces `ctx?.` with `ctx.` in all of the sophos.xg pipelines.

Sort ecs.yml and fields.yml by name

ab1fe1c

andrewkroh merged commit 919fe81 into elastic:main Apr 21, 2022

This was referenced Apr 21, 2022

[Filebeat] sophos.xg - Handle new log field names elastic/beats#31388

Merged

[Filebeat] Add sophos XG log samples containing new field names elastic/beats#31038

Closed

andrewkroh mentioned this pull request May 2, 2022

[production] Promote many packages from snapshot elastic/package-storage#4302

Merged

Navigation Menu

Uh oh!

[sophos] Various improvements and log samples from 18.5 - #3127

[sophos] Various improvements and log samples from 18.5#3127
andrewkroh merged 25 commits into
elastic:mainfrom
andrewkroh:sophos/fix-field-names-uri-parts

andrewkroh commented Apr 19, 2022 •

edited

Loading

elasticmachine commented Apr 19, 2022 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Apr 20, 2022

Uh oh!

Labels

3 participants

Uh oh!

Conversation

andrewkroh commented Apr 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

elasticmachine commented Apr 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

��� GitHub comments

elasticmachine commented Apr 20, 2022

Uh oh!

Labels

3 participants

andrewkroh commented Apr 19, 2022 •

edited

Loading

elasticmachine commented Apr 19, 2022 •

edited

Loading

�� GitHub comments